Rufus / Luccas chat

Present: ../people/rufuspollock and ../people/luccas-mateus

#done/process

Next steps (Rufus)

• A clear explanation of our products and their naming @rufuspollock
  • relation to what we already have and how it will evolve
• Plan for monorepo
  • Purpose and outline
  • Layout
  • Core packages
    • metastore (and its api)
  • Relation to Portal.JS
• Target for creating something with monorepo (or refactoring something existing)
  • Create a demo site from an example
  • https://datahub-enterprise.datopian.com is using datahub-publisher
  • one of the existing production sites

Extras

• Copy over datahub-publisher roadmap into obsidian
• Clarify how datahub-publisher relates

Discussion

• portaljs-template and datahub-publisher are completely distinct and not connected
• Luccas: DataHub has become a catch-all term for anything not related to a client project

What are the needs we have for our product and for clients (partial)

• Quickly, reliabily build elegant open data portal sites (based on CKAN or another backend)
• Deploy an "enterprise" data management system

Define terms

• What is Portal.JS? It is a library for building the frontend for data portals build in a headless style and especially using CKAN or "DataHub Enterprise" as the backend
• What is headless DMS approach? aka JADStack?
• What is DataHub (X)?
  • What do we call what is running DataHub.io currently? Ans: DataHub "Legacy" or to be specific DataHub v2.0
  • Is DataHub v2 being developed or maintained? No, it is legacy and is only receiving critical patches (if at all)
  • What about DataHub v3 at https://tech.datopian.com/datahub/v3/ This is now deprecated but this evolved in late 2021/2022 into what we named DataHub Pages
  • What is DataHub? DataHub is our overall name for our main product line
    • Is Portal.JS part of DataHub? Yes, though the term Portal.JS needs clarifying
  • What are the products under that

Products

DataHub OpenGov: targetted at those especially in national gov, cities or IGOs who want an open data portal (with minimal customization)

DataHub Enterprise: is targetted at enterprises who want an internal data management catalog and data management system

DataHub Consumer: a self-service data storage and sharing platform.

Tech stack questions

MOVED to ../projects/product-plan-2023

• Enterprise stuff is using:
  • NextJS
  • User management: github
  • MetaStore: github
  • Blob Storage: s3
• OpenGov
  • NextJS
  • MetaStore: CKAN
  • Blob Storage: CKAN? s3?
  • User management: CKAN

We will define a "primary" backend (almost certainly not CKAN) and perhaps support one other backend e.g. CKAN because of demand from clients.

Why not CKAN as backend?

• Versioning is painful (impossible to do well)
• Slow
• Heavy
• Hard to contribute to
• Hard to scale
• Application is not complex

Monorepo

🚚 MOVED to ../projects/monorepo-2023

Subject: creating a monorepo

Purpose: we deploy customized "frontends" (usually based on "portal.js") and these have a shared set of components (aka portal.js) plus build off a standard template (or templates) aka "examples". Once an app is created it needs to be maintained and evolved.

The suggestion is to consolidate these different "apps" into a single monorepo for easiest maintenance.

Compared the situation today where we boot a new repo for every client.

packages/
  core
examples/
apps/
  client apps etc

What is the API for the query layer

• review portaljs-template https://github.com/datopian/portaljs-template/blob/master/lib/queries.ts
• Review metastore-lib https://github.com/datopian/metastore-lib-js and https://github.com/datopian/metastore-lib
• Portal.JS: uses graphql in the CKaN example https://github.com/datopian/portal.js/blob/main/examples/ckan/graphql/queries.ts

Concern - initial discussion

• We are joining all our projects in one repo
• What is actually shared?
• portal.js is using react 17 rather than react 18
  • we use --force because it is using react 17
  • whilst new projects use react 18
• Portal.JS we started building and then stopped maintaining
  • so every time … we start again

=> we aren't maintaining portal.js and we should!

Luccas list of projects he knows of

MOVED to ../projects/product-plan-2023

PortalJS-Template

A template for PortalJS websites, ideally next time we build a PortalJS website we just clone the template and start working from there, it already has all the queries, all the types, etc

https://github.com/datopian/portaljs-template

Multi-tenant CKAN/CKAN Cities

Usually CKAN is deployed as an instance, for a particular government agency/NGO/Company etc, which then gets divided into organizations inside that "macro-organization". Multi-tenant CKAN tries to make a instance where multiple NGO/Companies/Government Agencies etc can share the same CKAN deployment.

https://github.com/datopian/portal.js-ckan

Monorepo PortalJS

Trying to setup all PortalJS deployments into a single repo as to share queries, types, code etc

https://github.com/datopian/Nx-portals

Datahub-Publisher

A Data Management System that uses Github as an backend, as of right now completely serverless, tries to use datapackage/frictionless tools as most as possible

https://github.com/datopian/datahub-publisher

datopian website

The Datopian blog/newsletter