SSEN Markdown Rendering Pipeline Integration

Situation

• Our current client SSEN uses CKAN + PortalJS frontend https://data.ssen.co.uk/ with code at: https://github.com/datopian/portals-monorepo/tree/main/apps/portal-sse (See also https://github.com/datopian/ssen for background)
• Wants to use "Flowershow as CMS now" i.e. to publish from markdown files with rich components
• Specifically, want creation of a separate section of the data portal site in which to be published and maintained:
  • Data Roadmap of data to published
  • Ability to update once every x weeks
  • List of triage / responses received with status update
  • Ability to update once every x weeks
  • List of completed accepted Data Triage Responses
  • List of completed denied Data Triage Responses
• In response we stated we would: "integrate a solution for content curation and publication. We will support data-rich components so that editors can:"
  • Create and update the roadmap component.
  • Create tables with a list of triage/responses.
  • Create tables with statuses etc.

In terms of tech stuff:

• seems to have been built off of old Flowershow - e.g. there is this whole /content folder with some old Flowershow template stuff and it has some remains of it's structure and files, .e.g contentlayer.config or our old "getters" https://github.com/datopian/portals-monorepo/tree/4ddc04d554cd76ea68a21171be0aef7ecfaa0c78/apps/portal-sse/content
• But the markdown parsing/rendering is not really executed anywhere, weird…

Problem

• Markdown rendering is probably not working because contentlayer is not even installed
• It is using the "old" datahub/portaljs/flowershow system with contentlayer. This isn't what we use anymore.
• We aren't sure whether content is local or remote …

Questions:

• Do they want to add content to their portal by publishing stuff "data rich" markdown docs? ✅2024-03-22 Yes
• Does this content use live viz? e.g. table or graph components loading data from their portal? ✅2024-03-22 ❌ not totally clear. requirements below need charts but not charts off live data
• Does this content live in a git(hub) repo?
• Is that repo separate from portal(js) repo or part of it? aka can they have their Markdown content in the same repo, e.g. in /content folder?
• Do we have a written statement of what was promised ot the client? ✅2024-03-22 YEs, see below
• Is it part of the main website? ✅2024-03-22 yes.
• ASIDE: importing word documents would be a cool feature …
• Who can do development for the client?
• What is the timeline for this work? 🔑 the preferred timeline is 2-3 weeks from now

Hypothesis

• Enable local (or remote) markdown publishing in the portal using DataHub OS
  • this shouldn't be more complicated than our previous efforts of of upgrading old Flowershow sites to latest Flowershow etc (e.g. on Life Itself ecosystem).
  • integrate this with the latest Flowershow template and add PortalJS components to it so that they can be used in Markdown
• Create a short doc/tutorial/example for doing what they want, specifically
  • A roadmap display (may need to check what this)
  • Creating tables (either direct in markdown or from json, yaml, csv …)
• BONUS 🥇 Write up a short tutorial on how to integrate markdown rendering into a DataHub OS project.

Tasks

Upgrade to latest DataHub template (including datarich markdown rendering):

• install mddb, all the remark/rehype plugins and @portaljs/components
• copy over add app route for handling markdown rendering, .e.g /[[...slug]].tsx from Flowershow template
• test that it works with example markdown (with some portaljs vis) files from /content folder

Roadmap and tables stuff

• create /roadmap.tsx page with exmple Roadmap component 💬2024-04-01 suggest this just markdown if we can possibily do it. Maybe use vega/vegalite or even tailwind. 👍
• (if needed) create any other components and pass them to MDX so that can be used in Markdown 💬2024-04-01 hope would be no custom MDX components.
• clean-up unneded old Flowershow files
• show POC to the client and get feedback?

Rabbit holes / risks

• Clarifying whether markdown is local or remote - remote is probably a bit (but not much) more complicate. But switching between two will add effort
• Remote markdown publishing limits bespoke tailwind usage - is that ok?

No goes

• Create a special component for the roadmap - we are assuming this is something that can be created in vega or similar

Appendix: Notes from meeting with Anu 2024-03-22

Source: https://github.com/datopian/product/blob/main/meetings/product-team-weekly.md#ssen-portal

🚚2024-04-01 Most of this merged above

SSEN portal

Our current client who uses CKAN + PortalJS frontend wants to use Flowershow as CMS now. i.e., we'd like to make this integration (probably we already have it!).

I'm not sure if Flowershow vs DataHub as they share similar features but this client wants to have our PortalJS comopnents lib available so that they can do charts etc within the CMS. They are open to use GitHub as editor.

Links: portal - https://data.ssen.co.uk/ and repo - https://github.com/datopian/portals-monorepo/tree/main/apps/portal-sse

Questions

• By this do you mean: they want to add content to their portal by publishing stuff "data rich" markdown docs? ✅2024-03-22 Yes
• Does this content use live viz? e.g. table or graph components loading data from their portal? ✅2024-03-22 ❌ not totally clear. requirements below need charts but not charts off live data
• Does this content live in a git(hub) repo?
• Is that repo separate from portal(js) repo or part of it?
• Do we have a written statement of what was promised ot the client? ✅2024-03-22 YEs, see below
• Is it part of the main website? ✅2024-03-22 yes.
• ASIDE: importing word documents would be a cool feature …
• Who can do development for the client?
• What is the timeline for this work? 🔑 the preferred timeline is 2-3 weeks from now

Client Requirements doc

• Creation of a separate Section of the data portal site in which to be published and maintained:
  • Data Roadmap of data to published
  • Ability to update once every x weeks
  • List of triage / responses received with status update
  • Ability to update once every x weeks
  • List of completed accepted Data Triage Responses
  • List of completed denied Data Triage Responses

Response:

• We will integrate a solution for content curation and publication. We will support data-rich components so that editors can:
  • Create and update the roadmap component.
  • Create tables with a list of triage/responses.
  • Create tables with statuses etc.

5 days

Notes

• Clients from last summer
• Very happy with us
• BigQuery for smart meter data …
• Now, they want to create more content. Atm only one page "Data roadmap" but they want to be able to write data-driven articles
• Fully open to use github and markdown and whatever we propose
• They want more components for portaljs