Standup

Present: Joao, Rufus, [Ola at appointment]

Updates:

• João
  • Fixed issue on the Data Tables improvements PR (~2h spent)
  • I'm working on https://github.com/datopian/datahub-next/issues/31, just did some investigations for now (~1h)
  • Would like to finish the Graph components ticket https://github.com/datopian/datahub-next/issues/17 asap as well

Issue 31

• First, I took a look at the CLI tool (md to sqlite) that was mentioned in the ticket, and also at the next-contentlayer package (was curious to understand how it was triggering the reindex on reload/build)
• Then I started sketching how to lib/markdowndb.ts file should look like e.g how could we have global configs for that so tha we don't have to always pass the folderPath and dbPath to the functions (that's why next-contentlayer could be useful)
• Also took a look into the sqlite3 npm package
• Spent only ~1h on this up until now

indexFolder(folderPath, sqliteDb)

interface Database {

  getFileInfo()

  getTags
  
  query(query: DatabaseQuery)
}

Test:

function testBuildingADatabase() {
  const db = new sqlite3.Database(':memory:');
  const pathToFixturesFolder = 'tests/fixtures/markdowndb/'
  markdowndb.indexFolder(pathToFixturesFolder, db);
  
  // there are files in fixtures, check they were created
  // not needed for now, let's work with raw db
  // const ourdb = markdowndb.Database(db)
  
  // ensure there is a "files" table
  // select anything from the table 

  // check 3 files in the table
  // db.select ...
  
}

function indexFolder(..) {
  // create db structure (for now we can assume db is empty - later we can be more sophisticated)
  
  // walk folder
  
  // grab markdown file
  
  // extract frontmatter with gray-matter
  
  // store in db according to structure we have design
}

Actions

• Pick a basic sqlite library eg.
  • https://github.com/WiseLibs/better-sqlite3 - look preferably to standard basic one
  • https://www.npmjs.com/package/sqlite3 - standard basic one
  • https://github.com/knex/knex - typescript example looks pretty neat. however may be more than we need.