DataHub Cloud functional tests

Summary

Situation: Lack of automated tests in the DataHub Cloud. We only have the staging environment used to test changes manually before merging to production. Problem: Absence of tests complicates the process of making changes, increasing the chances of issues. Appetite: 3 days Solution: Implement testing, only functional tests for now, to ensure core functionalities are working as expected.

Situation

• We have launched the first version of the app and we're currently onboarding some clients.
• We make a lot of changes to the app.
• We have staging and production environments (staging is for testing before pushing to production).
• We test changes manually.
• We don't have any automated tests implemented.

Problem

• Having no automated tests makes introducing new features to the site (as well as making other minor changes and fixing bugs) very error-prone, as changes made to one part of the app could potentially break another part, and currently there is no way to catch it without manually testing both.
• With manual testing only it would be extremely hard (or close to impossible) to cover all functionalities each time we make some changes to the code, and it's obviously time-consuming.

Appetite

• 3 days

Solution

• We should focus on things that:
  • Break core functionality.
  • Can affect many users.
• Set up automated functional tests with Playwright.
• Create a suite of tests for the current set of functionalities.
• Start utilizing Test-Driven Development approach from now on.

What to test

• Login functionality
  • Redirecting to login (from any page) if not logged in
  • Redirected to the dashboard (from /login) if already logged in
• Site creation
  • Dashboard shows the placeholder/message image if no sites were created
  • User can create a new site
  • The newly created site is added to the dashboard
  • Clicking on the site card redirects to the settings page for this site
  • Clicking on the link inside the site card redirects to the home page for that site (and it renders correctly = with no errors)
• Site settings
  • The site has correct repo URL and deployed URL
  • User can rename website
  • User can update branch
    • Dashboard shows not synced
    • User should be able to sync
  • User can delete website
  • Dashboard shows not synced after updating the GitHub repo ( Really think this shouldn't be implemented )
  • User can sync
• Published sites
  • datapackage.json/yml/yaml files (or datapackage in frontmatter) will be picked up.
  • Test rendering of dataset layout for README.md/index.md to ensure all necessary information is displayed.
  • Test rendering of story layout for pages that doesn't have datasets to ensure proper display of content.
  • User can use data-rich components
  • Description is extracted
  • Title is extracted correctly
  • Paths work as expected
    • Sub-directories work correctly ( one nested file )
    • / displays Readme.md or index.md

Rabbit holes

No-goes

• Don't try to fix issues discovered during testing but rather open a new issue.
• We don't want unit tests for now.
• Testing custom domains.
• Testing authentication.
• Testing markdown syntax rendering.
• Custom domains work (won't test for now as it's complicated)
• MD syntax (headers, tables, callouts, etc.) (won't test as it's already have tests of their own)
• Running the functional tests as a GitHub workflow