This is mainly about Phase II (May) and Phase III (June)
There were two phases in this shaping. The shaping here is mainly about phase II which addresses issues with our solution in phase I. However, ultimately we provide a consolidated version because that is what is useful unless you are in the details.
General version (incorporating phase I / II)
WarningThis is in progress and will evolve as we learn from our iterations.
TODO:
- incorporate material from here 2024-05-03-hypothesis-tree-for-pitch-2422
- incorporate material from 1st pass on workflow design in ../README#Workflow
Context: high level ShapeUp workflow
The following is the high-level workflow we are following:
- Inbox of Raw Ideas/Bugs: Bugs, ideas etc get captured (somewhere)
- Some ideas: are selected for shaping
- Bugs just get logged unless super important/urgent.
- Bugs are worked on when we have downtime
- Some shaped ideas are selected ("bet on") and implemented
More detailed version:
Workflow notes
- Raw inbox …
Needs / Job stories
- Capture idea/bug: i have an idea, found a bug etc … what do i do?
- => 📥 Inbox for dumping: Have an inbox where stuff gets dumped (preferably without becoming detailed issues … but that is probably unavoidable)
- Shaping: I want to know what to shape next. what is being shaped. what is shaped. what is bet on.
- Shipping: I want to see what to work on next as the lead developer?
- Overview of backlog I am the product lead / CEO i want to see what is in the backlog and what progress we have made …
- Show the Roadmap: at the big picture level in terms of features show what is planned and done.
TODO: for each job story. idea about solution. current solution. new solution.
Solution
TODO: merge up solution from phase II with any carry-over info from phase I.
- Phases: Raw, Shape, Bet on, Share
FAQs
Phase III: June 2024, adding phases
UPDATE: 2024-06-26, looking at this we can merge Raw and Ideas as "Raw". NB Ideas Backlog does not even exist in ShapeUp.
Introduce concept of phases distinct from status.
Phases correspond to the major parts of the life cycle of a feature/bug/task. They relate directly to the ShapeUp cycle concepts. Currently, we have identified five phases:
- Inbox: ie. new stuff that has come in (plus basic analysis of those items)
- Shaping: items selected for shaping and in progress for that
- Bet on: what is bet on and will be implemented
- Share: shipped and ready to share …
Situation
- we use a single field
status
and its values for all stages of lifecycle of an item - we identify raw inbox by stuff having "new" tag
- have introduced "roadmap" field to indicate stuff that is higher priority or somehow "on the roadmap"
Problem with current system
- 🔥 No way to see shaped backlog no way to see shaped items that are not yet bet on but which are analysed (either discarded or "later")
- 🔥 Lack clear ideas inbox no clear ideas inbox: no way to really see what is taken out of new but not yet ready for shaping
- New status is a bad smell
🆕 New
status field only really relevant for one phase and not needed in all other stages. This leads to it constantly having to be "hidden" on every board - Can't distinguish someday across phases I can't see difference in e.g. 💤 someday between something shaped and a raw inbox item that was discarded in that way
- (minor) roadmap yes/no is being abused just to make things visible day to day
- Done category is unclear and we don't have a clear share stage Atm we have the
✅ Done
category. Want to distinguish done in various ways. However, if we add these as status we have a lot of noise …:- Feature is "done" and ready to deploy
- Deployed / shipped i.e. live on site
- Announced / publicized e.g. in discord, in a blog post (if the feature merits it)
- Shaping status is unclear Can't tell progress in important "shaping" phase as currently that is designated by backlog - so can't see what shaping is in progress
Job stories
Motivated by these questions / needs / job stories:
- What is raw backlog (ie. new stuff that has come in) … so that i can review incoming things and make sure things are not missed and move them to the next step
- What is in ideas backlog (i.e. analysed and prioritised to some extent) so that this can be reviewed regularly
- What is being shaped: so that i can see progress and see things that were shaped but not yet bet on
- What is shaped and discarded or not yet bet on (and status of that shaping)
- What is bet on: so we can track implementation
Hypothesis
- Introduce phase concept with 5 phases
- Inbox
- Shape
- Bet on
- Share
- Update status field in following way:
- remove 🆕 New (and New becomes: raw + backlog)
- Remove Discussion (?) (Discussion becomes: Ideas + backlog or similar)
- New workflow pattern: if something is new and does not get prioritized we close as wontfix
- New share phase allows to clarify meaning of Done in Bet On
Phase | Backlog | In Progress | Done |
---|---|---|---|
Inbox | Just arrived | Being analysed | Analysis complete |
Shape | Selected for shaping | Shaping in progress | Shaping complete |
Bet on | Selected for implementation | In progress | Shipped (and live) |
Share | Live & ready to share | Sharing in progress | Sharing complete |
How this addresses the problems
- 🔥 No way to see shaped backlog no way to see shaped items that are not yet bet on but which are analysed (either discarded or "later") Use shape phase
- 🔥 Lack clear ideas inbox no clear ideas inbox: no way to really see what is taken out of new but not yet ready for shaping Use ideas phase
- New status is a bad smell
🆕 New
status field only really relevant for one phase and not needed in all other stages. This leads to it constantly having to be "hidden" on every board 🆕 New is removed as part of status - Can't distinguish someday across phases I can't see difference in e.g. 💤 someday between something shaped and a raw inbox item that was discarded in that way Have phases now
- (minor) roadmap yes/no is being abused just to make things visible day to day
- Done category is unclear and we don't have a clear share stage Atm we have the
✅ Done
category. Want to distinguish done in various ways. However, if we add these as status we have a lot of noise …: New share category plus clarity meaning of Done for bet on- Feature is "done" and ready to deploy
- Deployed / shipped i.e. live on site
- Announced / publicized e.g. in discord, in a blog post (if the feature merits it)
- Shaping status is unclear Can't tell progress in important "shaping" phase as currently that is designated by backlog - so can't see what shaping is in progress Having shape phase
Appendix: analysing status stages across phases
We could guess that there's basically:
- Backlog
- In progress
- Done
Plus minor ones
- (?) Ready / Next
- (?) Discussion
- Blocked
- (?) Paused
- Someday
Raw stages
- New
- Someday / Discarded (deleted? do we really want stuff this raw kept around)
- Worth further interest => go to ideas backlog
- Prioritized => moved in ideas backlog
- Further discussion
In ideas backlog
- "New" (not yet really processed here) = Backlog status
- Discussion = in progress or just backlog
- Evaluated e.g. has priority, has labels etc = Done
- Selected for shaping … = moved to next phase
In shaping
- Backlog = selected for shaping but work not begun
- In progress …
- Done
- Paused = use Someday
Phase II: May 2024, using github projects
Summary
- Appetite: 1-2d
- Situation: working in a single inbox issue and people are supposed to create check-mark items there …
- Complication: does not work well
- Hypothesis: switch to github projects (and more)
Situation
- Current plan as per Phase I below:
- Manage inbox of ideas and shaping in a single "[inbox]" issue https://github.com/datopian/datahub-next/issues/246
- Managing shape backlog in tasklist in Inbox issue **in "Shaping section" https://github.com/datopian/datahub-next/issues/246
- Have prioritized Inbox section there
- **Manage backlog of work in a single issue using tasklist in current major ship epic https://github.com/datopian/datahub-next/issues/330
https://github.com/datopian/datahub-next/issues/276**
- Manage inbox of ideas and shaping in a single "[inbox]" issue https://github.com/datopian/datahub-next/issues/246
- What happens in reality
- People add issues to inbox because easy and fast. Don't triage them.
- Tend to triage in meetings
- We don't have any overall roadmap of what we have shipped over time in terms of major features because it is spread across several epics
- Can't tell where we are with pieces of work in the current ship cycle as just a task list with no progress indicator
- People add issues to inbox because easy and fast. Don't triage them.
Terminology: infrastructure and jobs and the centrality of jobs
- Let's distinguish two different kinds of things that we need to have as part of any successful workflow
- infrastructure eg. project boards, automation
- jobs/tasks that someone has to do e.g. review items in the inbox periodically
- This is important because we can end up too focused on infrastructure, and frankly if jobs aren't going to get done it doesn't matter what infrastructure we have the system won't function (it will revert to whatever does work in terms of jobs for people)
- infrastructure is (relatively) easy to think about and set up … so it tends to get more attention …
- But in fact it is jobs that are key – unless they actually work for people they won't get done
Problems
- Single Inbox issue with tasks does not work well because people just create issues rather than adding in inbox issues plus do so in 2 different trackers
- Not using inbox issue: People add stuff in issues b/c a) standard flow b) can add more info in an issue - adding stuff as single line items limits info you can add e.g. for bugs we often want screenshots etc.
- Post in two places: inbox issue is in private tracker so not available to public. plus posting in public issue tracker is sometimes better for issue affecting public app.
- Hard to see state: difficult to show state i.e. where we are with pieces of work. One solution is to add some kind of emoji and comment but this is cumbersome and seems to be duplicating exactly what kanban board etc could do. Plus can only show latest comment.
- Duplication re shaping backlog: Work backlog involves shaping work (so duplicate stuff in 2 places both in work backlog and shaping list)
- Jobs are under-described and not assigned in current phase I workflow
- Missing feature changelog + roadmap Not possible see what features we have shipped (or have planned)
On other hand …
- Attraction of single issue inbox is that everything is visible in one place, with high information density and you can easily move things around (you can drag and drop between task lists in github)
- Could use a table view in github projects for similar info density but it does not have drag and drop easily between status categories (kanban supports moving status but is poor in terms of info density)
- Project limitations in github: any given ordering e.g. of priority items applied on all boards.
Solution
- Issues in public issue tracker github.com/datopian/datahub/issues
- If something specially private can go in datahub-next
- Create a public projects board associated with public tracker https://github.com/orgs/datopian/projects/47
- Misc recommendations / changes
- Stop having shaping issues separate from issue itself. Just have shaping be the first acceptance item on an issue. (Why: less noise, makes sense, and allows us to have one issue through states workflow)
Inbox and ideas backlog workflow
- Create issues: Create issues in issue tracker (for really minor/misc items can use an inbox issue)
- 🎱 Optional Bonus: Use gitmoji.dev and labels to classify types of issue e.g. bug, feature etc
- List of new issues e.g. for review: Create a view just for "New" issues: https://github.com/orgs/datopian/projects/47/views/4
- 🔁 use a workflow to auto-assign status "New" to issues added to project
- 💼 Review issues: periodically review issues e.g. weekly or monthly
- Use gitmoji.dev and labels to classify types
- indicate priority
- Use labels or category in project? Probably category because then we can sort more easily by this?
- Relationship of status e.g.
💤 Someday
to priority
Showing main feature roadmap … including backlog of major features etc
- Have a Roadmap kanban or table https://github.com/orgs/datopian/projects/47/views/1
- Add "Roadmap" category to issues (NOT a label b/c you can't have labels on non-issue tasks)
- ❓ is this just for cloud or for everything? I would say focus on cloud for now …
Viewing and modifying current ship cycle
- Have an epic issue
- List items in it
- Use a board (kanban or table) filtered on
tracked_by
with the epic issue - Indicate status with status category and then hill info with special substatus (emojis within the issue?)
How to find stuff to work on …
- Look at roadmap and shape something significant
- Look at general triaged ideas for bugs or high value/cost things
Shaping backlog
- Look at
Backlog (to shape)
category in everything, especially stuff in Roadmap
Implementation backlog
- Items in
Ready (shaped)
especially in Roadmap
TODOs:
- How do we distinguish inbox items that have been triaged from those that are truly "new"? ✅2024-05-27 ❌ not crucial for now and we can work this out through experimentation and then document
Tasks
- Open a project board ✅2024-05-03 repurposed PortalJS for now to DataHub https://github.com/orgs/datopian/projects/47
- Set up Inbox
- Auto-add issues to the board with a certain status ✅2024-05-03 auto-add from datahub repo with status New
- Create table board to just show new issues ✅2024-05-03 "Raw Inbox" https://github.com/orgs/datopian/projects/47/views/4
- Set up roadmap ✅2024-05-27 https://github.com/orgs/datopian/projects/47/views/1
Guesses …
- Triaged inbox
- Some basic classification
- At least some items have a priority
- Includes bugs and feature requests
- Shape board
- Implementation board
- Shaped items + bugs etc
- Roadmap board: showing the big picture of progress
Questions:
- Do we implement shaping as a task in itself or just as stages?
- How does this cohere with understanding where work is going on?
- How do we manage the ship cycle?
- Do we use a milestone or an "super/epic" issue to track the ship cycle?
- Do we view in kanban or issue or …
- Do we have sub iterations of e.g. 2 weeks or not?
- What's important in implementation backlog is to see easy-ness also of the items (is this just size?)
- Conceptually for roadmap i just want to know where a feature or item is in overall process and separation of shaping and implementation is odd to me – i just see that as part of some overall phase.
- But at the fine-grained level
- ❓ Have someday space (or just delete stuff?)
- Re Roadmaps: shall we create "uber epic" for each major use case e.g. wiki, dataset publishing etc.
FAQs
- Does the public board show private issues to non-team (e.g. issues in datahub-next)? ✅2024-05-27 No it doesn't, issues etc are only shown if you have permission to view them 🎉
Kanban board for ideas and shaping process
Inbox | To shape | In shaping | Shaped - ready to bet on | Selected | Discards | Implemented? |
---|---|---|---|---|---|---|
How this address problem list
- Single Inbox issue with tasks does not work well ✅ we use issues in main public issue tracker and auto-add status New and create a list to view that.
- Hard to see state: TODO
- Duplication re shaping backlog: Work backlog involves shaping work (so duplicate stuff in 2 places both in work backlog and shaping list)
- Jobs are under-described and not assigned in current phase I workflow
- Missing feature changelog + roadmap Not possible see what features we have shipped (or have planned)
How does X get their job done?
What matters is that people can get their job done … and they have a limited set of places to look …
- Product Owner browsing issues: I'm Rufus/Daniela and i'm browsing the main issue tracker and i notice a new bug that seem important and i want to flag that … ❓ How do i do that?
- Developer sees backlog: I'm Ola and i want to see what i'm working on next
- Product owner see inbox items: I want to see triaged inbox
Rabbit holes
- Do we do just in the open in public tracker or in private ✅2024-05-03 do everything we can in public. our greatest enemy is not people taking the ideas but lack of interest etc. working in open makes it easier to coordinate, makes us more visible etc.
Misc other points
- we aren't doing shaping cycles properly
Notes
2024-04-08
Shall we get rid of kanban boards for now?
- i think table view with grouping per iteration is often more useful …
- Plus we don't really do standup usefully with it …
- And we have a small team …
- What really matters is having some kind of aspiration every 2 weeks or so … (or even for the bigger aims …)
Phase I: Fewer Inboxes - early April 2024
Summary
- Situation: we needed to plan larger iterations and for that want a backlog of ideas / issues
- Problem: we had 4+ inboxes spread across
INBOX
issues, feedback from onboarding and more (see Appendix list of inboxes from 2024-04-08 meeting). This reflected deeper issue that we weren't clear about what we capture, where and how we process that into a backlog. - Solution: new worfklow detailed below. Key points:
- Adopt ShapeUp rough model with an Ideas inbox, Shaping backlog and implementation
- New Single Inbox using a single issue with task items and sections: https://github.com/datopian/datahub-next/issues/246
- To Shape backlog? in the epic in "Shaping section" https://github.com/datopian/datahub-next/issues/276
- Current ship backlog goes into an epic for current major cycle e.g. https://github.com/datopian/datahub-next/issues/276
Hypothesis
Created/Updated: 2024-04-09
This is an outline of the workflow of our product team.
Job stories
- Capture idea/bug: i have an idea, found a bug etc … what do i do?
- Shaping: …
- Shipping: I want to see what to work on next as the lead developer?
- Overview of backlog I am the product lead / CEO i want to see what is in the backlog and what progress we have made …
General shape-up process
Our Process
- What's missing here?
- difficult to know where we are in terms shipping etc. ✅2024-04-09 suggest we adopt terminology and concepts of https://basecamp.com/shapeup/3.4-chapter-13#work-is-like-a-hill re "scopes" and "hill diagram". And don't formally track given how small we are, just ask (or maybe use some emojis …)
- At present, don't have a shaped backlog ahead of a ship cycle …
- difficult to know where we are in terms shipping etc. ✅2024-04-09 suggest we adopt terminology and concepts of https://basecamp.com/shapeup/3.4-chapter-13#work-is-like-a-hill re "scopes" and "hill diagram". And don't formally track given how small we are, just ask (or maybe use some emojis …)
- Who selects for shaping? Daniela with Rufus mentoring and consulting
- Who does shaping? Depends on item e.g. is it tech or not. Generally, Ola with Rufus support on tech.
- Who does betting? Daniela with Rufus mentoring and consulting people
- Pattern: Inboxes rather than issues
- Everything goes into an inbox by default (or even your own inbox!)
- inbox for immediate issues/bugs with current release
- inbox for "ideas"
- we do NOT open issues for ideas or bugs immediately unless we are going to fix them right now …
- only when something is pulled from inbox for shaping does it get an issue (at that point a shaping issue)
- May separate bugs from features (inside the inbox, but the basic rule is dump first, process later!)
- it is also ok to periodically just throw stuff out. If it is important it will come back …
- Everything goes into an inbox by default (or even your own inbox!)
- 🚩 can have a prioritized ideas subsection of inbox (to save having to wade through everything everytime …)
- Where are things right now?
- Inbox? https://github.com/datopian/datahub-next/issues/246
- To Shape backlog? in the epic in "Shaping section" https://github.com/datopian/datahub-next/issues/276
- Where is to ship backlog? in the epic in "Implementation section" https://github.com/datopian/datahub-next/issues/276
- Don't worry about forgetting things
- If you really need to, put it in an inbox
Appendix: list of inboxes from 2024-04-08 meeting
See https://github.com/datopian/datahub-next/issues/325
- We have this epic issue https://github.com/datopian/datahub-next/issues/276 ✅2024-04-08 did not review as up to date
- We have the datahub-next backlog https://github.com/datopian/datahub-next/issues ✅2024-04-08 see https://github.com/orgs/datopian/projects/20/views/20. Moved some stuff into inbox. Deduplicated issue about dev docs
- Project kanban last iteration: https://github.com/orgs/datopian/projects/20/views/8 ✅2024-04-09 Rufus/Ola
- We have this inbox of bugs and things to fix https://github.com/datopian/datahub-next/issues/246 ✅ Ola
- We have the products backlog https://github.com/datopian/product/issues ✅2024-04-08 Yoana
- We have the list of datahub issues ✅2024-04-08 Daniela https://github.com/datopian/datahub/issues ✅ Ola (only copied over stuff related to DataHub Cloud)
- We have some feedback from onboardings https://github.com/datopian/product/blob/main/notes/feedback-from-dh-cloud-onboarding.md ✅2024-04-08 Daniela
- We have some feedback from the hackathon https://docs.google.com/document/d/1Gtrd9zUDfp-IwL2CqXd3aoFjPb1ZenH7lkQvw6rWlv8/edit ✅2024-04-08 Rufus
- What is the desired outcome of this meeting: Consolidate everything
- Decide which of these will be our one and only source of truth ✅2024-04-09 resolved in new workflow https://github.com/datopian/product/blob/main/README.md#workflow
- Update it to include everything we have in our backlog ✅2024-04-09 we will update backlog
- Prioritize 🚧2024-04-09 in progress!