Better software through software architecture and devops

@jamessnape

Category Archives: agile

  • Image of a white mug on a table with 'Everyone is entitled to my opinion' written on the side

    Teaser article for a series on how to do agile without starting from scratch.

    One of the core tenets of agile is continuous improvement and finding out what works for you. This comes from the principle that:

    At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly.

    For this reason I’m hesitant to provide a ready made process to folks. Saying “here, do all this and you will be agile” feels wrong. I’ve been the subject of one of these consulting led lean transformations so I know what its like to have a bunch of outside folks tell you how to operate.

    On the flip side I’ve been involved with agile projects since the start in 2001 and before that one of the precursors ‘rapid application development’ so I would say I have a lot of experience to share. I’ve worked in high performing teams and seen cases where agile has been misinterpreted and failing. I would like to share these experiences and resulting advice.

    Over the years I’ve developed an system of agile that works for me. My interpretations go beyond the scrum guide or agile principles. It continues to evolve as expected. Some of the articles on this site are going to describe this system, opinions on commonly accepted practices such as planning poker, and other suggestions.

    Whilst these work for me they may not work for you. There is no reason not to continue with your existing improvement process. If you read along you will find thoughts on Azure DevOps process models, team structure, giving value to epics, specific steps to running scrum ceremonies, tips like card styling and more.

    Feel free to borrow the ideas and make them your own.

    Photo by Steve Johnson on Unsplash

    This entry was posted in agile  and tagged #agile #opinion  on .
    Discuss this on Twitter or LinkedIn
  • "Queue for Coffee". You know they serve good coffee when you see this. A line of people queuing for coffee at a street vendor. Rusty's Market, Cairns Australia.

    Queues are used to load balance work between teams but its easy to overlook a decrease in efficiency they bring.

    Its fairly common knowledge that batching work causes inefficiencies. This is why a core technique for lean software delivery is map your value stream and then limit the amount of work in progress. I wanted to see just how much efficiency can be gained by removing queues so started to map this with an idea example.

    Traditional Batched delivery

    Imagine we have a fairly standard software delivery pipeline: specify, build, verify, and release. I’m being purposefully vague here as the actual work doesn’t matter. In a traditional waterfall style delivery you would generally work on the specify part first to create a big requirements document; chuck it over the wall to the next team to build; once they have done that the built code gets chucked at the next team to verify; and finally the finished product is released to live.

    It not that obvious but the requirements document, the built code and the release package are queues. They contain a batch of work for the next stage.

    Lets visualize this by creating an ideal delivery pipeline. Imagine we have 10 work items; all exactly the same size; so they all take exactly one day to complete at every stage. (I know this isn’t the case in reality but that would only makes things worse.) The delivery plan as work flows through the process is shown below:

    DayBacklogSpecifyBuildVerifyReleaseDone
    WaitingDoingDoneDoingDoneDoingDoneDoingDone
    110
    291
    3811
    4712
    5613
    6514
    7415
    8316
    9217
    10118
    1119
    1210
    1391
    14811
    15712
    16613
    17514
    18415
    19316
    20217
    21118
    2219
    2310
    2491
    25811
    26712
    27613
    28514
    29415
    30316
    31217
    32118
    3319
    3410
    3510
    3610
    Total101001010010551

    Each step in the above process has an active ‘Doing’ column and a waiting ‘Done’ queue. Since the output queue for one stage is the input queue for the next I leave the work items in the done queue until they can actively be worked on. This helps calculate the active time and waiting time metric for each process.

    Since the release stage doesn’t involve processing items at the individual level you can see the whole package is released in one go.

    If we add up the values in the columns and work out the active and waiting times and also calculate the average cycle time for this configuration we have a maximum work in progress of 10 and timings:

    Days
    Cycle Time29.5
    Active Time31
    Waiting Time255
    Total Time286
    Calendar Time36

    So what if we don’t batch the work into a document?

    Limiting WIP by Sequencing

    This time the only thing we will change is to pass work to the next queue as soon as we have completed it. Everything else stays the same.

    DayBacklogSpecifyBuildVerifyReleaseDone
    WaitingDoingDoneDoingDoneDoingDoneDoingDone
    110
    291
    3811
    47111
    561111
    6511111
    74111111
    83111112
    92111113
    101111114
    11111115
    1211116
    131117
    14118
    1519
    1610
    1710
    1810
    Total1010101010551

    Immediately we can see the whole thing is far shorter. Calculating the metrics for a WIP of 1 we have:

    Days
    Cycle Time11.5
    Active Time31
    Waiting Time75
    Total Time106
    Calendar Time18

    Note the active time is still 31 - its the same amount of effort as before but the whole thing is done in half the time because we removed 100 days of waiting time! Consequently the average cycle time has dropped by over half too. Remember nothing else changed apart from removing the batch.

    I could stop here but this is a good point to look at further optimization.

    Remove the Queues

    The next simplest optimization is to remove the queues between stages. This is harder in the real world due to uneven work size and arrival rate but we can look at the theoretical flow.

    DayBacklogSpecifyBuildVerifyReleaseReleaseDone
    WaitingDoingDoingDoingWaitingDoneDone
    110
    291
    3811
    47111
    561111
    651112
    741113
    831114
    921115
    1011116
    111117
    12118
    1319
    1410
    1510
    1610
    Total101010551

    The stats are better still - active time is still 31 days but there is only waiting at the end of the process because the release time still need to deliver a bunch of work into live in one go. Cycle time is only 10 days.

    Days
    Cycle Time10
    Active Time31
    Waiting Time55
    Total Time86
    Calendar Time16

    As this point the only optimization left is to ask if the release team can increase their deployment frequency. It could be possible since items are queuing at release.

    Increase Release frequency

    Whilst we could release every day like the very best DevOps practitioners lets assume there is a limit in the release process which means every other day is the best possible.

    DayBacklogSpecifyBuildVerifyReleaseReleaseDone
    WaitingDoingDoingDoingWaitingDoingDone
    110
    291
    3811
    47111
    561111
    651112
    7411112
    8311122
    9211112
    10111124
    1111112
    121126
    13112
    1428
    152
    1610
    Total101010155

    The stats for this configuration are interesting - for the first time the active time has gone up because we are asking the release team to do 4 more releases. Similarly, the calendar time remains the same because we are still limited by the time it takes to get the last work items released but there is far less waiting and the overall cycle time has dropped to just 5.5 days.

    Days
    Cycle Time5.5
    Active Time35
    Waiting Time15
    Total Time50
    Calendar Time16

    Summary

    This simple exercise demonstrates how hidden waiting time delays deliveries and smaller batch sizes can rectify that. In reality you will need some queuing between stages to balance out demand in this configuration. Look out for hidden queues:

    • Documents handed from one team to another - the entire document is a batch.
    • Pipelined iterations in scrum teams - where the requirements are written in the sprint before dev and the testing in the sprint after; the iteration backlog becomes the batch.
    • Done states between Doing states
    • Approval steps

    If you practice Scrum then aim to complete work in a sprint (specify, develop, and verify) or you are leaving waiting time on the table.

    Photo by David Clode on Unsplash

    This entry was posted in agile  and tagged #metrics #lean #devops  on .
    Discuss this on Twitter or LinkedIn
  • Woman placing sticky notes on wall.

    Right now I’m reading “Agile Data Warehousing Project Management: Business Intelligence Systems Using Scrum” which I can heartily recommend. This article is about one quote in particular:

    Unfortunately, tracking actual hours spent on the individual tasks is ineffective, aims at the wrong objectives, and drowns the developers in detail while undermining agile’s pillar of self-organization that is so crucial to a team’s velocity. Tracking actuals should be strongly discouraged.

    I would go one step further and say tracking actuals can end up having a negative effect on your scrum project. Before I get into why its worth pointing out that for short periods of time it can be useful particularly if you need to gather data on where you are spending your time (see the second point below). Once you have the data though - stop recording.

    So why is tracking actuals so counterproductive? Primarily because it subverts the iterative estimation method.

    One of the key activities in a scrum process is to update the remaining time estimate every day. From this estimate a burn down is calculated and the burn down is then used to monitor and adjust the sprint in order to deliver.

    Consider what happens when scrum team members are logging the amount of time they worked on a task. Simple maths tells the remaining time is the current estimate minus the hours worked so there is a temptation to skip the re-estimating step. As a result a sprint burns down at a constant rate (ignoring task completion as a backup progress indicator) and inevitably it is not until the end of the sprint that underestimation is discovered; too late to do anything about it.

    This second point can be good or bad depending on management and team culture - if you log time correctly and in detail you will know where you are spending time including interruptions, meetings, etc. There are two types of response from the team:

    1. Look at all this waste activity we are doing - how can we remove it and spend more time on value add work?
    2. How can I possibly deliver on my sprint commitments - look at the other work I had to do during the sprint.

    Obviously you want the first response.

    Finally, overshooting sprints is something to be avoided but often there are additional constraints on time logging that cause you to miss estimate future sprints as well. I know of many organisations where team members are required to log a fixed number of hours a day; no more, no less. You can imagine the havoc this causes - if you work 5 or 15 hours during one day on a task but have to log 8 then you create an inaccurate data set from which to base future sprint planning on.

    Photo by Jason Goodman on Unsplash

    This entry was posted in agile  and tagged #estimation #scrum  on .
    Discuss this on Twitter or LinkedIn
  • Southbourne Beach

    How would you change your behaviour if the person reviewing your code had the final say on whether it makes it into the source repository?

    I often see code reviews done in principle but not practice. The workflow goes something like this:

    • Developer assigned a new feature
    • Developer designs and codes up the feature
    • Developer checks the code into source control
    • A reviewer is found/assigned
    • The reviewer reviews code and finds a bunch of issues - a lot to do with the design
    • The developer fixes a few of the easy code related issues
    • Time is running out so the remaining issues are left
    • Developer moves to the next feature

    Contrast this with the usual open source workflow where the reviewer is the committer and not the developer; the reviewer is responsible for the final code quality and it easy for them to refuse your code.

    If you were the developer in this situation what would you do? Would you discuss your design with the reviewer before you started work? Would you keep the reviewer updated with your progress and any decisions you made? Would you make sure there were no surprises for the reviewer when they finally saw your code?

    If you did then the reviewer feedback should be minimal and relatively easy to fix, more importantly code quality is maintained and without an expensive rewrite.

    This entry was posted in agile  and tagged #code-quality #code-reviews  on .
    Discuss this on Twitter or LinkedIn