Better software through software architecture and devops

@jamessnape

Posts

  • Image of a white mug on a table with 'Everyone is entitled to my opinion' written on the side

    Teaser article for a series on how to do agile without starting from scratch.

    One of the core tenets of agile is continuous improvement and finding out what works for you. This comes from the principle that:

    At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly.

    For this reason I’m hesitant to provide a ready made process to folks. Saying “here, do all this and you will be agile” feels wrong. I’ve been the subject of one of these consulting led lean transformations so I know what its like to have a bunch of outside folks tell you how to operate.

    On the flip side I’ve been involved with agile projects since the start in 2001 and before that one of the precursors ‘rapid application development’ so I would say I have a lot of experience to share. I’ve worked in high performing teams and seen cases where agile has been misinterpreted and failing. I would like to share these experiences and resulting advice.

    Over the years I’ve developed an system of agile that works for me. My interpretations go beyond the scrum guide or agile principles. It continues to evolve as expected. Some of the articles on this site are going to describe this system, opinions on commonly accepted practices such as planning poker, and other suggestions.

    Whilst these work for me they may not work for you. There is no reason not to continue with your existing improvement process. If you read along you will find thoughts on Azure DevOps process models, team structure, giving value to epics, specific steps to running scrum ceremonies, tips like card styling and more.

    Feel free to borrow the ideas and make them your own.

    Photo by Steve Johnson on Unsplash

    This entry was posted in agile  and tagged #agile #opinion  on .
    Discuss this on Twitter or LinkedIn
  • "Queue for Coffee". You know they serve good coffee when you see this. A line of people queuing for coffee at a street vendor. Rusty's Market, Cairns Australia.

    Queues are used to load balance work between teams but its easy to overlook a decrease in efficiency they bring.

    Its fairly common knowledge that batching work causes inefficiencies. This is why a core technique for lean software delivery is map your value stream and then limit the amount of work in progress. I wanted to see just how much efficiency can be gained by removing queues so started to map this with an idea example.

    Traditional Batched delivery

    Imagine we have a fairly standard software delivery pipeline: specify, build, verify, and release. I’m being purposefully vague here as the actual work doesn’t matter. In a traditional waterfall style delivery you would generally work on the specify part first to create a big requirements document; chuck it over the wall to the next team to build; once they have done that the built code gets chucked at the next team to verify; and finally the finished product is released to live.

    It not that obvious but the requirements document, the built code and the release package are queues. They contain a batch of work for the next stage.

    Lets visualize this by creating an ideal delivery pipeline. Imagine we have 10 work items; all exactly the same size; so they all take exactly one day to complete at every stage. (I know this isn’t the case in reality but that would only makes things worse.) The delivery plan as work flows through the process is shown below:

    DayBacklogSpecifyBuildVerifyReleaseDone
    WaitingDoingDoneDoingDoneDoingDoneDoingDone
    110
    291
    3811
    4712
    5613
    6514
    7415
    8316
    9217
    10118
    1119
    1210
    1391
    14811
    15712
    16613
    17514
    18415
    19316
    20217
    21118
    2219
    2310
    2491
    25811
    26712
    27613
    28514
    29415
    30316
    31217
    32118
    3319
    3410
    3510
    3610
    Total101001010010551

    Each step in the above process has an active ‘Doing’ column and a waiting ‘Done’ queue. Since the output queue for one stage is the input queue for the next I leave the work items in the done queue until they can actively be worked on. This helps calculate the active time and waiting time metric for each process.

    Since the release stage doesn’t involve processing items at the individual level you can see the whole package is released in one go.

    If we add up the values in the columns and work out the active and waiting times and also calculate the average cycle time for this configuration we have a maximum work in progress of 10 and timings:

    Days
    Cycle Time29.5
    Active Time31
    Waiting Time255
    Total Time286
    Calendar Time36

    So what if we don’t batch the work into a document?

    Limiting WIP by Sequencing

    This time the only thing we will change is to pass work to the next queue as soon as we have completed it. Everything else stays the same.

    DayBacklogSpecifyBuildVerifyReleaseDone
    WaitingDoingDoneDoingDoneDoingDoneDoingDone
    110
    291
    3811
    47111
    561111
    6511111
    74111111
    83111112
    92111113
    101111114
    11111115
    1211116
    131117
    14118
    1519
    1610
    1710
    1810
    Total1010101010551

    Immediately we can see the whole thing is far shorter. Calculating the metrics for a WIP of 1 we have:

    Days
    Cycle Time11.5
    Active Time31
    Waiting Time75
    Total Time106
    Calendar Time18

    Note the active time is still 31 - its the same amount of effort as before but the whole thing is done in half the time because we removed 100 days of waiting time! Consequently the average cycle time has dropped by over half too. Remember nothing else changed apart from removing the batch.

    I could stop here but this is a good point to look at further optimization.

    Remove the Queues

    The next simplest optimization is to remove the queues between stages. This is harder in the real world due to uneven work size and arrival rate but we can look at the theoretical flow.

    DayBacklogSpecifyBuildVerifyReleaseReleaseDone
    WaitingDoingDoingDoingWaitingDoneDone
    110
    291
    3811
    47111
    561111
    651112
    741113
    831114
    921115
    1011116
    111117
    12118
    1319
    1410
    1510
    1610
    Total101010551

    The stats are better still - active time is still 31 days but there is only waiting at the end of the process because the release time still need to deliver a bunch of work into live in one go. Cycle time is only 10 days.

    Days
    Cycle Time10
    Active Time31
    Waiting Time55
    Total Time86
    Calendar Time16

    As this point the only optimization left is to ask if the release team can increase their deployment frequency. It could be possible since items are queuing at release.

    Increase Release frequency

    Whilst we could release every day like the very best DevOps practitioners lets assume there is a limit in the release process which means every other day is the best possible.

    DayBacklogSpecifyBuildVerifyReleaseReleaseDone
    WaitingDoingDoingDoingWaitingDoingDone
    110
    291
    3811
    47111
    561111
    651112
    7411112
    8311122
    9211112
    10111124
    1111112
    121126
    13112
    1428
    152
    1610
    Total101010155

    The stats for this configuration are interesting - for the first time the active time has gone up because we are asking the release team to do 4 more releases. Similarly, the calendar time remains the same because we are still limited by the time it takes to get the last work items released but there is far less waiting and the overall cycle time has dropped to just 5.5 days.

    Days
    Cycle Time5.5
    Active Time35
    Waiting Time15
    Total Time50
    Calendar Time16

    Summary

    This simple exercise demonstrates how hidden waiting time delays deliveries and smaller batch sizes can rectify that. In reality you will need some queuing between stages to balance out demand in this configuration. Look out for hidden queues:

    • Documents handed from one team to another - the entire document is a batch.
    • Pipelined iterations in scrum teams - where the requirements are written in the sprint before dev and the testing in the sprint after; the iteration backlog becomes the batch.
    • Done states between Doing states
    • Approval steps

    If you practice Scrum then aim to complete work in a sprint (specify, develop, and verify) or you are leaving waiting time on the table.

    Photo by David Clode on Unsplash

    This entry was posted in agile  and tagged #metrics #lean #devops  on .
    Discuss this on Twitter or LinkedIn
  • Code on a laptop screen with dark theme

    Metrics are only useful if they help you improve. Code coverage KPIs are most often circumvented. Liberal use of the ExcludeFromCodeCoverage attribute is to be avoided.

    Abuse of the ExcludeFromCodeCoverage attribute

    I once worked on project that had a mandatory code coverage target. If your commit didn’t maintain the overall coverage ratio of 75% then it was rejected. The reasoning came from good intentions; a high code coverage is good therefore we will create the mandate that is must be high. It had some unfortunate side effects though. The first unintended consequence is that developers only wrote enough tests to keep the value above the target instead of considering how they needed to test their code. The second consequence was the proliferation of [ExcludeFromCodeCoverage] attributes adoring all sorts of classes.

    The attribute was originally designed for generated code but more recently I’ve seen it be applied where the code that is hard to test or too simple to test. Ultimately this hides code from testing metrics. Your code coverage metric is no longer accurate nor useful.

    Yeah, we are at 85% code coverage ignoring the code we excluded because it was hard to test.

    My immediate response to this is “How come it isn’t 100% then?”

    I would much rather have a lower, but accurate, code coverage metric so I consider the use of this attribute harmful on any code that isn’t generated.

    I wish code analysis tools like SonarCube or the Roslyn analyzers treated this attribute like SuppressMessageAttribute - the Justification property should be filled out when applied. Better still, just fix the issue and avoid the need to use either of them.

    Why should I test properties?

    The consensus on Should you Unit Test simple properties? seems pretty much for testing and you can use the examples here to test yours. I prefer property based testing though because it can find edge cases you didn’t think of. There is a great intro at Property-Based Testing with C# using FsCheck. Your code will effectively look like:

    [Property]
    public Property Set_Then_Get_Returns_Same(string exampleValue)
    {
        var target = new ClassYouWantToTest();
        target.PropertyToTest = exampleValue;
        return (target.PropertyToTest == exampleValue).ToProperty();
    }
    

    FsCheck will generate a bunch of random values to try this test with so these 4 lines of code are resulting in hundreds of unique tests for this property including, for this example: blank string, null value, very short, very long, non-printing, accent characters, etc.

    Why should I test … something else?

    I will come back and add more examples as I encounter them.

    Summary

    • Code coverage metrics are only useful if they help improve the software quality.
    • Mandatory targets can lead to harmful practices such as excluding code from testing or writing superficial tests.
    • ExcludeFromCodeCoverage attributes should be treated in the same way as SuppressMessage - they hide warnings that should really be fixed.

    Photo by Luca Bravo on Unsplash

    This entry was posted in code  and tagged #metrics #testing #csharp  on .
    Discuss this on Twitter or LinkedIn
  • Red Arrows flying over Bournemouth Pier

    If you already know how to code in one language then using GitHub CoPilot Chat can be super helpful as a virtual colleague when you are working in new or unfamiliar territory.

    I have a little confession to make, this blog site hasn’t entirely been written by me. I mean I did a lot of the typing but I’ve had a little help from Github CoPilot.

    By trade I’m more on the backend side of things with C#, APIs and databases. I understand Typescript, HTML and CSS but its not natural for me to write. In the office I would find someone with complimentary skills to help out but this kind of personal project doesn’t have that luxury.

    GitHub CoPilot normally sits monitoring your code as you type offering completion suggestions which is do find a little annoying since they tend to pop up just as you are typing and then disappear. Maybe one day I’ll get use to it in the same way I have intellisense. Until then I far prefer the newer CoPilot Chat - its a tool window that sits next to your code and you chat to it like you would IM a colleague. The clever part is it can reference the code you are looking at.

    Initially I just created an empty folder and run git init to initialize a blank repository. Lets kick off with a starter to get things going.

    First step to creating an Astro site

    This doesn’t look right; the Astro Docs say its far simpler to call npm create astro@latest instead. They may be equivalent but for now I’m going to trust the documentation. In fact I got pretty far with the docs and their Build a Blog Tutorial. Whilst I was doing this I pretty much forgot about CoPilot Chat since the panel isn’t always visible. I did get stuck later on though - trying to complete the archive list on the right I couldn’t work out how to sort based on year and month. I figured it would be an idiomatic thing in JavaScript so asked:

    How to sort by year and month descending

    Notice how the code I was looking at has been used as a reference. Here is another example when I was building the monthly archive:

    How to map date to year and month

    I really like how the code comes with an explanation. This is teaching me as I work. The next challenge was to convert a month ordinal into a name for display.

    Public code block

    Now this is frustrating. It seems that since I get my CoPilot license through my work at Avanade, they have flipped a switch to ensure I don’t copy any public code and open them up to legal issues. I can see the rationale but I’m just trying to learn here. I did wonder if the typo in the question had reduced the set of results coming back and that’s why it thought I might be copying.

    Correcting the typo did work.

    Convert month ordinal to name

    This tool is super helpful when things aren’t working too. For example, first time I viewed the archive list there were duplicates in the month names. Psychic debugging tells me that there is probably a duplicate for every individual post in that month because the Set class wasn’t working as expected so I asked why.

    Why are there duplicate months

    Ah, in JavaScript, Dates are objects not values and compared by reference.

    Similarly I had a warning about a possible null value. In C# I would use a null coalescing operator ?? with a default value. I want to know the JavaScript equivalent and could have asked but thought it best to just ask for the outcome I wanted.

    Can draft default to false

    For the last example I have, I had used CoPilot to help me build a simple search form that would submit the query by opening a new window on Bing with the search as a query parameter. I also appended site:snape.me to restrict the results back to this blog. Hard coded would have worked but I knew I already had the site name in the config and it would be better if loaded from there.

    Lookup site in config

    This is really useful because this isn’t just about writing JavaScript, it tells me how to use the libraries I have included in the project. Alas it doesn’t work though but not for long:

    Why won't the form submit

    I used Co-Pilot to fix a lot of additional issues along the way but mostly boring TypeScript warnings so won’t inflict those on you.

    So would I recommend this approach?

    Absolutely, if you are a beginner or someone with skills in a different software area it can save a whole pile of time iterating whilst you find a solution. It will save you time looking up stuff on the web but just be aware that it might not always give you the best solution. It’s a tool not a replacement but a lot less annoying than tapping your coworkers on the shoulder to ask dumb questions. Don’t leave your brain at home though.

    This entry was posted in code  and tagged #astro #javascript #typescript #github-copilot #copilot-chat #gen-ai  on .
    Discuss this on Twitter or LinkedIn
  • Bournemouth Beach

    Converting code between Javascript technologies isn't simple but GPT/Copilots can do a lot of the heavy lifting.

    Its been nearly 10 years since I last posted an article on this site. Having children and other priorities pushed blogging lower down the priority stack. Plus the site used hosted on https://jamessnape.wordpress.com/ and getting embedded code to look good was an effort.

    My kids are older now and I’m interested in using this site to learn about new technologies; specifically Astro, Svelte, and Tailwind CSS.

    Once of the challenges I’ve had creating this new implementation is there are now a lot of privacy laws in place such as GDPR. Since I use Google Analytics I have to include a Privacy Policy and cookie consent pop-up 1.

    As a guide I followed Creating a Cookie Consent Banner for Google Tag Manager to add mine here. This example uses Solid JS for the implementation of the consent functionality and its not familiar to me so I turned to Bing Copilot for help.

    First step is to ask a general question “how to convert solidjs to svelte?

    To convert SolidJS to Svelte, you can follow the steps below:

    1. First, install the latest version of Svelte by running the following command in your terminal: npm install svelte@latest

    2. Next, create a new Svelte component file with the same name as your SolidJS component file. For example, if your SolidJS component file is named MyComponent.js, create a new file named MyComponent.svelte.

    3. Copy the contents of your SolidJS component file into your new Svelte component file.

    4. Replace any SolidJS-specific syntax with equivalent Svelte syntax. You can refer to the official Svelte documentation for more information on Svelte syntax.

    5. Finally, compile your Svelte component file into a JavaScript file that creates the DOM and CSS files that style it. You can use SvelteKit, the official application framework from the Svelte team, to handle this process. Run the following command in your terminal: npm run dev

    I hope this helps! Let me know if you have any other questions.

    This is a nice high level plan but the 4th point is a little light on detail so next question “expand on point 4 please”.

    Sure, I can explain more about point 4. Point 4 is about replacing any SolidJS-specific syntax with equivalent Svelte syntax. This means that you need to change the way you write your component logic, state, props, events, and other features that are different between SolidJS and Svelte.

    Here are some examples of how to convert some common SolidJS syntax to Svelte syntax:

    • Component logic: In SolidJS, you write your component logic inside a function that returns JSX. In Svelte, you write your component logic inside a <script> tag that can export variables and functions to the template. For example:
    // SolidJS
    import { createSignal } from "solid-js";
    
    function Counter() {
      const [count, setCount] = createSignal(0);
    
      function increment() {
          setCount(count() + 1);
      }
    
      return (
        <div>
          <p>Count: {count()}</p>
          <button onClick={increment}>+1</button>
        </div>
      );
    }
    

    HTML

    <!-- Svelte -->
    <script>
      let count = 0;
    
      function increment() {
        count++;
      }
    </script>
    
    <div>
      <p>Count: {count}</p>
      <button on:click={increment}>+1</button>
    </div>
    
    • State: In SolidJS, you use createSignal to create reactive state variables that are updated by calling setter functions. In Svelte, you use let to declare reactive state variables that are updated by assigning new values. For example:
    // SolidJS
    import { createSignal } from "solid-js";
    
    function App() {
      const [name, setName] = createSignal("World");
    
      function handleChange(e) {
        setName(e.target.value);
      }
    
      return (
        <div>
          <h1>Hello {name()}!</h1>
          <input type="text" value={name()} onInput={handleChange} />
        </div>
      );
    }
    

    HTML

    <!-- Svelte -->
    <script>
      let name = "World";
    
      function handleChange(e) {
        name = e.target.value;
      }
    </script>
    
    <div>
      <h1>Hello {name}!</h1>
      <input type="text" bind:value={name} on:input={handleChange} />
    </div>
    
    • Props: In SolidJS, you use props to access the properties passed to your component from the parent component. In Svelte, you use export to declare the properties that your component expects from the parent component. For example:
    // SolidJS
    function Greeting(props) {
      return <p>Hello, {props.name}!</p>;
    }
    

    HTML

    <!-- Svelte -->
    <script>
      export let name;
    </script>
    
    <p>Hello, {name}!</p>
    
    • Events: In SolidJS, you use onClick, onInput, onChange, and other event handlers to listen to user interactions on your elements. In Svelte, you use on:click, on:input, on:change, and other event listeners to listen to user interactions on your elements. For example:
    // SolidJS
    function Button(props) {
      return <button onClick={props.onClick}>{props.label}</button>;
    }
    

    HTML

    <!-- Svelte -->
    <script>
      export let onClick;
      export let label;
    </script>
    
    <button on:click={onClick}>{label}</button>
    

    These are just some of the syntax differences between SolidJS and Svelte. You can find more information on Svelte syntax in the [official Svelte documentation]. I hope this helps! Let me know if you have any other questions.

    There is quite a lot of specific information in this response to I started to convert the code but didn’t understand the following:

    createEffect(() => {
        setIsMounted(true);
        // get cookie approval after component is mounted
        setCookies(getCookieConsent());
      });
    

    …so another question: “Convert the following solidjs code to svelte: plus the code above”. The response is both accurate and uncannily expands the example to include the complete code.

    To convert the following SolidJS code to Svelte, you can use the onMount lifecycle function from Svelte, which runs after the component is first rendered to the DOM. For example:

    // SolidJS
    import { createEffect } from "solid-js";
    
    function App() {
      const [isMounted, setIsMounted] = createSignal(false);
      const [cookies, setCookies] = createSignal(null);
    
      createEffect(() => {
        setIsMounted(true);
        // get cookie approval after component is mounted
        setCookies(getCookieConsent());
      });
    
      return (
        <div>
          <p>Is mounted: {isMounted() ? "Yes" : "No"}</p>
          <p>Cookies: {cookies()}</p>
        </div>
      );
    }
    

    HTML

    <!-- Svelte -->
    <script>
      import { onMount } from "svelte";
      let isMounted = false;
      let cookies = null;
    
      onMount(() => {
        isMounted = true;
        // get cookie approval after component is mounted
        cookies = getCookieConsent();
      });
    </script>
    
    <div>
      <p>Is mounted: {isMounted ? "Yes" : "No"}</p>
      <p>Cookies: {cookies}</p>
    </div>
    

    I hope this helps! Let me know if you have any other questions.

    This got me to the stage where I could convert the typescript part of of the code. Only the markup to go but some of it isn’t plain HTML, specifically the <Show> element. So I asked “What is the equivalent svelte construct to a solidjs <Show> element?“.

    The final response was all I needed.

    The equivalent Svelte construct to a SolidJS <Show> element is the {#if ...} block. The <Show> element in SolidJS is used to conditionally render a component based on a boolean expression. The {#if ...} block in Svelte does the same thing, but with a different syntax. For example:

    // SolidJS
    import { Show } from "solid-js/web";
    
    function App() {
      const [show, setShow] = createSignal(true);
    
      function toggle() {
        setShow(!show());
      }
    
      return (
        <div>
          <button onClick={toggle}>Toggle</button>
          <Show when={show()}>
            <p>This is shown when show is true</p>
          </Show>
        </div>
      );
    }
    

    HTML

    <!-- Svelte -->
    <script>
      let show = true;
    
      function toggle() {
        show = !show;
      }
    </script>
    
    <div>
      <button on:click={toggle}>Toggle</button>
      {#if show}
        <p>This is shown when show is true</p>
      {/if}
    </div>
    

    You can ask to convert the code in one go but I like being able to learn along the way. Asking GPT to break a problem down usually results in a better response.

    Final code

    The final code for the converted CookieConsent.svelte component looks like:

    <script lang="ts">
        declare function consentGranted(): void;
        declare function getCookieConsent(): string;
    
        import { onMount } from "svelte";
    
        let cookies = "unk";
        let isMounted = false;
        export let CookieConsent;
    
        const handleAccept = () => {
            cookies = "granted";
    
            // accepted cookie lasts for a year
            let d = new Date();
            let oneYear = new Date(d.getFullYear() + 1, d.getMonth(), d.getDate());
    
            document.cookie = "cookie-consent=granted; expires=" + oneYear + "; path=/";
            
            consentGranted();
        };
    
        const handleDecline = () => {
            cookies = "denied";
        
            // declined cookie only lasts for the session
            document.cookie = "cookie-consent=denied; path=/";
        };
    
      // this waits to load the cookie banner until the component is mounted
      // so that there is not a component flash
      onMount(() => {
            isMounted = true;
            // get cookie approval after component is mounted
            cookies = getCookieConsent();
        });
    
    </script>
    
    <!-- if there is no cookie for "cookie-consent", display the banner -->
    {#if isMounted}
        <div
        id="cookie-banner"
        class={`${
            cookies === "granted" || cookies === "denied" ? "hidden" : ""
        } fixed bottom-0 right-0 z-50 m-2 max-w-screen-sm rounded-lg border-2 border-slate-300 bg-purple-50 text-slate-800 shadow-xl`}
        >
        <div class="p-4 text-center">
            <p class="mb-4 text-sm sm:text-base">
            We use cookies to analyze our website and make your experience even
            better. To learn more, see our{" "}
            <a
                class="text-blue-600 underline hover:text-blue-700"
                href="/privacy/"
                target="_blank"
            >
                Privacy Policy.
            </a>
            </p>
    
            <div class="mx-auto">
            <button
                class="rounded-md bg-blue-600 p-2 text-white transition hover:bg-blue-700"
                on:click={handleAccept}
            >
                Accept
            </button>
            <button
                class="ml-2 rounded-md bg-transparent p-2 text-slate-600 transition hover:bg-gray-200"
                on:click={handleDecline}
            >
                Decline
            </button>
            </div>
        </div>
        </div>
    {/if}
    

    Footnotes

    1. If you didn’t see the popup then you may have an automatic blocker like Ghostery enabled.

    This entry was posted in code  and tagged #solidjs #svelte #typescript #gen-ai  on .
    Discuss this on Twitter or LinkedIn