When I talk to people about DevOps, Metrics are a frequent topic of discussion.

For many vendors of DevOps-related software this is another opportunity to throw their marketing machine on and promote a product as the “solution to all DevOps reporting problems”.

While the products available on the market are – in many cases – mature and up to the job; what everyone forgets about is that the data you get out is only as good as the data you put in.

Let me explain.

Get an idea of what changes are “worth it”

In recent years many large software companies have started including ways of collecting large amounts of telemetry into their software. In many cases this caused a backlash with their customer base. (just think about the Windows 10 launch in Europe)

The reason behind this is that many features in software products today just are not used, or are used by a tiny minority of the customer base. Telemetry helps us recognise what features we should invest in and what we can drop.

It is also very important that we collect the right telemetry. Staying on the Windows topic, think back to the introduction of Windows 8, where the start button was removed based on usage data. It certainly looks like the wrong subset of data was taken into consideration here.

Experimentation and Feedback are key

Staying on the Windows 8 example; the missing start button would not have been a problem, if it had been easy to patch the Windows 7 start menu back in, or to hide the old start menu behind a toggle.

If you can build your product in a way where you can experiment easily and collect customer feedback and usage data, then do it.

If you are working with legacy software, you may find it harder to implement ways for a effective experimentation, but you can still look at multi version deployments or feature toggles.

I will not go into deployment options in great detail in this post, but the way you deploy your application can definitely impede or encourage experimentation, and it’s worth considering that for architecture, re-architecture, and modernisation decisions.

Always deliver value

DevOps at its core is about quick value delivery to your consumer base.

Value is only realised once a product arrives at the consumer and a feedback loop can be established.

Your metrics should reflect this idea of a continuous value stream.

I once had a very awkward meeting with a test manager in a public sector company who asked me about how they should measure the value of their test division. What I was trying to put across in the meeting was that good testing and effective demotion of tests will eventually produce better software and that one should look at the entire cycle rather than a section of the process on its own.

But as the meeting was moving along, I noticed that the parts of the organisation were so separate that my ideas could not be understood.

When it comes to testing specifically, DevOps should be an enabler to collect more feedback in the field, rather than from a test team that works through defined scenarios. This does seem like an odd idea at first, but if we have secured all standard scenarios with automated tests, then there is no better place to test further than production. Yes, there is exploratory testing, and it is important, but what generates more value is the ability to quickly detect and respond to problems.

Metrics that work well in many organisations that embrace DevOps are: (in my opinion)

  • Mean Time to Discovery
    • ..and % incidents discovered before customer report
  • Mean Time to Resolution
  • Test Coverage
    • …but only when covering the right code and covering key scenarios
  • Cycle Time (from when the work started) and Lead Time (how long it is going to be for a change to happen)
    • …for the entire value stream, not just the development, test, or QA components

Be loud about it

If you want to embrace DevOps as an organisation you also need to embrace sharing.

Make sure your teams have a reason to talk to each other. There are many ways to achieve this:

  • Cross team brown bag
  • Monthly hackathon
  • Competitive sprint retrospectives
    • e.g. best sprint presentation medal
  • Weekly production incident drill
  • Idea generator events
  • Communities of practice
  • and loads more

…and once you are ready to tell people about what you did internally, give people the chance to know about it externally. (by going to conferences, meetups, and events) It is well worth it.

Broadening everyone’s skill set is another important factor. It will mean that teams can run the solutions that they have created. It also means that you will feel less of an impact from people moving around, and you can give people more of an opportunity to do so. In turn this helps you retain good people for longer.

There is so much more to cover when it comes to the cultural aspect of DevOps…

…but the important thing to remember is that a lot of the details will depend on your organisation.

The other point that I would like to stress is that all the points made in this article are only my opinion.

If you have a different opinion, or would like to discuss things, do not hesitate to reach out.

Explore the rest of the series: