Join Newsletter

Learning from Incidents

YOW! CTO Summit 2019 Melbourne

Aligning to a DevOps culture has seen many organisations gain a distinct competitive advantage in their marketplace - especially if they started changing their thinking early which Seek did. Frequent daily deployments, teams owning what they build, the ability to iterate and deliver Products faster, and a greater emphasis on collaboration with much less of "that's not my job", has achieved many benefits. But there is flipsides to this rapid rate of change, and depending on your perspective, how you capitalise on it could be the next big advantage you can take.

When teams gain greater autonomy to make technology choices the amount of diversification in your enterprise grows rapidly - especially when you are on the bleeding edge of what the major cloud providers are releasing. This increase in diversification will place greater cognitive loads on the people operating and building the system, to a point where an ability to mental model your systems becomes impossible. Incidents and failure will still be a part of normal system functions, still just as complex, but more asynchronous and therefore more difficult to diagnose the reverberations of failure through the system. How you embrace failure in this greater field of diversification, learn from it and use it, is what will set you apart.

This presentation will discuss how Seek has dealt with and collated extensive amounts of data on "Normal Accidents" over the last several years. We will demonstrate how incident analysis and involvement of teams in post-mortem rituals, has paved the way to many starting viewing our diverse software stack as the Socio-Technical system it is, and how appreciating the "Human Factors" elements of incidents are important to building greater resiliency in the system. We will discuss how involvement of technology people in incident investigation and facilitation will lead to richer amounts of data, that can be fed back into the delivery cycle and continuously improve the reliability and resiliency of your products We will also discuss the traps and pitfalls to avoid such as obsessing over the Root Cause and why the “5 Why’s” technique of incident analysis can be flawed.

Andrew Hatch

Head of Platform Engineering



Hailing from a background predominantly based in IT consultancy roles working on small through to large scale IT solutions in roles spanning across Architecture, Development, Testing, IM and DevOps in Australia and India, Andrew has based himself at Seek since 2014. Initially focused on evolving a traditional IT operations team to adopt and follow software engineering principles and automation to deliver better outcomes for the business, Andrew then helped guide the migration and delivery of production applications and systems into the cloud.

Currently managing a team of 12 highly skilled engineers working with all of Seeks Product delivery streams and IT teams, Andrew has a real passion for working in dynamic, innovative and challenging environments building high-performing teams of talented and unique people while still enjoying getting his hands dirty with code. A life-long inhabitant of the Mornington Peninsula, Andrew loves spending time with his wife and two kids, surfing waves down the coast and fiddling with old Datsun’s