Devel-ops: Bringing development practices to operations tasks

29th July 2019 Jonathan Relf DEVEL-OPS

Commify is the team behind a global portfolio of business messaging brands. We work with more than 45,000 companies, helping them transform their mobile communications with their customers and staff.

We provide SMS, voice, web, IP/OTT, email and intelligent multichannel messaging services both on a self-serve basis (through an online platform or API), and as tailored solutions to more complex needs. www.commify.com

BRINGING DEVELOPMENT PRACTICES TO OPERATIONS TASKS DEVEL-OPS

THE IRON AGE OF I.T.

THE CLOUD AGE OF I.T.

Infrastructure As Code @jbjon

“UNRELIABLE SOFTWARE DEPENDING ON RELIABLE HARDWARE TO RELIABLE SOFTWARE RUNNING ON UNRELIABLE HARDWARE” Infrastructure As Code @jbjon

https://www.slideshare.net/bgracely/enterprise-devops-emc-world-2015-jez-humble-spotlight-session

Martin Fowler, DevOpsCulture, 2015 @jbjon

“ONE COMMON ANTI-PATTERN WHEN INTRODUCING DEVOPS TO AN ORGANIZATION IS TO ASSIGN SOMEONE THE ROLE OF ‘DEVOPS’ OR TO CALL A TEAM A ‘DEVOPS TEAM’. DOING SO PERPETUATES THE KINDS OF SILOS THAT DEVOPS AIMS TO BREAK DOWN…” Martin Fowler, DevOpsCulture, 2015 @jbjon

DEVOPS SHOULD NOT BE VS Devs Ops

DEVOPS SHOULD BE MORE:

Jez Humble @jbjon

DEVOPS IS “A CROSS-DISCIPLINARY COMMUNITY OF PRACTICE DEDICATED TO THE STUDY OF BUILDING, EVOLVING AND OPERATING RAPIDLYCHANGING RESILIENT SYSTEMS AT SCALE.” Jez Humble @jbjon

@jbjon

Unknown @jbjon

“THE DEFINITION OF INSANITY IS REPEATING THE SAME MISTAKES OVER AND OVER AGAIN AND EXPECTING DIFFERENT RESULTS” Unknown @jbjon

Jonathan Relf @jbjon

“THE DEFINITION OF INSANITY IN I.T. OPERATIONS IS MANUALLY REPEATING A TASK OVER AND OVER AGAIN AND EXPECTING THE SAME RESULTS” Jonathan Relf @jbjon

Jeff Sussna, author of “Designing Delivery” @jbjon

“DEVOPS HAS BECOME MORE ABOUT THOSE DETAILS (CONTAINERS, SERVERLESS, SRE, OBSERVABILITY, ETC.) AND LESS ABOUT THE BIG PICTURE. AND, TO SOME DEGREE THE CULTURAL ASPECTS OF DEVOPS HAVE TAKEN A BACK SEAT TO A FOCUS ON FAST LINEAR DELIVERY. THIS PENDULUM WILL NEED TO SWING BACK AT SOME POINT.” Jeff Sussna, author of “Designing Delivery” @jbjon

2017

2018

https://landscape.cncf.io/

THOUGHTWORKS RADAR @jbjon https://www.thoughtworks.com/radar/tools

https://blogs.technet.microsoft.com/xdot509/2013/07/24/getting-started-with-windows-azure-part-2-what-are-cloud-services/

WHY BOTHER DEFINING YOUR OWN INFRASTRUCTURE IN CODE? @jbjon

REGULATORY REQUIREMENTS

3RD PARTY SOLUTION UNCERTAINTY

HARDEN APPLICATION HOSTING

AVOID VENDOR LOCK-IN

THE FOUR STAGES OF INFRASTRUCTURE PROVISIONING CONFIGURING MAINTENANCE TERMINATION @jbjon

PROVISIONING @jbjon

PROVISIONING ▸ Easier since virtualisation @jbjon

PROVISIONING ▸ Easier since virtualisation ▸ Started using ISO images @jbjon

PROVISIONING ▸ Easier since virtualisation ▸ Started using ISO images ▸ Risk of out of date, unpatched VMs @jbjon

CONFIGURATION @jbjon

CONFIGURATION ▸ Can be where most configuration drift is added if not automated @jbjon

CONFIGURATION ▸ Can be where most configuration drift is added if not automated ▸ ‘Tinkering’ @jbjon

CONFIGURATION ▸ Can be where most configuration drift is added if not automated ▸ ‘Tinkering’ ▸ ‘Snowflake servers’ @jbjon

CONFIGURATION ▸ Can be where most configuration drift is added if not automated ▸ ‘Tinkering’ ▸ ‘Snowflake servers’ ▸ Configuration Management software like Puppet & Chef @jbjon

MAINTENANCE @jbjon

MAINTENANCE ▸ Updates or upgrades of software components @jbjon

MAINTENANCE ▸ Updates or upgrades of software components ▸ From security patches to in-place upgrades of O.S. @jbjon

MAINTENANCE ▸ Updates or upgrades of software components ▸ From security patches to in-place upgrades of O.S. ▸ Ordering of patches may affect outcome @jbjon

MAINTENANCE ▸ Updates or upgrades of software components ▸ From security patches to in-place upgrades of O.S. ▸ Ordering of patches may affect outcome ▸ Scripting can help ensure consistency @jbjon

TERMINATION @jbjon

TERMINATION ▸ Fear of shutting off servers @jbjon

TERMINATION ▸ Fear of shutting off servers ▸ Treating servers like “pets, not cattle” @jbjon

TERMINATION ▸ Fear of shutting off servers ▸ Treating servers like “pets, not cattle” ▸ Anti-pattern: Celebrating up-time @jbjon

TERMINATION ▸ Fear of shutting off servers ▸ Treating servers like “pets, not cattle” ▸ Anti-pattern: Celebrating up-time ▸ Plan for the fact an instance could disappear @jbjon

4 GOALS OF INFRASTRUCTURE AS CODE @jbjon

4 GOALS OF INFRASTRUCTURE AS CODE Kief Morris @jbjon

4 GOALS OF INFRASTRUCTURE AS CODE I.T. INFRASTRUCTURE SUPPORTS & ENABLES CHANGE Kief Morris @jbjon

4 GOALS OF INFRASTRUCTURE AS CODE Kief Morris @jbjon

4 GOALS OF INFRASTRUCTURE AS CODE CHANGES TO THE SYSTEM ARE ROUTINE WITHOUT DRAMA OR STRESS Kief Morris @jbjon

4 GOALS OF INFRASTRUCTURE AS CODE Kief Morris @jbjon

4 GOALS OF INFRASTRUCTURE AS CODE I.T. STAFF SPEND THEIR TIME ON VALUABLE THINGS… NOT REPETITIVE TASKS Kief Morris @jbjon

4 GOALS OF INFRASTRUCTURE AS CODE Kief Morris @jbjon

4 GOALS OF INFRASTRUCTURE AS CODE USERS ARE ABLE TO DEFINE, PROVISION, AND MANAGE THE RESOURCES THEY NEED WITHOUT I.T. STAFF TO DO IT FOR THEM Kief Morris @jbjon

DEFINITION TOOLS @jbjon

DEFINITION TOOLS ▸ Good Infrastructure As Code tools @jbjon

DEFINITION TOOLS ▸ Good Infrastructure As Code tools ▸ have scriptable interfaces @jbjon

DEFINITION TOOLS ▸ Good Infrastructure As Code tools ▸ have scriptable interfaces ▸ can be run unattended @jbjon

DEFINITION TOOLS ▸ Good Infrastructure As Code tools ▸ have scriptable interfaces ▸ can be run unattended ▸ can be tailored through config @jbjon

DEFINITION TOOLS ▸ Good Infrastructure As Code tools ▸ have scriptable interfaces ▸ can be run unattended ▸ can be tailored through config ▸ allow tasks to be defined in code @jbjon

DEFINITION TOOLS ▸ Good Infrastructure As Code tools ▸ have scriptable interfaces ▸ can be run unattended ▸ can be tailored through config ▸ allow tasks to be defined in code ▸ the definition files become ‘living documentation’ @jbjon

VERSION CONTROL @jbjon

VERSION CONTROL ▸ Natural part of development workflow @jbjon

VERSION CONTROL ▸ Natural part of development workflow ▸ branching, rollbacks, ownership @jbjon

VERSION CONTROL ▸ Natural part of development workflow ▸ branching, rollbacks, ownership ▸ Single point of truth @jbjon

VERSION CONTROL ▸ Natural part of development workflow ▸ branching, rollbacks, ownership ▸ Single point of truth ▸ Living documentation @jbjon

CONTINUOUS INTEGRATION @jbjon

CONTINUOUS INTEGRATION ▸ Early feedback about potentially breaking changes @jbjon

CONTINUOUS INTEGRATION ▸ Early feedback about potentially breaking changes ▸ Changes tested in isolation from production @jbjon

CONTINUOUS INTEGRATION ▸ Early feedback about potentially breaking changes ▸ Changes tested in isolation from production ▸ Can apply to infrastructure, server, and configuration changes @jbjon

CONTINUOUS INTEGRATION ▸ Early feedback about potentially breaking changes ▸ Changes tested in isolation from production ▸ Can apply to infrastructure, server, and configuration changes ▸ “Does this produce the instance I expect?” @jbjon

CONTINUOUS INTEGRATION ▸ Early feedback about potentially breaking changes ▸ Changes tested in isolation from production ▸ Can apply to infrastructure, server, and configuration changes ▸ “Does this produce the instance I expect?” ▸ “Does this instance have all the features I expect?” @jbjon

CONTINUOUS INTEGRATION ▸ Early feedback about potentially breaking changes ▸ Changes tested in isolation from production ▸ Can apply to infrastructure, server, and configuration changes ▸ “Does this produce the instance I expect?” ▸ “Does this instance have all the features I expect?” ▸ “Is this instance configured for its role correctly?” @jbjon

BUILD PIPELINES @jbjon

BUILD PIPELINES ▸ Avoid ‘automation fear’ @jbjon

BUILD PIPELINES ▸ Avoid ‘automation fear’ ▸ Build regularly as well as on changes @jbjon

BUILD PIPELINES ▸ Avoid ‘automation fear’ ▸ Build regularly as well as on changes ▸ Pipelines maintaining templates ensures up-to-date images @jbjon

BUILD PIPELINES ▸ Avoid ‘automation fear’ ▸ Build regularly as well as on changes ▸ Pipelines maintaining templates ensures up-to-date images ▸ Reduces manual knowledge fading @jbjon

BUILD PIPELINES ▸ Avoid ‘automation fear’ ▸ Build regularly as well as on changes ▸ Pipelines maintaining templates ensures up-to-date images ▸ Reduces manual knowledge fading ▸ Services used to build can be provisioned temporarily @jbjon

BUILD PIPELINES ▸ Avoid ‘automation fear’ ▸ Build regularly as well as on changes ▸ Pipelines maintaining templates ensures up-to-date images ▸ Reduces manual knowledge fading ▸ Services used to build can be provisioned temporarily ▸ Packer supports building machine images @jbjon

AUTOMATED TESTING @jbjon

AUTOMATED TESTING ▸ One of the best practices to borrow from Development @jbjon

AUTOMATED TESTING ▸ One of the best practices to borrow from Development ▸ Not relying on ‘Green builds’ @jbjon

AUTOMATED TESTING ▸ One of the best practices to borrow from Development ▸ Not relying on ‘Green builds’ ▸ Frameworks like ServerSpec (http://serverspec.org) @jbjon

CONTINUOUS DELIVERY @jbjon

CONTINUOUS DELIVERY ▸ Not ‘Continuous Deployment’ @jbjon

CONTINUOUS DELIVERY ▸ Not ‘Continuous Deployment’ ▸ Being able to update Test / Lab environments regularly @jbjon

CONTINUOUS DELIVERY ▸ Not ‘Continuous Deployment’ ▸ Being able to update Test / Lab environments regularly ▸ Risk increases with ‘Time Since Last Success’ @jbjon

CODE ANALYSIS @jbjon

CODE ANALYSIS ▸ Emerging area drawing from Dev practices @jbjon

CODE ANALYSIS ▸ Emerging area drawing from Dev practices ▸ Config management declarative languages have ‘lint’ tools @jbjon

Kief Morris @jbjon

“A NETFLIX TEAM KNEW THAT A PERCENTAGE OF AWS INSTANCES, WHEN PROVISIONED, WILL PERFORM MUCH WORSE THAN THE AVERAGE INSTANCE SO THEY WROTE THEIR PROVISIONING SCRIPTS TO IMMEDIATELY TEST THE PERFORMANCE OF EACH NEW INSTANCE. IF IT DOESN’T MEET THEIR STANDARDS, THE SCRIPT DESTROYS THE INSTANCE AND TRIES AGAIN WITH A NEW INSTANCE.” Kief Morris @jbjon

HIGH PERFORMING ORGANISATIONS @jbjon

HIGH PERFORMING ORGANISATIONS ▸ deploy 200x more frequently @jbjon

HIGH PERFORMING ORGANISATIONS ▸ deploy 200x more frequently ▸ with 2,555x faster lead times @jbjon

HIGH PERFORMING ORGANISATIONS ▸ deploy 200x more frequently ▸ with 2,555x faster lead times ▸ recover 24x faster @jbjon

HIGH PERFORMING ORGANISATIONS ▸ deploy 200x more frequently ▸ with 2,555x faster lead times ▸ recover 24x faster ▸ and have 3x lower change failure rates @jbjon

HIGH PERFORMING ORGANISATIONS ▸ deploy 200x more frequently ▸ with 2,555x faster lead times ▸ recover 24x faster ▸ and have 3x lower change failure rates ▸ have better employee loyalty @jbjon

HIGH PERFORMING ORGANISATIONS ▸ deploy 200x more frequently ▸ with 2,555x faster lead times ▸ recover 24x faster ▸ and have 3x lower change failure rates ▸ have better employee loyalty ▸ spend 22% less time on unplanned work and rework @jbjon

5 STAGES OF DEVOPS EVOLUTION @jbjon

WAYS TO START INTRODUCING THIS @jbjon

WAYS TO START INTRODUCING THIS ▸ Start small @jbjon

WAYS TO START INTRODUCING THIS ▸ Start small ▸ Script everything @jbjon

WAYS TO START INTRODUCING THIS ▸ Start small ▸ Script everything ▸ Automate the process @jbjon

WAYS TO START INTRODUCING THIS ▸ Start small ▸ Script everything ▸ Automate the process ▸ Run it regularly @jbjon

WAYS TO START INTRODUCING THIS ▸ Start small ▸ Script everything ▸ Automate the process ▸ Run it regularly ▸ Test the changes in a safe environment @jbjon

WAYS TO START INTRODUCING THIS ▸ Start small ▸ Script everything ▸ Automate the process ▸ Run it regularly ▸ Test the changes in a safe environment ▸ Monitor all the things @jbjon

AGILE ENGINEERING WORKS WHEN WE SHARE WHAT WORKS, INTRODUCE CONSISTENCY AND COLLABORATE

QUESTIONS?

ABOUT ME ▸ Jonathan Relf ▸ Solutions Architect @ Commify ▸ about.me/jbjon @jbjon