TalEs aftEr midnight for whom thE pagEr tolls

TalEs aftEr midnight for whom thE pagEr tolls An anthology of short cautionary horror storiEs

@cEdmax WEbmastEr bEforE it was cool TEch LEad Condé Nast IntErnational TalEs aftEr Midnight

TalEs aftEr midnight for whom thE pagEr tolls “ThErE arE 5 typEs of EnginEErs: ThosE that havE brokEn production and thosE who arE about to brEak production” Matt BoylE

ChaptEr #1 Don’t DEPLOY that CODE

Don’t DEPLOY that CODE

Don’t DEPLOY that CODE

Don’t DEPLOY that CODE

Don’t DEPLOY that CODE

Don’t DEPLOY that CODE

Don’t DEPLOY that CODE

Don’t DEPLOY that CODE

☠ WHAT WENT WRONG 🧘 WHAT WENT WELL

  • thE procEss was pronE to Errors
  • our rollback stratEgy 🤓 What wE lEarnt
  • AUTOMATE ALL THE THINGS Don’t DEPLOY that CODE

ChaptEr #5 THE CSS OF DOOM

http://www.winpix.net/content/images/2010/03/16_png_transparent_ie6.jpg THE CSS OF DOOM

http://www.winpix.net/content/images/2010/03/16_png_transparent_ie6.jpg THE CSS OF DOOM

http://www.winpix.net/content/images/2010/03/16_png_transparent_ie6.jpg THE CSS OF DOOM

img { behavior: url(“pngbehavior.htc”); } THE CSS OF DOOM

var supported = (/MSIE (5.5)|[6789]/ .test(navigator.userAgent) && navigator.platform == “Win32”); var blankSrc = “blank.gif”; if (supported) fixImage(); THE CSS OF DOOM

function fixImage() { var src = element.src; if ( /.png$/.test( src.toLowerCase() ) ) { element.src = blankSrc; element.runtimeStyle.filter = “progid:DXImageTransform.Microsoft.AlphaImageLoader(src=’” + src + “’,sizingMethod=’scale’)”; } } THE CSS OF DOOM

var supported = (/MSIE (5.5)|[6789]/ .test(navigator.userAgent) && navigator.platform == “Win32”); var blankSrc = “blank.gif”; if (supported) fixImage(); THE CSS OF DOOM

THE CSS OF DOOM

THE CSS OF DOOM

☠ WHAT WENT WRONG

🧘 WHAT WENT WELL I don’t EvEn know whErE to start:

  • I havE a fun story to tEll
  • using thE prod db for monitoring - assuming thE CSS CAn’t BREAK anything
  • our rollback stratEgy
  • siloEd & BlamE culturE 🤓 What I lEarnt
  • You nEEd to undErstand thE systEm THE CSS OF DOOM

ChaptEr #3 POLAROID

Replica Replica Polaroid

Replica Replica Polaroid

Polaroid

Polaroid

Polaroid Run books didn’t havE anything on thE issuE ThE support Escalation paths wErE wrong WE FAILED TO COMMUNICATE THE FIX

☠ WHAT WENT WRONG 🧘 WHAT WENT WELL

  • Monitoring wasn’t sEtup for thE rEplicas
  • wE CAUGHT THE PROBLEM BEFORE IT
  • ThE runbooks wErEn’t updatEd BECAME CRITICAL 🤓 What wE lEarnt
  • you can’t control EvErything - communication is KEy Polaroid

AppEndix POST MORTEM

THE DOCUMENT

Summary Post MortEm

THE DOCUMENT

Summary TimElinE Post MortEm

THE DOCUMENT

Summary TimElinE Impact Post MortEm

THE DOCUMENT

Summary TimElinE Impact Diagnosis Post MortEm

THE DOCUMENT

Summary TimElinE Impact Diagnosis Actions Post MortEm

SUGGESTIONS Post MortEm KEEp thE mEEting small KEEp it tEchnical Focus on thE whys? MakE surE thE actions arE rEalistic KEEp it blamElEss

TalEs aftEr midnight for whom thE pagEr tolls TakEaways

Sh*t will go wrong, BE prEparEd

KEEp your docs updatEd Try to undErstand your APP/STACK Don’t BLAME PEOPLE BUT PROCESSES