Making (Business) Sense of Automated Testing

Iancho Dimitrov linkedin.com/in/iancho @iandim Agile Testing Days, Potsdam November 7th, 2019

GOAL To efficiently and sustainably provide actionable insight on the state of quality through efficient test execution. Agile Testing Days 2019, Potsdam – Making (Business) Sense of Automated Testing 2

Agile Testing Days 2019, Potsdam – Making (Business) Sense of Automated Testing 3

Trophies | Scars | Part-of-my-job Agile Testing Days 2019, Potsdam – Making (Business) Sense of Automated Testing 4

Agile Testing Days 2019, Potsdam – Making (Business) Sense of Automated Testing 5

Agile Testing Days 2019, Potsdam – Making (Business) Sense of Automated Testing 6

• Test Automation is not always - appropriate / necessary - cost effective • Even when appropriate and cost effective it often fails to deliver results – and to survive Agile Testing Days 2019, Potsdam – Making (Business) Sense of Automated Testing 7

Possible BENEFITS • reduced – cost of testing – time needed for testing (i.e. “time to market/production”) • more time spent by QA Engineers on meaningful work with higher added value Agile Testing Days 2019, Potsdam – Making (Business) Sense of Automated Testing 8

Possible BENEFITS • • test EXECUTION coverage o broader (incl. versions, environments, devices) o deeper (variations of test data) o tests that are impossible to do manually DevOps, CI/CD Agile Testing Days 2019, Potsdam – Making (Business) Sense of Automated Testing 9

Possible BENEFITS • test more often –> catch bugs early –> reduced bug-fixing cost • higher reliability (people make mistakes in repetitive tasks) • motivated, fulfilled and happy QA Engineers • can be used for monitoring on production Agile Testing Days 2019, Potsdam – Making (Business) Sense of Automated Testing 10

Source: “The Evolution of Test Automation 2018” QASymphony and TechWell Survey 11

…but it comes with its COSTS • commercial tools - licenses • infrastructure resources • team members training • PoC implementation Agile Testing Days 2019, Potsdam – Making (Business) Sense of Automated Testing 12

…but it comes with its COSTS • new tests design and implementation • (!) framework architecture setup, maintenance & enhancement • (!) tests execution, results analysis, reporting and defect tracking • (!) tests maintenance • test data management Agile Testing Days 2019, Potsdam – Making (Business) Sense of Automated Testing 13

Major RISKS • focus on tool selection – instead of on strategy, plan and process • slow start, high initial costs – and canceled prematurely • (!) exploding maintenance costs (changes in SUT, platforms, etc.) • lack of needed skills – and high hopes on “snake oil” tools Agile Testing Days 2019, Potsdam – Making (Business) Sense of Automated Testing 14

Major RISKS • wrong expectations to replace almost all other QA activities • wrong expectations for quick payback • (!) failure to achieve recognizable goals (esp. with none defined) • unhappy QA Engineers • TA assets to become “shelfware” – in full or partially Agile Testing Days 2019, Potsdam – Making (Business) Sense of Automated Testing 15

Source: “The Evolution of Test Automation 2018” QASymphony and TechWell Survey 16

Source: “The Evolution of Test Automation 2018” QASymphony and TechWell Survey 17

WRAP-UP • Test Automation is NOT a fit for every project • Many possible benefits – but they come at a cost • Many risks – achieving long-term success not trivial • Hard to start and carry on Agile Testing Days 2019, Potsdam – Making (Business) Sense of Automated Testing 18

Agile Testing Days 2019, Potsdam – Making (Business) Sense of Automated Testing 19

Agile Testing Days 2019, Potsdam – Making (Business) Sense of Automated Testing 20

Agile Testing Days 2019, Potsdam – Making (Business) Sense of Automated Testing 21

a separate long-term project initiative Agile Testing Days 2019, Potsdam – Making (Business) Sense of Automated Testing 22 22

2. Evaluate for yourself first – benefit multipliers? • long-term project? • release often? • many combinations to test? ENVs, devices, OSs, data, etc. • CI/CD? Agile Testing Days 2019, Potsdam – Making (Business) Sense of Automated Testing 23

Agile Testing Days 2019, Potsdam – Making (Business) Sense of Automated Testing 24

3. Propose a phased approach • Research & Analysis (invest in it!) • Proof of Concept

• 2-4 weeks • Trial Phase
• 2-3 months • Ongoing initiative - long term Agile Testing Days 2019, Potsdam – Making (Business) Sense of Automated Testing 25

4. Gather an experienced team • TA (initiatives) Leader • TA Technical Expert / Architect Agile Testing Days 2019, Potsdam – Making (Business) Sense of Automated Testing 26

5. Motivated proposal • initial findings – TA applicable? • a comprehensive plan for the PoC • high-level plan for the Trial Phase • expected benefits Agile Testing Days 2019, Potsdam – Making (Business) Sense of Automated Testing 27

5. In the project plans include: • costs/efforts estimation (budget) • expected results & deliverables • risks & mitigation plan • team and time allocation • implementation timeline …and be pessimistic in all estimations Agile Testing Days 2019, Potsdam – Making (Business) Sense of Automated Testing 28

6. Move on – step by step • Get buy-in from the Budget Owner (adapt proposal if necessary) • Do a successful PoC – and demonstrate the results • Prepare an updated and detailed TA Trial Phase project plan • Pitch the TA Trial Phase proposal, get approval and go forward • Make sure to keep the key stakeholder(s) up to date with progress at all times Agile Testing Days 2019, Potsdam – Making (Business) Sense of Automated Testing 29

Agile Testing Days 2019, Potsdam – Making (Business) Sense of Automated Testing 30

Agile Testing Days 2019, Potsdam – Making (Business) Sense of Automated Testing 31

Agile Testing Days 2019, Potsdam – Making (Business) Sense of Automated Testing 32

Agile Testing Days 2019, Potsdam – Making (Business) Sense of Automated Testing 33

Agile Testing Days 2019, Potsdam – Making (Business) Sense of Automated Testing 34

Agile Testing Days 2019, Potsdam – Making (Business) Sense of Automated Testing 35

Agile Testing Days 2019, Potsdam – Making (Business) Sense of Automated Testing 36

Agile Testing Days 2019, Potsdam – Making (Business) Sense of Automated Testing 37

Agile Testing Days 2019, Potsdam – Making (Business) Sense of Automated Testing 38

Agile Testing Days 2019, Potsdam – Making (Business) Sense of Automated Testing 39

a separate long-term project initiative Agile Testing Days 2019, Potsdam – Making (Business) Sense of Automated Testing 40 40

Agile Testing Days 2019, Potsdam – Making (Business) Sense of Automated Testing 41

2. Set & track TA-Specific Measurable Goals. Examples: • Regression Test Suite for ”Sprint X-1” – 100% automated • Smoke Test Suite – up-to-date all the time • All test run results analyzed and distributed in time (e.g. < 2 hrs.) • Effort spent for maintenance < 15%, and all tests working Agile Testing Days 2019, Potsdam – Making (Business) Sense of Automated Testing 42

2. Set & track Measurable & Realistic Goals. Anti-patterns: • “test coverage” (analyzed and well-reported test runs matter) • “test automation of sprint delivery, within the sprint” • bugs found Agile Testing Days 2019, Potsdam – Making (Business) Sense of Automated Testing 43

3. Reusable components • 2 aspects – technical & functional • well documented • (!) strict process for technical design …crucial for both speed of implementation & maintenance! Agile Testing Days 2019, Potsdam – Making (Business) Sense of Automated Testing 44

Agile Testing Days 2019, Potsdam – Making (Business) Sense of Automated Testing 45

Agile Testing Days 2019, Potsdam – Making (Business) Sense of Automated Testing 46

Agile Testing Days 2019, Potsdam – Making (Business) Sense of Automated Testing 47

Agile Testing Days 2019, Potsdam – Making (Business) Sense of Automated Testing 48

4. Automated “test automation” • configurable runs – different ENV/devices/SUT version • test data prep. & clean-up • scheduling • parallel runs Agile Testing Days 2019, Potsdam – Making (Business) Sense of Automated Testing 49

4. Automated “test automation” • 100% separation of Data from Tests • (user friendly) reports, accessible by all team members – incl. screenshots, video recordings, logs Agile Testing Days 2019, Potsdam – Making (Business) Sense of Automated Testing 50

4. Automated “test automation” • integration with Test Management system • non-TA team members enabled to execute test suites Agile Testing Days 2019, Potsdam – Making (Business) Sense of Automated Testing 51

4. Automated “test automation” • treat it as a software development initiative: – start with the end in mind – architecture – code reviews Agile Testing Days 2019, Potsdam – Making (Business) Sense of Automated Testing 52

Agile Testing Days 2019, Potsdam – Making (Business) Sense of Automated Testing 53

Agile Testing Days 2019, Potsdam – Making (Business) Sense of Automated Testing 54

Agile Testing Days 2019, Potsdam – Making (Business) Sense of Automated Testing 55

Track and provide transparency on: • Results achieved • Time spent • Work done • Resources used …regularly – on Sprint/Release basis, but at least once in 2 or 3 weeks. Agile Testing Days 2019, Potsdam – Making (Business) Sense of Automated Testing 56

Sample KPIs for Execution, Results Analysis and Reporting: • Number Test Suite runs • Number Test runs (and % of successful runs) • Number Executed “User actions”(→ man-hours saved) • Percentage “Test Run Results” reports delivered in time • Number Automated Test cycle time Agile Testing Days 2019, Potsdam – Making (Business) Sense of Automated Testing 57

Sample KPIs for New Tests Design & Implementation: • Percentage Sprint regression TA goal achieved (X out of Y) • Number New tests automated + “user actions” • Number New reusable components created • Number New Smoke tests automated • Number Test Cases designed for automation (and %) Agile Testing Days 2019, Potsdam – Making (Business) Sense of Automated Testing 58

Sample KPI for Maintenance: • Number Existing tests adapted/maintained Framework/Tooling Enhancement: • < highlights on framework enhancements implemented > Agile Testing Days 2019, Potsdam – Making (Business) Sense of Automated Testing 59

Sample KPIs for Time spent (quantity & percentage) on: • run results analysis and reporting • new tests implementation • test maintenance • framework/tooling enhancement • other – test data preparation, process optimization, etc. Agile Testing Days 2019, Potsdam – Making (Business) Sense of Automated Testing 60

KPIs on Doing Test Automation Right (continuous improvement) • track progress on achieving the goals in your prioritized list • track “health” status of all items already achieved (as they tend to deteriorate if not taken care of) Agile Testing Days 2019, Potsdam – Making (Business) Sense of Automated Testing 61

The regular TA Status Report should be: • short, but comprehensive – 2-3 slides • addressed to and discussed with key stakeholders: - Project Manager / Budget Owner - Test Manager - Scrum Master / Dev Team Leader

• Product Owner / Lead BA / Product Manager Agile Testing Days 2019, Potsdam – Making (Business) Sense of Automated Testing 62

TA Status Report – sample elements Effort Distribution (%) Sprint TA Scope DONE 120 100 80 60 40 20 0 Sprint 1 New Tests Impl. Sprint 2 Results Analysis Sprint 3 Maintenance Agile Testing Days 2019, Potsdam – Making (Business) Sense of Automated Testing Sprint 4 Framework Enhancements 63

TA Status Report – sample elements Agile Testing Days 2019, Potsdam – Making (Business) Sense of Automated Testing 64

Agile Testing Days 2019, Potsdam – Making (Business) Sense of Automated Testing 65

Agile Testing Days 2019, Potsdam – Making (Business) Sense of Automated Testing 66

Agile Testing Days 2019, Potsdam – Making (Business) Sense of Automated Testing 67

Agile Testing Days 2019, Potsdam – Making (Business) Sense of Automated Testing 68

Agile Testing Days 2019, Potsdam – Making (Business) Sense of Automated Testing 69

Agile Testing Days 2019, Potsdam – Making (Business) Sense of Automated Testing 70

Agile Testing Days 2019, Potsdam – Making (Business) Sense of Automated Testing 71

Agile Testing Days 2019, Potsdam – Making (Business) Sense of Automated Testing 72

Iancho Dimitrov iancho.d@gmail.com linkedin.com/in/iancho @iandim Agile Testing Days 2019, Potsdam – Making (Business) Sense of Automated Testing 74

Agile Testing Days 2019, Potsdam – Making (Business) Sense of Automated Testing 75

Set up Process & Responsibilities for: • planning & case design → technical design → implementation • run → results analysis → reporting & defect tracking • (!) proactive test maintenance • framework/tooling enhancement Agile Testing Days 2019, Potsdam – Making (Business) Sense of Automated Testing 76

Make the most of TA (execution) results • distribute run results to all relevant team members (mailgroups) • meaningful descriptions – incl. screenshots, input data, logs • regularly update all team members on progress on key TA goals: - could be once in 3, 4 or 6 months - semi-formal gathering (e.g. a Lunch&Learn/Brown-bag session) - gather feedback Agile Testing Days 2019, Potsdam – Making (Business) Sense of Automated Testing 77

Assure links & traceability between artifacts: • User Story / Feature • Test Case • Test Run Reports • Defects Agile Testing Days 2019, Potsdam – Making (Business) Sense of Automated Testing 78

Design for automation: • the right test cases • designed for automated testing • decomposed to the needed level of reusable components (vs. automating tests designed for manual testing) Agile Testing Days 2019, Potsdam – Making (Business) Sense of Automated Testing 79

Cross-functional team members involvement in TA • Developers – assuring TA-friendly software • BA/PO/PMs – design & prioritization, esp. “what’s next” • Test Management – integration of TA in the QA Strategy & Plan …and vice versa - TA Engineers – biz domain; features, requirements Agile Testing Days 2019, Potsdam – Making (Business) Sense of Automated Testing 80

• Do Test Automation Right - so many things to address: - make a goals list – prioritized - reap the low-hanging fruit (things mostly within your control) - work on achieving the goals, track progress • Measure | Communicate | Optimize Agile Testing Days 2019, Potsdam – Making (Business) Sense of Automated Testing 81

• Don’t expect to achieve everything quickly • Make a custom list for your initiative – and prioritize it • Reap the low-hanging fruit • Set deadlines, track progress – and work on these one step at a time Agile Testing Days 2019, Potsdam – Making (Business) Sense of Automated Testing 82

☺ Iancho Dimitrov iancho.d@gmail.com LinkedIn.com/in/iancho Agile Testing Days 2019, Potsdam – Making (Business) Sense of Automated Testing 83