Elasticsearch: Mining Metrics from Logs

A presentation at Stoke-On-Tech October 2018 in October 2018 in Stoke-on-Trent, UK by Jonathan Relf

Slide 1

Slide 1

ELASTICSEARCH: MINING METRICS FROM LOGS

18th October 2018 Jonathan Relf

Slide 2

Slide 2

About Commify

www.commify.com

Slide 3

Slide 3

Title Slide

MINING METRICS FROM LOGS

Slide 4

Slide 4

Seeing Inside

We need to be able to see inside software to know its working ok. Any inspection has a cost. We should be as concise & low impact as possible.

Slide 5

Slide 5

The Power of Logs

Slide 6

Slide 6

The Trouble with Logs

Slide 7

Slide 7

The Power of Metrics

Slide 8

Slide 8

The Trouble with Metrics

Slide 9

Slide 9

Initial reaction to metrics

Slide 10

Slide 10

The start of metrics at Esendex

Slide 11

Slide 11

'Pulse'

https://graphiteapp.org

Slide 12

Slide 12

'Pulse' - zoomed

Slide 13

Slide 13

Shifting balance

Slide 14

Slide 14

Elastic Stack

Slide 15

Slide 15

Centralised Logging

Slide 16

Slide 16

Metrics with Metadata

One solution that I’ve seen used with great success is a log document that’s like a “metric with metadata”. If there’s one thing to take away from today it’s this phrase.

Slide 17

Slide 17

Json Log documents

Slide 18

Slide 18

Correlations: 15 minute snapshot

Slide 19

Slide 19

Sample Metrics

Slide 20

Slide 20

Sample Metics - zoomed

Slide 21

Slide 21

Drilling down

Slide 22

Slide 22

Drilling down - zoomed

Slide 23

Slide 23

Kibana in action

Slide 24

Slide 24

Visualisation drill-downs

Slide 25

Slide 25

Dashboards

Slide 26

Slide 26

Using Elasticsearch with Grafana

Slide 27

Slide 27

Challenges

Slide 28

Slide 28

Next Steps

Slide 29

Slide 29

Timelion

Slide 30

Slide 30

Timelion screenshot

Slide 31

Slide 31

Machine Learning

Slide 32

Slide 32

Machine Learning screenshot

Slide 33

Slide 33

Elastic Stack benefits

Slide 34

Slide 34

Metrics with metadata proposition

Slide 35

Slide 35

Elastic Stack power

Slide 36

Slide 36

Additional X-Pack benefits

Slide 37

Slide 37

Summary

Slide 38

Slide 38

Questions?

Slide 39

Slide 39

About Me