Amsterdam | April 2 - 3, 2019 Monitoring OVH: 300k servers, 28 DCs… and one Metrics platform Horacio Gonzalez @LostInBrittany @LostInBrittany

Who are we? Introducing myself and introducing OVH @LostInBrittany

Horacio Gonzalez @LostInBrittany Spaniard lost in Brittany, developer, dreamer and all-around geek Flutter @LostInBrittany

OVH : Key Figures 1.3M Customers worldwide in 138 Countries 1.5 Billions euros investment over five years 28 Datacenters (growing) 350k Dedicated Servers 200k Private cloud VMs running 650k Public cloud Instances created in a month 20TB bandwidth capacity

2 500 Employees in 19 countries 18 Years of Innovation 35 Points of presence 4TB Anti DDoS capacity Hosting capacity : 1.3M Physical Servers @LostInBrittany

OVH: A Global Leader on Cloud 200k Private cloud VMs running 1 Dedicated IaaS Europe 2018 27 Datacenters Own 15 Tbps Hosting capacity : 1.3M Physical Servers 360k Servers already deployed 2020 50 Datacenters Netwok with 35 PoPs

1.3M Customers in 138 Countries @LostInBrittany

Ranking & Recognition 1st European Cloud Provider* 1st Hosting provider in Europe 1st Provider Microsoft Exchange Certified vCloud Datacenter Certified Kubernetes platform (CNCF) Vmware Global Service Provider 2013-2016 Veeam Best Cloud Partner of the year (2018) @LostInBrittany

Netcraft 2017 -

OVH: Our solutions Cloud Web Hosting Mobile Hosting Telecom VPS Containers ▪ Dedicated Server Domain names VoIP Public Cloud Compute ▪ Data Storage Email SMS/Fax Private Cloud ▪ Network and Database CDN Virtual desktop Serveur dédié Security Object Storage Web hosting Cloud HubiC Over theBox ▪ Licences Cloud Desktop Securities MS Office Hybrid Cloud Messaging MS solutions @LostInBrittany

Once upon a time… Because I love telling tales @LostInBrittany

This talk is about a tale… A true one nevertheless @LostInBrittany

And as in most tales It begins with a mission @LostInBrittany

And a band of heroes Engulfed into the adventure @LostInBrittany

They fight against mishaps And all kind of foes @LostInBrittany

They build mighty fortresses Pushing the limits of possible @LostInBrittany

And defend them day after day Against all odds @LostInBrittany

But we don’t know yet the end Because this tale isn’t finished yet @LostInBrittany

It begins with a mission Build a metrics platform for OVH @LostInBrittany

Why do we need metrics? To make better decisions by using numbers @LostInBrittany

Why do we need metrics? We want our code to add value @LostInBrittany

Why do we need metrics? We need to make better decisions about our code @LostInBrittany

Why do we need metrics? Code adds value when it runs not when we write it @LostInBrittany

Why do we need metrics? We need to know what our code does when it runs @LostInBrittany

Why do we need metrics? We can’t do this unless we measure it @LostInBrittany

Why do we need metrics? We have a mental model of what our code does @LostInBrittany

Why do we need metrics? This representation can be wrong @LostInBrittany

Why do we need metrics? We can’t know until we measure it @LostInBrittany

Find the bottleneck ‘’ “The app is slow.” - User @LostInBrittany

Find the bottleneck ‘’ “The app is slow.” - User “The page takes 500ms!” - Ops @LostInBrittany

Find the bottleneck ? SQL Query? Template Rendering? Session Storage? @LostInBrittany

Find the bottleneck ? We don’t know @LostInBrittany

Find the bottleneck

With observability: SQL Query………………………….53ms Template Rendering……….1ms Session Storage……………315ms @LostInBrittany

Find the bottleneck

With observability: SQL Query………………………….53ms Template Rendering……….1ms Session Storage……………315ms @LostInBrittany

Why do we need metrics? We improve our mental model by measuring what our code does @LostInBrittany

Why do we need metrics? We use our mental model to decide what to do @LostInBrittany

Why do we need metrics? A better mental model makes us better at deciding what to do @LostInBrittany

Why do we need metrics? Better decisions makes us better at generating value @LostInBrittany

Why do we need metrics? Measuring make your App better @LostInBrittany

It began with a mission Build a metrics platform for OVH @LostInBrittany

A metrics platform for OVH For all OVH @LostInBrittany

Building OVH Metrics One Platform to unify them all, One Platform to find them, One Platform to bring them all and in the Metrics monitor them @LostInBrittany

What is OVH Metrics? Managed Cloud Platform for Time Series @LostInBrittany

OVH monitoring story We had lots of partial solutions… @LostInBrittany

OVH monitoring story One Platform to unify them all What should we build it on? @LostInBrittany

OVH monitoring story Including a really big @LostInBrittany

OpenTSDB drawbacks OpenTSDB RowKey Design ! @LostInBrittany

OpenTSDB Rowkey design flaws ● .regex. => full table scans ● High cardinality issues (Query latencies) We needed something able to manage hundreds of millions time series OpenTSBD didn’t scale for us @LostInBrittany

OpenTSDB other flaws ● ● ● ● ● Compaction (or append writes) /api/query : 1 endpoint per function? Asynchronous Unauthenticated … @LostInBrittany

Scaling OpenTSDB @LostInBrittany

Metrics needs First need: To be massively scalable @LostInBrittany

Analytics is the key to success Fetching data is only the tip of the iceberg @LostInBrittany

Analysing metrics data To be scalable, analysis must be done in the database, not in user’s computer @LostInBrittany

Metrics needs Second need: To have rich query capabilities @LostInBrittany

Enter Warp 10… Open-source Time series Database @LostInBrittany

More than a Time Series DB Warp 10 is a software platform that ● Ingests and stores time series ● Manipulates and analyzes time series @LostInBrittany

Manipulating Time Series with Warp 10 A true Time Series analysis toolbox ○ Hundreds of functions ○ Manipulation frameworks ○ Analysis workflow @LostInBrittany

Manipulating Time Series with Warp 10 A Time Series manipulation language WarpScript @LostInBrittany

Did you say scalability? From the smallest to the largest… @LostInBrittany

More Warp 10 goodness ● Secured & multi tenant ● Synchronous (transactions) ● In memory Index ● Better Performance ● No cardinality issues ● Better Scalability ● Lockfree ingestion ● Versatile ● WarpScript Query Language (standalone, distributed) ● Support more data types @LostInBrittany

OVH Observability Metrics Platform @LostInBrittany

Metrics Data Platform @LostInBrittany

Building an ecosystem From Warp 10 to OVH Metrics @LostInBrittany

Multi-protocol Why to choose? We need them all! @LostInBrittany

Open source monitoring tools @LostInBrittany

Open source monitoring tools Why choose? Let’s support all of them! @LostInBrittany

Metrics Platform @LostInBrittany

Metrics Platform graphite influx https:// opentsdb .<region>.metrics.ovh.net prometheus warp10 … @LostInBrittany

Metrics Live In-memory, high-performance Metrics instances @LostInBrittany

In-memory: Metrics live +120 million of writes/s @LostInBrittany

In-memory: Metrics live @LostInBrittany

Monitoring is only the beginning OVH Metrics answer to many other use cases @LostInBrittany

Use cases families • • • • Billing Monitoring IoT (e.g. bill on monthly max consumption) ……………………………………………..……. (APM, infrastructure,appliances,…) …..…………………………… (Manage devices, operator integration, …) …………………………………………….…………………. Geo Location (Manage localized fleets) ……..………………… @LostInBrittany

Use cases • • • • • • DC Temperature/Elec/Cooling map Pay as you go billing (PCI/IPLB) GSCAN Monitoring ML Model scoring (Anti-Fraude) Pattern Detection for medical applications @LostInBrittany

SREing Metrics With a great power comes a great responsibility @LostInBrittany

Metrics’ own metrics 432 000 000 000 datapoints / day @LostInBrittany

Metrics’ own metrics 10 Tb / day @LostInBrittany

Metrics’ own metrics 5 000 000 dp/s @LostInBrittany

Metrics’ own metrics 500 000 000 series @LostInBrittany

Our clusters size GRA: BHS: ● 150 nodes ● 2 PB ● 1.1 Gbps ● 30 nodes ● 400 TB ● 120 Mbps @LostInBrittany

Our cluster architecture @LostInBrittany

Detecting errors Before it’s too late @LostInBrittany 85

Extract errors from logs @LostInBrittany

Tailor Forward logs and extract metrics! @LostInBrittany

Monitoring the JVM @LostInBrittany

Documentation @LostInBrittany

JVM GC The good, the bad and the ugly @LostInBrittany

The good @LostInBrittany

The bad @LostInBrittany

… and the ugly #java #jdk11 #zgc @LostInBrittany

Monitoring HBase @LostInBrittany

Number of open regions @LostInBrittany

Queues length @LostInBrittany

Number of read and write requests @LostInBrittany

Preserve data locality @LostInBrittany

Host health @LostInBrittany

Pokédex Inventory all animals. @LostInBrittany

Merging all data sources @LostInBrittany

Global visualization @LostInBrittany

Correlate information @LostInBrittany

Sacha The best tamer @LostInBrittany

An awesome CLI @LostInBrittany

Retrieving bare informations @LostInBrittany

Create region map @LostInBrittany

Move region to another region server @LostInBrittany

Drain regions of the region server @LostInBrittany

Managing multiple hardware profiles @LostInBrittany

Balance the cluster @LostInBrittany

Conclusion That’s all folks! @LostInBrittany