Continuous SQL with Apache Streaming (FLaNK and FLiP)

APACHE NIFI Enable easy ingestion, routing, management and delivery of any data anywhere (Edge, cloud, data center) to any downstream system with built in end-to-end security and provenance ACQUIRE • Advanced tooling to industrialize flow development (Flow Development Life Cycle) • • • • DELIVER PROCESS FTP HASH ENCRYPT GEOENRICH FTP SFTP MERGE TALL SCAN SFTP HL7 EXTRACT EVALUATE REPLACE HL7 UDP DUPLICATE EXECUTE TRANSLATE UDP XML SPLIT CONVERT XML HTTP ROUTE TEXT HTTP EMAIL ROUTE CONTENT EMAIL HTML ROUTE CONTEXT HTML IMAGE SYSLO G CONTROL RATE IMAGE SYSLO G Over 300 Prebuilt Processors Easy to build your own Parse, Enrich & Apply Schema Filter, Split, Merger & Route Throttle & Backpressure DISTRIBUTE LOAD • • • • Guaranteed Delivery Full data provenance from acquisition to delivery Diverse, Non-Traditional Sources Eco-system integration 8

Flink SQL Key Takeaway: Rich SQL grammar with advanced time and aggregation tools — specify Kafka partition key on output SELECT foo AS _eventKey FROM sensors — use event time timestamp from kafka — exactly once compatible SELECT eventTimestamp FROM sensors — nested structures access SELECT foo.’bar’ FROM table; — must quote nested column — timestamps SELECT * FROM payments WHERE eventTimestamp > CURRENT_TIMESTAMP-interval ‘10’ second; — unnest SELECT b., u. FROM bgp_avro b, UNNEST(b.path) AS u(pathitem) — aggregations and windows SELECT card, MAX(amount) as theamount, TUMBLE_END(eventTimestamp, interval ‘5’ minute) as ts FROM payments WHERE lat IS NOT NULL AND lon IS NOT NULL GROUP BY card, TUMBLE(eventTimestamp, interval ‘5’ minute) HAVING COUNT(*) > 4 — >4==fraud — try to do this ksql! SELECT us_west.user_score+ap_south.user_score FROM kafka_in_zone_us_west us_west FULL OUTER JOIN kafka_in_zone_ap_south ap_south ON us_west.user_id = ap_south.user_id; 13

Flink SQL SELECT location, station_id, latitude, longitude, observation_time, weather, temperature_string, relative_humidity, wind_string, wind_dir, wind_degrees, wind_mph, pressure_in, dewpoint_string, dewpoint_f, dewpoint_c FROM weather2 WHERE location is not null and location <> ‘null’ and trim(location) <> ” and location like ‘%NJ’ SELECT HOP_END(eventTimestamp, INTERVAL ‘1’ SECOND, INTERVAL ‘30’ SECOND) as windowEnd, count(“close”) as closeCount, sum(cast(“close” as float)) as closeSum, avg(cast(“close” as float)) as closeAverage, min(“close”) as closeMin, max(“close”) as closeMax, sum(case when “close” > 14 then 1 else 0 end) as stockGreaterThan14 FROM stocksraw GROUP BY HOP(eventTimestamp, INTERVAL ‘1’ SECOND, INTERVAL ‘30’ SECOND) © 2021 Cloudera, Inc. All rights reserved. 14