TS T RYU CRT UAR E LNE S SNN E SYS
THE
H H O W M O R E M E A N I N G F U L C O D E C A N M A K E Y O U R P R O J E C T M O R E R E S I L I E N T & M A I N TA I N A B L E H
OF
THE TYRANNY OF STRUCTURELESSNESS
For a number of years I have been familiar with the observation that the quality of programmers is a decreasing function of the density of GOTO statements in the programs they produce 🎤💧 EDSGER DIJKSTRA
THE TYRANNY OF STRUCTURELESSNESS
For a number of years I have been familiar with the observation that the quality of programmers is a decreasing function of the density of GOTO statements in the programs they produce 🎤💧 EDSGER DIJKSTRA
THE TYRANNY OF STRUCTURELESSNESS B R O O K LY N Z E L E N K A , @ e x p e d e
THE TYRANNY OF STRUCTURELESSNESS B R O O K LY N Z E L E N K A , @ e x p e d e
• Cofounder/CTO at Fission • https://fission.codes • Make DevOps & Backend obsolete
😈
• Spending a lot of time with IPFS & DIDs • PLT & VM enthusiast • Ethereum Core Developer • Primary author of Witchcraft Suite & Exceptional
THE TYRANNY OF STRUCTURELESSNESS B R O O K LY N Z E L E N K A , @ e x p e d e
• Cofounder/CTO at Fission • https://fission.codes • Make DevOps & Backend obsolete
😈
• Spending a lot of time with IPFS & DIDs • PLT & VM enthusiast • Ethereum Core Developer • Primary author of Witchcraft Suite & Exceptional
THE BIG IDEA
W H O ’S O RG LO O KS L I K E T H I S ? 🧠
,
,
⚙
,
THE BIG IDEA
W H O ’S O RG LO O KS L I K E T H I S ? 🧠
,
,
⚙
,
THE BIG IDEA
HOW ABOUT THIS? -
,
⚙
,
,
THE BIG IDEA
HOW ABOUT THIS? -
,
⚙
,
,
THE BIG IDEA
QUESTIONS We want more type of features over time. As a result, complexity grows at an exponential rate.
THE BIG IDEA
QUESTIONS We want more type of features over time. As a result, complexity grows at an exponential rate. How do you make Elixir code more flexible and easier to reason about at scale?
THE BIG IDEA
QUESTIONS We want more type of features over time. As a result, complexity grows at an exponential rate. How do you make Elixir code more flexible and easier to reason about at scale? Do you think that the patterns we use today are the best possible patterns for software?
THE BIG IDEA
QUESTIONS We want more type of features over time. As a result, complexity grows at an exponential rate. How do you make Elixir code more flexible and easier to reason about at scale? Do you think that the patterns we use today are the best possible patterns for software? How will you write code in 2020, 2025, and 2050?
GOTOS CONSIDERED HARMFUL
W H AT ’ S S O B A D A B O U T H AV I N G C O N T R O L? 🦶 🔫
• GOTOs • Low level instruction • Literally how the machine is going to see it • Extremely flexible • Highly concrete • Huge number of implicit states
GOTOS CONSIDERED HARMFUL
W H AT ’ S S O B A D A B O U T H AV I N G C O N T R O L? 🦶 🔫
• GOTOs • Low level instruction • Literally how the machine is going to see it • Extremely flexible • Highly concrete • Huge number of implicit states
Line 1 Line 2 Line 3 Line 4 Line 5 — GOTO Line 6
GOTOS CONSIDERED HARMFUL
W H AT ’ S S O B A D A B O U T H AV I N G C O N T R O L? 🦶 🔫
• GOTOs • Low level instruction • Literally how the machine is going to see it • Extremely flexible • Highly concrete • Huge number of implicit states
Line 1 Line 2 Line 3 Line 4 Line 5 — GOTO Line 6
GOTOS CONSIDERED HARMFUL
W H AT ’ S S O B A D A B O U T H AV I N G C O N T R O L? 🦶 🔫
• GOTOs • Low level instruction • Literally how the machine is going to see it • Extremely flexible • Highly concrete • Huge number of implicit states
Line 1 Line 2 Line 3 Line 4 Line 5 — GOTO Line 6
GOTOS CONSIDERED HARMFUL
W H AT ’ S S O B A D A B O U T H AV I N G C O N T R O L? 🦶 🔫
• GOTOs • Low level instruction • Literally how the machine is going to see it • Extremely flexible • Highly concrete • Huge number of implicit states
Line 1 Line 2 Line 3 Line 4 Line 5 — GOTO Line 6
💥
GOTOS CONSIDERED HARMFUL
STRUCTURED PROGRAMMING
• Subroutines • Loops • Switch/branching • Named routines
while
GOTOS CONSIDERED HARMFUL
T H E N E X T G E N E R AT I O N 🚀
• Functions • Map • Reduce • Filter • Constraint solvers
GOTOS CONSIDERED HARMFUL
TRADEOFFS
• Exchange granular control for structure • Meaning over mechanics • More human than machine • Safer! • Spectrum • Turing Tarpit • Church Chasm • Haskell Fan Fiction
COMPLEXITY
THE BAD KIND ☠
• Probably pretty familiar with this • Euphemism for • Complicated • Inconsistent • No plan • “Unstructured mess”
COMPLEXITY
C O M P L E X ! = C O M P L I C AT E D
• Complex: interconnected parts • Complicated: difficult to understand
COMPLEXITY
ACTO R A BYS S 🕳 1.
Each step is very simple
2.
Reasoning about dynamic organisms is hard 1.
Remember to store your data for crash recovery
2.
Called collaborator may not be there
3.
Complexity grows faster than linear
4.
Find common factors — your abstraction
COMPLEXITY
ACTO R A BYS S 🕳 1.
Each step is very simple
2.
Reasoning about dynamic organisms is hard 1.
Remember to store your data for crash recovery
2.
Called collaborator may not be there
3.
Complexity grows faster than linear
4.
Find common factors — your abstraction
FIGHTING GenSoup
G O O D I N T E R FA C E S ! = G O O D A B S T R A C T I O N S
• GenServer & co are actually pretty low level • Please add some semantics! • Don’t reinvent the wheel every time 🎡 • Let’s look at a very common example
FIGHTING GenSoup
W H AT D I D W E G E T ?
• Common interface • Encapsulate the detail • Don’t have to think about mechanics anymore
ABSTRACTION & DSLS
COMMONALITIES
• They clearly have a similar structure • NOT equally expressive •
Enumerable
• Always converted to List •
Witchcraft.Functor
ABSTRACTION & DSLS
COMMONALITIES
• Different, but also have similar structure • Not very pipeable because 2 paths • …lots of duplicate code
ABSTRACTION & DSLS
COMMONALITIES
• Different, but also have similar structure • Not very pipeable because 2 paths • …lots of duplicate code • Why limit to only to two ways?
ABSTRACTION & DSLS
S TA R T F R O M R U L E S
• Describe what the overall solution looks like • Choose how it gets run contextually
ABSTRACTION & DSLS
TWO-PHASE
• Always a two-phase process • Abstract, then concrete • Do concretion at application boundary
ABSTRACTION & DSLS
TWO-PHASE
• Always a two-phase process • Abstract, then concrete • Do concretion at application boundary
ABSTRACTION & DSLS
TWO-PHASE
• Always a two-phase process • Abstract, then concrete • Do concretion at application boundary
ABSTRACTION & DSLS
I M P R O V I N G K e r n e l — FA L L B A C K K E Y S
• Composition is at the heart of modularity
ABSTRACTION & DSLS
I M P R O V I N G K e r n e l — FA L L B A C K K E Y S
• Composition is at the heart of modularity • Orthogonality is at the heart of composition
ABSTRACTION & DSLS
I M P R O V I N G K e r n e l — FA L L B A C K K E Y S
• Composition is at the heart of modularity • Orthogonality is at the heart of composition
ABSTRACTION & DSLS
I M P R O V I N G K e r n e l — FA L L B A C K K E Y S
• Composition is at the heart of modularity • Orthogonality is at the heart of composition
ABSTRACTION & DSLS
I M P R O V I N G K e r n e l — FA L L B A C K K E Y S
• Composition is at the heart of modularity • Orthogonality is at the heart of composition • Let’s abstract default values!
ABSTRACTION & DSLS
I M P R O V I N G K e r n e l — FA L L B A C K K E Y S
• Composition is at the heart of modularity • Orthogonality is at the heart of composition • Let’s abstract default values!
ABSTRACTION & DSLS
I M P R O V I N G K e r n e l — FA L L B A C K K E Y S
• Composition is at the heart of modularity • Orthogonality is at the heart of composition • Let’s abstract default values!
ABSTRACTION & DSLS
I M P R O V I N G K e r n e l — FA L L B A C K K E Y S
• Composition is at the heart of modularity • Orthogonality is at the heart of composition • Let’s abstract default values!
ABSTRACTION & DSLS
I M P R O V I N G K e r n e l — FA L L B A C K K E Y S
• Composition is at the heart of modularity • Orthogonality is at the heart of composition • Let’s abstract default values!
🔁
ABSTRACTION & DSLS
I M P R O V I N G K e r n e l — FA L L B A C K K E Y S
• Composition is at the heart of modularity • Orthogonality is at the heart of composition • Let’s abstract default values! • More focused (does one thing)
🔁
ABSTRACTION & DSLS
I M P R O V I N G K e r n e l — FA L L B A C K K E Y S
• Composition is at the heart of modularity • Orthogonality is at the heart of composition • Let’s abstract default values! • More focused (does one thing) • More general (works everywhere)
🔁
ABSTRACTION & DSLS
I M P R O V I N G K e r n e l — FA L L B A C K K E Y S
• Composition is at the heart of modularity • Orthogonality is at the heart of composition • Let’s abstract default values! • More focused (does one thing) • More general (works everywhere) • Ad hoc function extension
🔁
ABSTRACTION & DSLS
IMPROVING Kernel — BANG FUNCTIONS
💣
Works everywhere Any data Any error struct
Abstracted out 💣 foo!/* from foo/*
🆗 ⏩ 💣
Any flow (esp. pipes) Super easy to test
ABSTRACTION & DSLS
IMPROVING Kernel — BANG FUNCTIONS
💣
Works everywhere Any data Any error struct
Abstracted out 💣 foo!/* from foo/*
🆗 ⏩ 💣
Any flow (esp. pipes) Super easy to test BONUS Disambiguate between nil value and actual errors
ABSTRACTION & DSLS
N O T E : M E TA P H O R
• Concept: Flow-ability is very core to Elixir’s ethos • Kernel.|>/2
Because it’s easier now
ABSTRACTION & DSLS
N O T E : M E TA P H O R
• Concept: Flow-ability is very core to Elixir’s ethos • Kernel.|>/2 • Consistent flow metaphor / punning on existing metaphor • Exceptional: ~>/2 and >>>/2
Because it’s easier now
ABSTRACTION & DSLS
N O T E : M E TA P H O R
• Concept: Flow-ability is very core to Elixir’s ethos • Kernel.|>/2 • Consistent flow metaphor / punning on existing metaphor • Exceptional: ~>/2 and >>>/2
Because it’s easier now
ABSTRACTION & DSLS
N O T E : M E TA P H O R
• Concept: Flow-ability is very core to Elixir’s ethos • Kernel.|>/2 • Consistent flow metaphor / punning on existing metaphor • Exceptional: ~>/2 and >>>/2
Because it’s easier now
ABSTRACTION & DSLS
N O T E : M E TA P H O R
• Concept: Flow-ability is very core to Elixir’s ethos • Kernel.|>/2 • Consistent flow metaphor / punning on existing metaphor • Exceptional: ~>/2 and >>>/2
Because it’s easier now
ABSTRACTION & DSLS
N O T E : M E TA P H O R
• Concept: Flow-ability is very core to Elixir’s ethos • Kernel.|>/2 • Consistent flow metaphor / punning on existing metaphor • Exceptional: ~>/2 and >>>/2
Because it’s easier now
ABSTRACTION & DSLS
W H AT ’ S G A I N E D ?
• Clear • Composable • Greater reuse ♻ • User choice • Increased testability • Simple example: is_exception?/1 • Could still add protocol to get even more power
ABSTRACTION & DSLS
H O W T O E AT T H E E L E P H A N T
• By feature? • By behaviour? • By structure / properties!
STRUCTURE
A S S O C I AT I V I T Y
• Not a data structure • Not a function • An interface & rules! p a t e m w o l f e h t e t o (N
) r ho
STRUCTURE
H OW TO C O M P O S E P RO P E RT I E S
• A structure of structures • Keep it in your brain • Enforce with TypeClass
POWER UP
EXPLICIT ASSUMPTIONS
• Parallel pipes • Concurrency = partial order • Monotonic • All loops must be linearized
t
POWER UP
EXPLICIT ASSUMPTIONS
• Parallel pipes • Concurrency = partial order • Monotonic • All loops must be linearized • Properties • Serial composition • Parallel composition • Explicit evaluation strategy
t
POWER UP
U P S H OT
• Higher semantic density (focused on meaning not mechanics) • Declarative, configurable data flow 🤯
POWER UP
U P S H OT
• Higher semantic density (focused on meaning not mechanics) • Declarative, configurable data flow 🤯 • Extremely extensible • defimpl Dataflow,
for: %Stream{}
• defimpl Dataflow,
for: %Distributed{}
• defimpl Dataflow,
for: %Broadway{}
POWER UP
U P S H OT
• Higher semantic density (focused on meaning not mechanics) • Declarative, configurable data flow 🤯 • Extremely extensible • defimpl Dataflow,
for: %Stream{}
• defimpl Dataflow,
for: %Distributed{}
• defimpl Dataflow,
for: %Broadway{}
• Model-testable
POWER UP
U P S H OT
• Higher semantic density (focused on meaning not mechanics) • Declarative, configurable data flow 🤯 • Extremely extensible • defimpl Dataflow,
for: %Stream{}
• defimpl Dataflow,
for: %Distributed{}
• defimpl Dataflow,
for: %Broadway{}
• Model-testable • Composable with other pipes and change evaluation strategies
A CALL FOR LIBRARIES
SUMMARY
• Can plug into / extend • Single-threaded context • Distributed context
A CALL FOR LIBRARIES
SUMMARY
• Can plug into / extend • Single-threaded context • Distributed context • Dynamic hybrid contexts
A CALL FOR LIBRARIES
EXTEND RAILROAD PROGRAMMING
Happy Path (Continue)
Error Case (Skip)
No Effect (Afterwards)
A CALL FOR LIBRARIES
EXTEND RAILROAD PROGRAMMING
Happy Path (Continue)
Error Case (Skip)
No Effect (Afterwards)
A CALL FOR LIBRARIES
EXTEND RAILROAD PROGRAMMING
Happy Path (Continue)
Error Case (Skip)
No Effect (Afterwards)
SUMMARY
KEEP IN MIND 1. Protocols-for-DDD (P4D3) 2. Add a semantic layer 3. How do you locally test your distributed system? Look at the properties! 4. Under which conditions does your code work? What are your assumptions? 5. Prop testing is useful for structured abstractions 6. You should be able to code half-asleep
https://fission.codes https://talk .fission.codes https://tools.fission.codes
L B E D A N K T, A M ST E R D A M 🎉 brooklyn@fission.codes g i t h u b . c o m /e x p e d e @expede
REWRITING A PHOENIX APP
THE PROBLEM
• Inherited project • First go at Phoenix • “We don’t believe in tests”
REWRITING A PHOENIX APP
THE PROBLEM
• Inherited project • First go at Phoenix • “We don’t believe in tests” • Very brittle, many bugs 😱
REWRITING A PHOENIX APP
THE PROBLEM
• Inherited project • First go at Phoenix • “We don’t believe in tests” • Very brittle, many bugs 😱 • Now tight deadline — no time to do a total rewrite • …or was there? 😏
REWRITING A PHOENIX APP
S T R AT E G Y : G E N E R A L I Z AT I O N !
• Deescalate • Running code • Get to “net new” features • Two things that people don’t generally associate with abstraction!
REWRITING A PHOENIX APP
THE (TEMPORARY) SOLUTION
•Generate all of MVVC •Compile-time functions
•Incl. validation
•Spend 90% time on macro
•Write •Test!
REWRITING A PHOENIX APP
TRADEOFFS
• Implement very fast • Flexible • Makes a lot of assumptions • Macro magic • Difficult to read, extend, &c • Low-level implementation • The macro swamp
C R DT CAS E ST U DY
SCENARIO
• Distributed / uncontrolled topology
💻
• Eventual consistency
💻
💻
💻
💻
💻
C R DT CAS E ST U DY
SCENARIO
• Distributed / uncontrolled topology
💻
• Eventual consistency
💻
💻
🚦
💻
💻
💻
C R DT CAS E ST U DY
SCENARIO
• Distributed / uncontrolled topology
💻
• Eventual consistency
💻
💻
🚦
💻
💻
💻
C R DT CAS E ST U DY
SCENARIO
• Distributed / uncontrolled topology
💻
• Eventual consistency
💻
💻
🚦
💻
💻
💻
C R DT CAS E ST U DY
SCENARIO
• Distributed / uncontrolled topology
💻
• Eventual consistency
💻
• High resilience • Infinite(ish) time tolerance • Decentralized (e.g. via gossip)
💻
🚦
💻
• Self-healing 💻
💻
C R DT CAS E ST U DY
SCENARIO
• Distributed / uncontrolled topology
💻
• Eventual consistency
💻
• High resilience • Infinite(ish) time tolerance • Decentralized (e.g. via gossip)
💻
💻
• Self-healing 💻
💻
C R DT CAS E ST U DY
SCENARIO
• Distributed / uncontrolled topology
💻
• Eventual consistency
💻
• High resilience • Infinite(ish) time tolerance • Decentralized (e.g. via gossip)
💻
💻
• Self-healing 💻
💻
C R DT CAS E ST U DY
SCENARIO
• Distributed / uncontrolled topology
💻
• Eventual consistency
💻
• High resilience • Infinite(ish) time tolerance • Decentralized (e.g. via gossip)
💻
💻
• Self-healing • Two variants • State-based • Operation-based ✅
💻
💻
C R DT CAS E ST U DY
B A S I C : P O S I T I V E / N E G AT I V E C O U N T E R • Two sets • Increments • Decrements • Operations • Internal • Increment • Decrement • Compare • External • Merge • Query
C R DT CAS E ST U DY
B A S I C : P O S I T I V E / N E G AT I V E C O U N T E R • Two sets • Increments • Decrements • Operations • Internal • Increment • Decrement • Compare • External • Merge • Query
• Challenges • Negative counts • Double deletes • Solutions • Only delete existing items • Commutativity
C R DT CAS E ST U DY
OBSERVE/REMOVE SE T • Two sub-sets • Add • Remove • Operations • Internal • Add • Remove • Compare • Tag • External • Merge — semigroup • Query
C R DT CAS E ST U DY
OBSERVE/REMOVE SE T • Two sub-sets • Add • Remove • Operations • Internal • Add • Remove • Compare • Tag • External • Merge — semigroup • Query
• Challenges • Add-after-remove • Unique tags vs re-add • Can only delete what you know about • But deleting an unknown element is the same as a noop, so great!
• Remove has priority over add • Wall clock timestamps are unreliable • Need monotonically increasing structure • Solutions • Only delete existing items • Commutativity • Protocols • Collectable • Enumerable • Semigroup (merge) -> Commutative (CRDT) • “Meet semilattice”
W H AT E V E N I S E L I X I R ?
STRUCTURAL INTEGRIT Y ☝ Production Elixir can actually be pretty unstructured
W H AT E V E N I S E L I X I R ?
STRUCTURAL INTEGRIT Y ☝ Production Elixir can actually be pretty unstructured
• Imperative • Lots of ambient state • Spread across processes • Ordering & timing bugs
W H AT E V E N I S E L I X I R ?
STRUCTURAL INTEGRIT Y ☝ Production Elixir can actually be pretty unstructured
• Imperative • Lots of ambient state • Spread across processes • Ordering & timing bugs
• Functional • Well-defined structured functions (map, filter, reduce)
• Immutable references
W H AT E V E N I S E L I X I R ?
W H Y N O T F U L LY S T R U C T U R E D ?
• Naive structure can be rigid! • Composition • Orthogonality
W H AT E V E N I S E L I X I R ?
W H Y N O T F U L LY S T R U C T U R E D ?
• Naive structure can be rigid! • Composition • Orthogonality
Credit: psihedelisto
POWER UP
Data dominates. If you’ve chosen the right data structures and organized things well, the algorithms will almost always be selfevident. Data structures, not algorithms, are central to programming. ⚙ ~ROB PIKE, 5 RULES OF PROGRAMMING
ABSTRACTION & DSLS
DEFINITIONS
Abstract 📈
42
\ Specific
“Community”
General ]👪_
Conference speakers
L
Concrete 🚚
ABSTRACTION & DSLS
DEFINITIONS
Abstract 📈
42
\ Specific
“Community”
General ]👪_
Conference speakers
L
Concrete 🚚
ABSTRACTION & DSLS
DEFINITIONS
Abstract 📈
42
\ Specific
“Community”
General ]👪_
Conference speakers
L
Concrete 🚚
TS T RYU CRT UAR E LNE S SNN E SYS
THE
H H O W M O R E M E A N I N G F U L C O D E C A N M A K E Y O U R P R O J E C T M O R E R E S I L I E N T & M A I N TA I N A B L E H
OF