class: center, middle Ravi Chandra Padmala neena@nilenso.com --- class: center, middle ## First things first --- ### Erlang: What is it? - Programming Language - Many light weight processes communicating via messages & inboxes - Comes with a lot of distributed systems tools out of the box - 1986 language designed for modern problems --- ### Erlang: Who uses it? - WhatsApp - AdRoll - RabbitMQ/Ejabberd/CouchDB (you?) --- ### Erlang: Common OTP Behaviours 1. GenServers - Sync - Async 2. Supervisors - Restart processes --- class: center, middle # Let it crash --- ### What are crashes - Fault - Incorrect state in the system - Error - Unhandled faults that propogate through a system - Failure - When a server stops responding or a program abnormally terminates From: https://www.michaelnygard.com/blog/2016/11/fault-error-failure/ --- ### Dealing with Faults - Load shedding - Circuit breakers - Bulkheads - Fail fast --- ### Dealing with Faults #### Load shedding Listeners dropping messages intelligently to protect themselves --- ### Dealing with Faults #### Circuit breakers Clients temporarily avoiding accessing problematic services --- ### Dealing with Faults #### Bulkheads Partitioning a resource pool into isolated pools to contain failure --- ### Dealing with Faults #### Fail fast Failing early where possible --- ### Dealing with Faults - Load shedding - Circuit breakers - Bulkheads - Fail fast --- class: center, middle # Let it crash --- ### What Crashes? The usual: 1. Unsafe input 2. External resources 3. Divide by zero --- ### What Crashes? The usual: 1. Unsafe input 2. External resources 3. Divide by zero Aren't these bugs? --- ### What Crashes? The usual: 1. Unsafe input 2. External resources 3. Divide by zero Aren't these bugs? Are bugs okay now? --- class: middle # Types of bugs 1. business logic bugs 2. availability/reliability related bugs --- class: center, middle # Let it crash --- # Designing Process Trees --- # Designing Process Trees 1. Separate and pool expensive resources - Connections - Caches --- # Designing Process Trees 1. Separate and pool expensive resources - Connections - Caches 2. Have processes that are - Isolated - Idempotent (or Rejectable) --- # Designing Process Trees 1. Separate and pool expensive resources - Connections - Caches 2. Have processes that are - Isolated - Idempotent (or Rejectable) 3. Commmunicate mindfully - Queuing - Capacity planning --- # Designing Process Trees ### Let's build a web server --- # Designing Process Trees ### Let's build a web server 1. Isolated --- # Designing Process Trees ### Let's build a web server 1. Isolated 2. Rejectable --- # Designing Process Trees ### Let's build a web server 1. Isolated 2. Rejectable Yay reliable request handler processes! --- # Designing Process Trees ### Let's build a web server 1. Isolated 2. Rejectable Yay reliable request handler processes! But what about the connection acceptor processes? --- # Designing Process Trees ### Let's build a web server 1. Isolated 2. Rejectable Yay reliable request handler processes! Yay reliable connection acceptor processes! --- # Designing Process Trees ### Let's build a web server 1. Isolated 2. Rejectable Yay reliable request handler processes! Yay reliable connection acceptor processes! Now we've built (a tiny part of) *Ranch* - https://github.com/ninenines/ranch --- # Designing Process Trees ### Let's build a chat server --- # Designing Process Trees ### Let's build a chat server Rooms - Handle join/exits - Fan out messages --- # Designing Process Trees ### Let's build a chat server - *Processes sending messages to other processes* - Sometimes, processes are stateful - Dependencies across processes - They may share external dependencies --- # Designing Process Trees ### Let's build a chat server #### Processes sending messages to other processes --- # Designing Process Trees ### Let's build a chat server #### Processes sending messages to other processes 1. Isolated --- # Designing Process Trees ### Let's build a chat server #### Processes sending messages to other processes 1. Isolated 2. Idempotent // Rejectable (??) --- # Designing Process Trees ### Let's build a chat server - Processes sending messages to other processes - *Sometimes, processes are stateful* - Dependencies across processes - They may share external dependencies --- # Designing Process Trees ### Let's build a chat server #### Stateful Processes 1. Separate and pool expensive resources - Connections - Caches 2. Have processes that are - Isolated - Idempotent (or Rejectable) - *Rehydratable?* --- # Designing Process Trees ### Let's build a chat server - Processes sending messages to other processes - Sometimes, processes are stateful - *Dependencies across processes* - They may share external dependencies --- # Designing Process Trees ### Let's build a chat server 3. Commmunicate mindfully - Queuing - Capacity planning --- # Designing Process Trees ### Let's build a chat server - Many processes streaming data around - Dependencies across processes - Sometimes, stream processors are stateful - *They may share external dependencies* --- # Designing Process Trees ### Let's build a chat server 1. Separate and pool expensive resources - Connections - Caches 3. Commmunicate mindfully - Queuing - Capacity planning --- # Designing Process Trees ### Let's build a chat server - Many processes streaming data around - Dependencies across processes - Sometimes, stream processors are stateful - They may share external dependencies Are we done? --- # Designing Systems in Erlang Distributed systems come with a lot of complexities Erlang puts these complexities front and center If we address them well, we end up with a formidable system --- class: middle # Let it Crash --- class: middle, inverse # Let it Crash, but ensure it recovers --- class: center, middle, inverse # After the Crash neena@nilenso.com