Skip to content

raghu298/system-design-resources

 
 

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

56 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

System Design Resources

These are the best resources for System Design on the Internet.

Video Processing

Transcoding Videos at Scale: https://www.egnyte.com/blog/2018/12/transcoding-how-we-serve-videos-at-scale/

Facebook Video Broadcasting: https://engineering.fb.com/ios/under-the-hood-broadcasting-live-video-to-millions/

Netflix Video Encoding at Scale: https://netflixtechblog.com/high-quality-video-encoding-at-scale-d159db052746

Netflix Shot based encoding: https://netflixtechblog.com/optimized-shot-based-encodes-now-streaming-4b9464204830

Cluster and Workflow Management

Facebook Cluster Management: https://engineering.fb.com/data-center-engineering/twine/

Google Autopilot - Autoscaling: https://dl.acm.org/doi/pdf/10.1145/3342195.3387524

Netflix Workflow Orchestration: https://netflix.github.io/conductor/

Opensource Workflow Management: https://github.com/spotify/luigi

Meta Hardware Management: https://engineering.fb.com/2020/12/09/data-center-engineering/how-facebook-keeps-its-large-scale-infrastructure-hardware-up-and-running/

Meta Capacity Assignment: https://engineering.fb.com/2022/09/06/data-center-engineering/viewing-the-world-as-a-computer-global-capacity-management/

Amazon EC2: https://www.allthingsdistributed.com/2015/07/under-the-hood-of-the-amazon-ec2-container-service.html

Intra-Service Messaging

What is a message queue: https://www.cloudamqp.com/blog/what-is-message-queuing.html

AirBnb Idempotency: https://medium.com/airbnb-engineering/avoiding-double-payments-in-a-distributed-payments-system-2981f6b070bb

Nginx Service Mesh: https://www.nginx.com/learn/service-mesh/

Meta Async Task Computing: https://engineering.fb.com/2023/01/31/production-engineering/meta-asynchronous-computing/

Message Queue Antipattern

DB as queue Antipattern: http://blog.codepath.com/2012/11/15/asynchronous-processing-in-web-applications-part-1-a-database-is-not-a-queue/

Using a database as a message queue: https://softwareengineering.stackexchange.com/questions/231410/why-database-as-queue-so-bad

Anti-pattern of DB as a queue: http://mikehadlow.blogspot.com/2012/04/database-as-queue-anti-pattern.html

Drawbacks of DB as a queue: https://www.cloudamqp.com/blog/why-is-a-database-not-the-right-tool-for-a-queue-based-system.html

Service Mesh

Kubernetes Service Mesh: https://akomljen.com/kubernetes-service-mesh/

Kubernetes Sidecar:https://www.weave.works/blog/introduction-to-service-meshes-on-kubernetes-and-progressive-delivery

Service Mesh: https://www.weave.works/blog/introduction-to-service-meshes-on-kubernetes-and-progressive-delivery

NginX Service Mesh: https://www.nginx.com/learn/service-mesh/

Practical System Design

Facebook Messenger Optimisations: https://spectrum.ieee.org/how-facebooks-software-engineers-prepare-messenger-for-new-years-eve

YouTube Architecture: http://highscalability.com/youtube-architecture

YouTube scalability 2012: https://www.youtube.com/watch?v=w5WVu624fY8

Distributed Design Patterns: http://horicky.blogspot.com/2010/10/scalable-system-design-patterns.html

Monolith to Microservice: https://martinfowler.com/articles/break-monolith-into-microservices.html

Zerodha Tech Stack: https://zerodha.tech/blog/hello-world/

Distributed File System

Open Source Distributed File System: https://docs.ceph.com/en/latest/architecture/

Amazon S3 Performance hacks: https://aws.amazon.com/blogs/aws/amazon-s3-performance-tips-tricks-seattle-hiring-event/

Amazon S3 object expiration: https://aws.amazon.com/blogs/aws/amazon-s3-object-expiration/

Time Series Databases

Pintrest Time Series Database: https://medium.com/pinterest-engineering/goku-building-a-scalable-and-high-performant-time-series-database-system-a8ff5758a181

Uber Time Series DB: https://eng.uber.com/aresdb/

TimeSeries Relational DB: https://blog.timescale.com/blog/time-series-data-why-and-how-to-use-a-relational-database-instead-of-nosql-d0cd6975e87c/

Facebook Gorilla Time Series DB: http://www.vldb.org/pvldb/vol8/p1816-teller.pdf

Rate Limiting

Circuit Breaker Algorithm: https://martinfowler.com/bliki/CircuitBreaker.html

Uber Rate Limiter: https://github.com/uber-go/ratelimit/blob/master/ratelimit.go

In Memory Database - Redis

Redis Official Documentation : https://redis.com/

Learn Redis through Redis University : https://university.redis.com/

Redis Open Source Repo : https://github.com/redis/redis

Redis Architecture : https://medium.com/opstree-technology/redis-cluster-architecture-replication-sharding-and-failover-86871e783ac0

Network Protocols

What is HTTP: https://engineering.cred.club/head-of-line-hol-blocking-in-http-1-and-http-2-50b24e9e3372

QUIC Protocol: https://www.akamai.com/blog/performance/http3-and-quic-past-present-and-future

TCP Protocol algorithms: https://ee.lbl.gov/papers/congavoid.pdf (First 10 pages are important)

WebRTC: https://webrtc.github.io/webrtc-org/blog/2012/07/23/a-great-introduction-to-webrtc.html

WebSockets: https://datatracker.ietf.org/doc/html/rfc6455#section-1.2

Dynamic Source Routing using QUIC: https://fb.watch/fSEbI4KHlA/

Chess Engine Design

Chess Engine Building: https://www.youtube.com/watch?v=U4ogK0MIzqk

Subscription Management System

Subscription Manager: https://netflixtechblog.com/building-a-rule-based-platform-to-manage-netflix-membership-skus-at-scale-e3c0f82aa7bc

Google Docs

Operational Transform: http://www.codecommit.com/blog/java/understanding-and-applying-operational-transformation

Google Docs: https://www.youtube.com/watch?v=uOFzWZrsPV0&list=PLXDe3d8o9VFtydBV5biyz9iS3WqKsBMD5&index=3

API Design

API Design: https://medium.com/airbnb-engineering/building-services-at-airbnb-part-1-c4c1d8fa811b

Swagger APIs: https://swagger.io/docs/specification/about/

NoSQL Database Internals

Cassandra Architecture: https://docs.datastax.com/en/archived/cassandra/3.0/cassandra/architecture/archIntro.html

Google BigTable Architecture: https://static.googleusercontent.com/media/research.google.com/en//archive/bigtable-osdi06.pdf

Amazon Dynamo DB Internals: https://www.allthingsdistributed.com/2007/10/amazons_dynamo.html

Design Patterns in Amazon Dynamo DB: https://www.youtube.com/watch?v=HaEPXoXVf2k

Internals of Amazon Dynamo DB: https://www.youtube.com/watch?v=yvBR71D0nAQ

NoSQL Database Algorithms

Hyperloglog Algorithm: https://odino.org/my-favorite-data-structure-hyperloglog/

Log Structured Merge Tree: https://www.cs.umb.edu/~poneil/lsmtree.pdf

Sorted String Tables and Compaction Strategies: https://github.com/scylladb/scylla/wiki/SSTable-compaction-and-compaction-strategies

Leveled Compaction Cassandra: https://www.datastax.com/blog/leveled-compaction-apache-cassandra

Scylla DB Compaction: https://github.com/scylladb/scylla/wiki/SSTable-compaction-and-compaction-strategies

Indexing in Cassandra: https://www.bmc.com/blogs/cassandra-clustering-columns-partition-composite-key/

Database Replication

Database replication: https://dev.mysql.com/doc/refman/8.0/en/replication.html

Netflix Data replication - Change Data Capture: https://netflixtechblog.com/dblog-a-generic-change-data-capture-framework-69351fb9099b

LinkedIn Logging Usecases: https://engineering.linkedin.com/distributed-systems/log-what-every-software-engineer-should-know-about-real-time-datas-unifying

Containers and Docker

Facebook Twine Containerization: https://engineering.fb.com/developer-tools/zookeeper-twine/

CloudFlare Containerization: https://blog.cloudflare.com/cloud-computing-without-containers/

Docker Architecture: https://docs.docker.com/get-started/overview/#docker-architecture

Capacity Estimation

Google Capacity Estimation: https://www.youtube.com/watch?v=modXC5IWTJI

Scalability at YouTube 2012: https://www.youtube.com/watch?v=G-lGCC4KKok

Back of envelope Calculations at AWS: https://www.youtube.com/watch?v=-3qetLv2Yp0

Capacity Estimation: http://static.googleusercontent.com/media/research.google.com/en//people/jeff/stanford-295-talk.pdf

Publisher Subscriber

Oracle Publisher Subscriber: https://docs.oracle.com/cd/B10501_01/appdev.920/a96590/adg15pub.htm

Amazon Pub Sub Messaging: https://aws.amazon.com/pub-sub-messaging/

Asynchronous processing: http://blog.codepath.com/2013/01/06/asynchronous-processing-in-web-applications-part-2-developers-need-to-understand-message-queues/

Async Request Response: https://www.enterpriseintegrationpatterns.com/patterns/conversation/RequestResponse.html

Event Driven Architectures

Martin Fowler- Event Driven Architecture: https://www.youtube.com/watch?v=STKCRSUsyP0

Event Driven Architecture: https://martinfowler.com/articles/201701-event-driven.html

Software Architectures

Hexagonal Architecture: https://netflixtechblog.com/ready-for-changes-with-hexagonal-architecture-b315ec967749

Hexagonal architecture (Alistair Cockburn) https://alistair.cockburn.us/hexagonal-architecture/

The Clean Code by Robert C. Martin (Uncle Bob) https://blog.cleancoder.com/uncle-bob/2012/08/13/the-clean-architecture.html

CQRS https://martinfowler.com/bliki/CQRS.html

DomainDrivenDesign https://martinfowler.com/bliki/DomainDrivenDesign.html

Microservices

Monolith Architecture: https://buttercms.com/books/microservices-for-startups/should-you-always-start-with-a-monolith/

Monoliths vs Microservices: https://articles.microservices.com/monolithic-vs-microservices-architecture-5c4848858f59

Microservices: http://highscalability.com/blog/2018/4/5/do-you-have-too-many-microservices-five-design-attributes-th.html

Uber Nanoservices antipattern: https://www.youtube.com/watch?v=kb-m2fasdDY

Uber Domain oriented microservice: https://eng.uber.com/microservice-architecture/

Distributed Transactions consistency Patterns

Transactional outbox https://microservices.io/patterns/data/transactional-outbox.html

SAGAS Long lived transactions (LLTs) https://www.cs.cornell.edu/andru/cs711/2002fa/reading/sagas.pdf

Load Balancing

Load Balancer with Sticky Sessions: https://stackoverflow.com/questions/10494431/sticky-and-non-sticky-sessions

Citrix what is load balancing: https://www.citrix.com/en-in/solutions/app-delivery-and-security/load-balancing/what-is-load-balancing.html

Nginx Load Balancing: https://www.nginx.com/resources/glossary/load-balancing/

Consistent hashing: https://michaelnielsen.org/blog/consistent-hashing/

Alerts and Anomaly Detection

Outlier Detection: https://towardsdatascience.com/outlier-detection-with-isolation-forest-3d190448d45e

Anomaly Detection: https://towardsdatascience.com/machine-learning-for-anomaly-detection-and-condition-monitoring-d4614e7de770

Uber Real Time Monitoring and Root Cause Analysis Argos: https://eng.uber.com/argos-real-time-alerts/

Microsoft Anomaly Detection: https://www.youtube.com/watch?v=12Xq9OLdQwQ&t=0s

Facebook Data Engineering: https://engineering.fb.com/2016/05/09/core-data/introducing-fblearner-flow-facebook-s-ai-backbone/

LinkedIn Real Time Alerting: https://engineering.linkedin.com/blog/2019/06/smart-alerts-in-thirdeye--linkedins-real-time-monitoring-platfor

LinkedIn Isolation Forests: https://engineering.linkedin.com/blog/2019/isolation-forest

Distributed Logging

Uber Distributed Request Tracing: https://eng.uber.com/distributed-tracing/

Pintrest Logging: https://medium.com/@Pinterest_Engineering/open-sourcing-singer-pinterests-performant-and-reliable-logging-agent-610fecf35566

Google Monitoring Infrastructure: https://www.facebook.com/atscaleevents/videos/959344524420015/

Metrics and Text Search Engine

Facebook real time text search engine: https://www.facebook.com/watch/?v=432864835468

Elastic Search Time Based Querying: https://www.elastic.co/guide/en/elasticsearch/guide/current/time-based.html

Elastic Search Aggregation: https://www.elastic.co/guide/en/elasticsearch/guide/current/aggregations.html

Single Point of Failure

Avoiding Single Points of Failure: https://medium.com/the-cloud-architect/patterns-for-resilient-architecture-part-3-16e8601c488e

Netflix Multi-Region Availability: https://netflixtechblog.com/active-active-for-multi-regional-resiliency-c47719f6685b

Oracle Single Points of failure: https://docs.oracle.com/cd/E19693-01/819-0992/fjdch/index.html

DNS single point of failure 2004: http://www.tenereillo.com/GSLBPageOfShame.htm

Sharding: https://medium.com/@jeeyoungk/how-sharding-works-b4dec46b3f6

Location Based Services

Google S2 library: https://blog.christianperone.com/2015/08/googles-s2-geometry-on-the-sphere-cells-and-hilbert-curve/

Batch Processing

Map Reduce Architecture: https://static.googleusercontent.com/media/research.google.com/en//archive/mapreduce-osdi04.pdf

Real Time Stream Processing

LinkedIn Brooklin- Real time data streaming: https://engineering.linkedin.com/blog/2019/brooklin-open-source

Netflix Real Time Stream Processing: https://netflixtechblog.com/keystone-real-time-stream-processing-platform-a3ee651812a

KSQLDB for Kafka: https://docs.ksqldb.io/en/latest/operate-and-deploy/how-it-works/

Caching

Google Guava Cache: https://github.com/google/guava/wiki/CachesExplained

Caching (See the README): https://github.com/ben-manes/caffeine/

Caching: http://highscalability.com/blog/2016/1/25/design-of-a-modern-cache.html

Microsoft Caching Guide: https://docs.microsoft.com/en-us/previous-versions/msp-n-p/dn589802(v%3dpandp.10)

Caching patterns: https://hazelcast.com/blog/a-hitchhikers-guide-to-caching-patterns/

Distributed concensus

Paxos: http://ifeanyi.co/posts/understanding-consensus/

Raft: https://raft.github.io/

Authorization

Designing an Authorization Model for an Enterprise: https://cerbos.dev/blog/designing-an-authorization-model-for-an-enterprise

The Architectural Patterns of Cloud-native Authorization Systems: https://www.aserto.com/blog/5-laws-cloud-native-authorization

Content Delivery Network

AWS CloudFront CDN with S3: https://aws.amazon.com/blogs/networking-and-content-delivery/amazon-s3-amazon-cloudfront-a-match-made-in-the-cloud/

Testing Distributed Systems

Deterministic Testing: https://www.youtube.com/watch?v=4fFDFbi3toc

System Design Resources

Designing Data Intensive Applications Book: https://amzn.to/3SyNAOy

WhitePapers: https://interviewready.io/blog/white-papers-worth-reading-for-software-engineers

InterviewReady Videos: https://interviewready.io

About

These are the best resources for System Design on the Internet

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published