Caching in System Design: The Fundamentals

#

Caching is one of the most important techniques used to improve the performance and scalability of distributed systems.

Almost every large-scale system uses caching somewhere.

A typical system might look like:

Client
  ↓
Browser Cache
  ↓
CDN
  ↓
Load Balancer
  ↓
Application
  ↓
Local Cache
  ↓
Distributed Cache
  ↓
Database
  ↓
Storage

The fundamental idea is simple:

Store frequently accessed data closer to where it is needed so that future requests can be served faster and more cheaply.

But caching is much more than adding Redis to an architecture.

A proper caching design requires us to answer questions such as:

• What should we cache?
• Where should we cache it?
• How long should we cache it?
• How do we know when cached data is stale?
• What happens when the cache is full?
• What happens when the cache goes down?
• How do we prevent millions of requests from hitting the database simultaneously?
• How do we distribute cached data across multiple machines?
• What consistency guarantees do we need?
• How much memory do we need?

This article builds the mental model required to answer those questions.

1. Why Do We Need Caching?

#

Consider a simple application:

Client
  ↓
Application Server
  ↓
Database

Suppose the database takes 20 ms to execute a query.

Now imagine, 10,000 requests / second. If every request goes directly to the database - 10,000 database queries / second.

The database becomes the bottleneck.

But suppose 90% of those requests ask for the same data.

For example:

GET /products/123

Instead of querying the database every time, we can store the result in a cache:

             +-------+
             | Cache |
             +-------+
                ↑
                |
Client → Application
                |
                ↓
             Database

Now:

90% → Cache
10% → Database

The database might only need to handle around 1,000 requests / second, instead of 10,000 requests / second.

Caching provides three major benefits.

Lower latency

#

Memory-based access is generally much faster than querying a database or making a remote service call.

Reduced load

#

Requests served from the cache do not reach the underlying database.

Better scalability

#

We can handle significantly more requests without scaling the database proportionally.

But caching also introduces a new problem:

We now have another copy of our data that must be managed correctly.

2. What Is a Cache?

#

A cache is a temporary storage layer that stores data that can be retrieved or reconstructed from another source.

For example:

Application
     ↓
   Redis
     ↓
 PostgreSQL

PostgreSQL is the source of truth.

Redis contains a derived copy.

This distinction is extremely important.

Database
   ↓
Source of Truth

Cache
   ↓
Derived State

If Redis disappears, we should ideally be able to reconstruct its contents from PostgreSQL.

Therefore:

A cache should generally be disposable.

If losing the cache means permanently losing critical data, then the system is no longer simply using a cache. We are effectively treating the cache as a database.

3. The Core Caching Model

#

A useful mental model is:

             Request
                ↓
             Cache
             /   \
          Hit     Miss
           ↓        ↓
         Return   Source
                    ↓
                  Cache
                    ↓
                  Return

A cache hit occurs when the requested data is already available in the cache.

A cache miss occurs when it is not.

For example:

GET user:123

Cache hit

#

Application
    ↓
Cache
    ↓
User data

Cache miss

#

Application
    ↓
Cache
    ↓
MISS
    ↓
Database
    ↓
User data
    ↓
Cache

This simple flow forms the basis of many caching architectures.

4. Cache Hit and Cache Miss

#

One of the most important metrics in caching is the cache hit rate.

Suppose we have:

900 cache hits
100 cache misses

Total requests:

1,000

Therefore:

Cache Hit Rate = 900 / 1000 = 90%

The cache hit rate tells us how often the cache successfully avoids accessing the underlying system.

The opposite is the cache miss rate:

Cache Miss Rate = 1 - Cache Hit Rate

So if:

Hit Rate = 90%

then:

Miss Rate = 10%

A high hit rate is generally desirable, but it should not be treated as the only measure of cache effectiveness.

For example:

99% hit rate

sounds excellent.

But if the remaining 1% represents extremely expensive database queries, the system may still have a serious performance problem.

We therefore need to look at:

• hit rate
• miss rate
• cache latency
• database load
• eviction rate
• memory usage
• request distribution

5. The Working Set

#

A database might contain billions of records.

But users may only access a small percentage of them frequently.

For example:

Database
1 billion records

But perhaps:

10 million records

receive 95% of all requests.

Those frequently accessed records represent the system’s working set.

Database
│
├── Rarely accessed data
│
├── Rarely accessed data
│
├── Rarely accessed data
│
└── Frequently accessed data
         ↑
         |
      Working Set

A cache does not necessarily need to contain the entire database.

The goal is often to keep the most valuable part of the working set in memory.

This is one of the most important ideas in cache design.

You don’t necessarily need to cache everything. You need to cache the right things.

6. What Makes Data a Good Candidate for Caching?

#

A good caching candidate usually has some combination of these characteristics:

Frequently Read
       +
Expensive to Obtain
       +
Relatively Stable

For example:

Product metadata

#

Product ID
Name
Description
Price
Images

Frequently accessed and relatively stable.

Configuration

#

Feature flags
Application configuration
Service configuration

Often read many times but changed infrequently.

Expensive computation

#

Recommendation generation
Analytics aggregation
Machine learning prediction

If computing the same result repeatedly is expensive, caching can be extremely valuable.

7. What Is a Bad Candidate for Caching?

#

Caching is not automatically beneficial.

Data may be a poor candidate when:

• it changes constantly
• it is accessed only once
• it is extremely large
• it has extremely high cardinality
• it requires strong consistency
• the cache entry costs more to maintain than the computation itself

For example: Current bank balance may have very different consistency requirements from Popular news article. The news article might tolerate 5 seconds of staleness, while a financial transaction may not.

Therefore:

The decision to cache is fundamentally a business and consistency decision, not just a performance decision.

8. Freshness vs Performance

#

Caching creates a fundamental trade-off:

Freshness
    ↑
    |
    |
    +----------------→ Performance

The more aggressively we cache, the less frequently we may contact the source of truth.

That improves performance.

But the cached value may become stale.

For example:

Database
Price = ₹100

Cache:

Price = ₹100

Then the database changes:

Database
Price = ₹120

But the cache still contains:

Price = ₹100

Now the cache is stale.

The system designer has to decide:

How much stale data can the business tolerate?

Possible answers might be:

0 seconds
1 second
10 seconds
5 minutes
1 hour
Eventually consistent

This requirement strongly influences the caching strategy.

9. Cache Location Matters

#

Caching can happen at many different layers.

A complete request path could look like:

User
  ↓
Browser Cache
  ↓
CDN
  ↓
Load Balancer
  ↓
Application
  ↓
Local Cache
  ↓
Distributed Cache
  ↓
Database
  ↓
Disk

Each layer has different properties.

The closer the cache is to the consumer:

Lower latency

but generally:

Smaller capacity

For example:

CPU Cache
    ↓
Extremely fast
Very small

RAM
    ↓
Fast
Larger

SSD
    ↓
Slower
Much larger

Database
    ↓
Even more expensive
Persistent

This gives us a general principle:

Fast storage tends to be smaller and more expensive per byte. Slow storage tends to be larger and cheaper per byte.

Caching allows us to keep the most valuable data in the faster layer.

10. Local Cache vs Distributed Cache

#

One of the first architectural decisions is where the cache should live.

Local Cache

#

The cache lives inside the application process.

Application Server
       |
       v
   Memory Cache

For example:

Map<String, User> cache;

Advantages:

• extremely low latency
• no network call
• simple
• no external infrastructure

But suppose we have three application servers:

+---------+     +---------+     +---------+
| App 1   |     | App 2   |     | App 3   |
| Cache   |     | Cache   |     | Cache   |
+---------+     +---------+     +---------+

Each server has its own copy.

This creates several problems:

• duplicated memory
• inconsistent values
• independent cache warming
• difficult invalidation
• different hit rates

11. Distributed Cache

#

A distributed cache is shared by multiple application instances.

             +----------------+
             | Redis Cluster  |
             +----------------+
               ↑      ↑     ↑
               |      |     |
              App1   App2   App3

All application instances can access the same cache.

Advantages:

• shared state
• larger aggregate capacity
• consistent cache population
• easier coordination

But now every cache access requires a network call.

Therefore:

Local Cache
     ↓
Fastest

Distributed Cache
     ↓
Fast, but network dependent

Database
     ↓
More expensive

12. Multi-Level Caching

#

Large systems often use both local and distributed caches.

For example:

Request
   ↓
Local Cache
   |
   | HIT
   ↓
Return

   |
   | MISS
   ↓
Distributed Cache
   |
   | HIT
   ↓
Return

   |
   | MISS
   ↓
Database

This creates multiple layers:

L1 → Local Cache
L2 → Distributed Cache
L3 → Database

The hottest data can be kept in the local cache.

The broader working set can live in the distributed cache.

The database remains the source of truth.

13. Caching Is Not Just Redis

#

When engineers hear “cache”, they often immediately think:

Redis

But caching is a much broader system-design concept.

Examples include:

CPU Cache
OS Page Cache
Browser Cache
CDN
Application Cache
Distributed Cache
Database Buffer Cache
DNS Cache
API Response Cache
Query Cache
Computation Cache
Semantic Cache

Redis is simply one technology used to implement a particular kind of cache.

The more important question is:

Where is the expensive operation happening, and can we avoid repeating it?

14. Caching as Memoization at Scale

#

At a programming level, memoization means:

Input
  ↓
Expensive Function
  ↓
Result

Instead of recomputing:

Input
  ↓
Check previous result
  ↓
If available → Return
If unavailable → Compute → Store

Caching is essentially the same idea applied at system scale.

For example:

calculateRecommendation(user123)

might take:

500 ms

If the result is valid for five minutes, we can store:

recommendation:user123

and reuse it.

Therefore:

Caching is often system-level memoization.

15. Caching Reduces More Than Latency

#

Caching is usually introduced because something is slow.

But the benefits go beyond latency.

Reduce database CPU

#

Fewer queries need to be executed.

Reduce database connections

#

Fewer requests need database connections.

Reduce network traffic

#

Less data travels between services.

Reduce infrastructure cost

#

Fewer database resources may be required.

Improve availability

#

The application may continue serving cached data even when the source is temporarily unhealthy.

For example:

Database temporarily slow
        ↓
Cache still healthy
        ↓
Popular data continues to be served

However, this comes with the possibility of serving stale data.

16. The Cache as a Shock Absorber

#

A useful way to think about caching is as a shock absorber between the application and expensive dependencies.

Without caching:

100,000 requests/sec
        ↓
100,000 database requests/sec

With caching:

100,000 requests/sec
        ↓
      Cache
        ↓
  95,000 requests served
        ↓
  5,000 database requests

The cache absorbs a large portion of the traffic.

This protects the database from traffic spikes.

But there is an important caveat:

The cache itself can become a bottleneck if it is not designed for the traffic pattern.

This becomes especially important with distributed caches, hot keys, and cache failures.

17. Cache Latency

#

When evaluating a cache, we should measure its latency.

For example:

Local Cache
    ↓
~nanoseconds / microseconds

Distributed Cache
    ↓
~sub-millisecond to milliseconds

Database
    ↓
milliseconds to tens/hundreds of milliseconds

The exact numbers depend heavily on:

• hardware
• network
• deployment topology
• payload size
• serialization
• database workload

The important concept is the relative difference.

If the cache is:

5 ms

and the database is:

10 ms

the benefit may be modest.

But if the cache is:

0.5 ms

and the database is:

50 ms

the benefit can be substantial.

18. Cache Hit Rate Is Not Everything

#

Consider two systems.

System A

#

Hit Rate = 99%
Cache Latency = 10 ms

System B

#

Hit Rate = 95%
Cache Latency = 0.5 ms

System B might still provide excellent overall performance.

Similarly, consider:

99% hit rate

where the remaining 1% generates enormous database queries.

Therefore, when evaluating a cache, look at the complete picture:

                Cache
                  |
      +-----------+-----------+
      |           |           |
   Hit Rate    Latency      Miss Cost
      |           |           |
      +-----------+-----------+
                  |
            Database Load

19. Cache Cost

#

Caching is not free.

A cache introduces:

• infrastructure
• memory cost
• network traffic
• serialization
• operational complexity
• invalidation complexity
• monitoring
• failure scenarios

Therefore the real question is:

Does the performance and scalability benefit justify the additional complexity and cost?

Sometimes the answer is no.

For example, if a database query takes:

2 ms

and is executed only:

100 times/sec

introducing a distributed cache might not be worth it.

But if the query takes:

100 ms

and receives:

50,000 requests/sec

caching may be transformative.

20. The Fundamental Caching Trade-offs

#

Most caching decisions can be reduced to a few fundamental trade-offs.

Freshness vs Performance

#

Long TTL
    ↓
Better performance
Potentially stale data

Short TTL
    ↓
Fresher data
More source requests

Local vs Distributed

#

Local
    ↓
Faster
Harder consistency

Distributed
    ↓
Shared
Network overhead

Cache Size vs Cost

#

Larger Cache
    ↓
Higher hit rate
Higher memory cost

Complexity vs Optimization

#

Simple Cache
    ↓
Easy to operate
Less optimization

Sophisticated Cache
    ↓
Better optimization
Higher operational complexity

21. The Most Important Questions in Cache Design

#

Whenever you introduce a cache into a system, ask these questions:

1. What are we caching?

#

Object?
Query?
API response?
Computation?
Static asset?

2. Why are we caching it?

#

Reduce latency?
Reduce DB load?
Reduce cost?
Improve availability?

3. Where should it live?

#

Browser?
CDN?
Local memory?
Distributed cache?

4. How is it populated?

#

Lazy?
Proactive?
On write?
On read?

5. How long is it valid?

#

Seconds?
Minutes?
Hours?
Forever?

6. How is it invalidated?

#

TTL?
Explicit delete?
Update?
Event?
Version?

7. What happens on a miss?

#

Database?
Another service?
Recompute?

8. What happens when the cache is full?

#

LRU?
LFU?
TTL?
Random?

9. What happens when the cache fails?

#

Fallback?
Fail open?
Fail closed?
Circuit breaker?

10. What happens during a traffic spike?

#

Hot key?
Stampede?
Avalanche?
Database overload?

These questions form the foundation of almost every caching discussion in system design.

22. A Simple Mental Model

#

Whenever you see a slow dependency in a system, ask:

Can I avoid calling it?
        ↓
Can I reuse a previous result?
        ↓
Can I store that result somewhere faster?
        ↓
How long can I safely reuse it?
        ↓
How do I know it is stale?
        ↓
What happens when it expires?
        ↓
What happens if millions of requests discover that it expired simultaneously?
        ↓
What happens if the cache itself fails?

This sequence takes us from:

"Let's add Redis."

to:

Actual System Design.

Conclusion

#

Caching is one of the most powerful techniques for improving the performance, scalability, and resilience of distributed systems.

But caching is not simply:

Application → Redis → Database

A good caching design requires us to understand:

What to cache
Where to cache
When to cache
How to populate it
How long to keep it
How to invalidate it
How to evict it
How to scale it
How to handle failures
How to measure it

The most important idea to remember is:

Caching is about avoiding expensive work by safely reusing a previous result.

Once we understand that principle, the same idea appears everywhere:

CPU Cache
OS Cache
Browser Cache
CDN
Application Cache
Distributed Cache
Database Cache
API Cache
Query Cache
Computation Cache
Semantic Cache

The technology changes.

The fundamental idea remains the same.

What’s Next?

#

In Part 2, we will look at the different types of caches in detail — from CPU caches and browser caches to CDNs, application caches, distributed caches, and database caches — and understand why each exists, where it sits in the request path, and what trade-offs it introduces.

Next → Types of Caches: From CPU to CDN