Cache

Introduction

In many applications, we may encounter scenarios where overlapping work is performed. For example, if we are developing a service that handles incoming requests, it is essential to avoid processing duplicate requests. By using the Cache module, we can enhance our application’s performance by preventing redundant work.

Key Features of Cache:

• Compositionality: Cache allows different parts of our application to perform overlapping work while still benefiting from compositional programming principles.

• Unification of Synchronous and Asynchronous Caches: The compositional definition of a cache through a lookup function unifies both synchronous and asynchronous caches, allowing the lookup function to compute values either synchronously or asynchronously.

• Deep Effect Integration: Cache is designed to work natively with the Effect library, supporting concurrent lookups, failure handling, and interruption without losing the power of Effect.

• Cache/Entry Statistics: Cache tracks metrics such as entries, hits, misses, helping us to assess and optimize cache performance.

How to Define a Cache?

A cache is defined by a lookup function that describes how to compute the value associated with a key if it is not already in the cache.

type Lookup<Key, Value, Error = never, Environment = never> = (
  key: Key
) => Effect.Effect<Value, Error, Environment>

The lookup function takes a key of type Key and returns an Effect that requires an environment of type Environment and can fail with an error of type Error or succeed with a value of type Value. Since the lookup function returns an Effect, it can describe both synchronous and asynchronous workflows.

In short, if you can describe it with an Effect, you can use it as the lookup function for a cache.

We construct a cache using a lookup function along with a maximum size and a time to live.

declare const make: <
  Key,
  Value,
  Error = never,
  Environment = never
>(options: {
  readonly capacity: number
  readonly timeToLive: Duration.DurationInput
  readonly lookup: Lookup<Key, Value, Error, Environment>
}) => Effect.Effect<Cache<Key, Value, Error>, never, Environment>

Once a cache is created, the most idiomatic way to work with it is the get operator. The get operator returns the current value in the cache if it exists, or computes a new value, puts it in the cache, and returns it.

If multiple concurrent processes request the same value, it will only be computed once. All other processes will receive the computed value as soon as it is available. This is managed using Effect’s fiber-based concurrency model without blocking the underlying thread.

Example

In this example, we call timeConsumingEffect three times in parallel with the same key. The Cache runs this effect only once, so concurrent lookups will wait until the value is available:

import { Effect, Cache, Duration } from "effect"

const timeConsumingEffect = (key: string) =>
  Effect.sleep("2 seconds").pipe(Effect.as(key.length))

const program = Effect.gen(function* () {
  const cache = yield* Cache.make({
    capacity: 100,
    timeToLive: Duration.infinity,
    lookup: timeConsumingEffect
  })
  const result = yield* cache
    .get("key1")
    .pipe(
      Effect.zip(cache.get("key1"), { concurrent: true }),
      Effect.zip(cache.get("key1"), { concurrent: true })
    )
  console.log(
    "Result of parallel execution of three effects" +
      `with the same key: ${result}`
  )

  const hits = yield* cache.cacheStats.pipe(
    Effect.map((stats) => stats.hits)
  )
  const misses = yield* cache.cacheStats.pipe(
    Effect.map((stats) => stats.misses)
  )
  console.log(`Number of cache hits: ${hits}`)
  console.log(`Number of cache misses: ${misses}`)
})

Effect.runPromise(program)
/*
Output:
Result of parallel execution of three effects with the same key: 4,4,4
Number of cache hits: 2
Number of cache misses: 1
*/

Concurrent Access

The cache is designed to be safe for concurrent access and efficient under concurrent conditions. If two concurrent processes request the same value and it is not in the cache, the value will be computed once and provided to both processes as soon as it is available. Concurrent processes will wait for the value without blocking operating system threads.

If the lookup function fails or is interrupted, the error will be propagated to all concurrent processes waiting for the value. Failures are cached to prevent repeated computation of the same failed value. If interrupted, the key will be removed from the cache, so subsequent calls will attempt to compute the value again.

Capacity

A cache is created with a specified capacity. When the cache reaches capacity, the least recently accessed values will be removed first. The cache size may slightly exceed the specified capacity between operations.

Time To Live (TTL)

A cache can also have a specified time to live (TTL). Values older than the TTL will not be returned. The age is calculated from when the value was loaded into the cache.

Operators

In addition to get, Cache provides several other operators:

• refresh: Similar to get, but triggers a re-computation of the value without invalidating it, allowing requests to the associated key to be served while the value is being re-computed.
• size: Returns the current size of the cache. Under concurrent access, the size is approximate.
• contains: Checks if a value associated with a specified key exists in the cache. Under concurrent access, the result is valid as of the check time but may change immediately after.
• invalidate: Evicts the value associated with a specified key.
• invalidateAll: Evicts all values in the cache.