Building Pipelines

Effect pipelines allow for the composition and sequencing of operations on values, enabling the transformation and manipulation of data in a concise and modular manner.

Why Pipelines are Good for Structuring Your Application

Pipelines are an excellent way to structure your application and handle data transformations in a concise and modular manner. They offer several benefits:

1. Readability: Pipelines allow you to compose functions in a readable and sequential manner. You can clearly see the flow of data and the operations applied to it, making it easier to understand and maintain the code.

2. Code Organization: With pipelines, you can break down complex operations into smaller, manageable functions. Each function performs a specific task, making your code more modular and easier to reason about.

3. Reusability: Pipelines promote the reuse of functions. By breaking down operations into smaller functions, you can reuse them in different pipelines or contexts, improving code reuse and reducing duplication.

4. Type Safety: By leveraging the type system, pipelines help catch errors at compile-time. Functions in a pipeline have well-defined input and output types, ensuring that the data flows correctly through the pipeline and minimizing runtime errors.

Functions vs Methods

The use of functions in the Effect ecosystem libraries is important for achieving tree shakeability and ensuring extensibility. Functions enable efficient bundling by eliminating unused code, and they provide a flexible and modular approach to extending the libraries’ functionality.

Tree Shakeability

Tree shakeability refers to the ability of a build system to eliminate unused code during the bundling process. Functions are tree shakeable, while methods are not.

When functions are used in the Effect ecosystem, only the functions that are actually imported and used in your application will be included in the final bundled code. Unused functions are automatically removed, resulting in a smaller bundle size and improved performance.

On the other hand, methods are attached to objects or prototypes, and they cannot be easily tree shaken. Even if you only use a subset of methods, all methods associated with an object or prototype will be included in the bundle, leading to unnecessary code bloat.

Extensibility

Another important advantage of using functions in the Effect ecosystem is the ease of extensibility. With methods, extending the functionality of an existing API often requires modifying the prototype of the object, which can be complex and error-prone.

In contrast, with functions, extending the functionality is much simpler. You can define your own “extension methods” as plain old functions without the need to modify the prototypes of objects. This promotes cleaner and more modular code, and it also allows for better compatibility with other libraries and modules.

pipe

The pipe function is a utility that allows us to compose functions in a readable and sequential manner. It takes the output of one function and passes it as the input to the next function in the pipeline. This enables us to build complex transformations by chaining multiple functions together.

The basic syntax of pipe is as follows:

import { pipe } from "effect"

const result = pipe(input, func1, func2, ..., funcN)

In this syntax, input is the initial value, and func1, func2, …, funcN are the functions to be applied in sequence. The result of each function becomes the input for the next function, and the final result is returned.

Here’s an illustration of how pipe works:

It’s important to note that functions passed to pipe must have a single argument because they are only called with a single argument.

Let’s see an example to better understand how pipe works:

import { pipe } from "effect"

// Define simple arithmetic operations
const increment = (x: number) => x + 1
const double = (x: number) => x * 2
const subtractTen = (x: number) => x - 10

// Sequentially apply these operations using `pipe`
const result = pipe(5, increment, double, subtractTen)

console.log(result) // Output: 2

In the above example, we start with an input value of 5. The increment function adds 1 to the initial value, resulting in 6. Then, the double function doubles the value, giving us 12. Finally, the subtractTen function subtracts 10 from 12, resulting in the final output of 2.

The result is equivalent to subtractTen(double(increment(5))), but using pipe makes the code more readable because the operations are sequenced from left to right, rather than nesting them inside out.

map

The Effect.map function is used to transform the value inside an effect. It takes a function and applies it to the value contained within the effect, creating a new effect with the transformed value.

The syntax for Effect.map is as follows:

import { pipe, Effect } from "effect"

const mappedEffect = pipe(myEffect, Effect.map(transformation))
// or
const mappedEffect = Effect.map(myEffect, transformation)
// or
const mappedEffect = myEffect.pipe(Effect.map(transformation))

In the code above, transformation is the function applied to the value, and myEffect is the effect being transformed.

Effects are Immutable

It’s important to note that effects are immutable, meaning that when you use Effect.map on an effect, it doesn’t modify the original data type. Instead, it returns a new copy of the effect with the transformed value.

Example

Consider a program that adds a small service charge to a transaction:

import { pipe, Effect } from "effect"

// Function to add a small service charge to a transaction amount
const addServiceCharge = (amount: number) => amount + 1

// Simulated asynchronous task to fetch a transaction amount from database
const fetchTransactionAmount = Effect.promise(() => Promise.resolve(100))

// Apply service charge to the transaction amount
const finalAmount = pipe(
  fetchTransactionAmount,
  Effect.map(addServiceCharge)
)

Effect.runPromise(finalAmount).then(console.log) // Output: 101

as

To map an Effect to a constant value, replacing the original value, use Effect.as:

import { pipe, Effect } from "effect"

const program = pipe(Effect.succeed(5), Effect.as("new value"))

Effect.runPromise(program).then(console.log) // Output: "new value"

flatMap

The Effect.flatMap function is used when you need to chain transformations that produce Effect instances. This is useful for asynchronous operations or computations that depend on the results of previous effects.

The Effect.flatMap function enables you to sequence computations that result in new Effect values, “flattening” any nested effect structures that arise.

The syntax for Effect.flatMap is as follows:

import { pipe, Effect } from "effect"

const flatMappedEffect = pipe(myEffect, Effect.flatMap(transformation))
// or
const flatMappedEffect = Effect.flatMap(myEffect, transformation)
// or
const flatMappedEffect = myEffect.pipe(Effect.flatMap(transformation))

In the code above, transformation is the function that takes a value and returns an Effect, and myEffect is the initial Effect being transformed.

Effects are Immutable

It’s important to note that effects are immutable, meaning that when you use Effect.flatMap on an effect, it doesn’t modify the original data type. Instead, it returns a new copy of the effect with the transformed value.

Example

import { pipe, Effect } from "effect"

// Function to apply a discount safely to a transaction amount
const applyDiscount = (
  total: number,
  discountRate: number
): Effect.Effect<number, Error> =>
  discountRate === 0
    ? Effect.fail(new Error("Discount rate cannot be zero"))
    : Effect.succeed(total - (total * discountRate) / 100)

// Simulated asynchronous task to fetch a transaction amount from database
const fetchTransactionAmount = Effect.promise(() => Promise.resolve(100))

const finalAmount = pipe(
  fetchTransactionAmount,
  Effect.flatMap((amount) => applyDiscount(amount, 5))
)

Effect.runPromise(finalAmount).then(console.log) // Output: 95

Ensuring All Effects Are Considered

It’s vital to ensure that all effects within Effect.flatMap contribute to the final computation. Neglecting any effect can lead to unexpected behaviors or incorrect outcomes:

Effect.flatMap((amount) => {
  // This effect will be ignored
  Effect.sync(() => console.log(`Apply a discount to: ${amount}`))
  return applyDiscount(amount, 5)
})

The Effect.sync above is ignored and does not influence the result of applyDiscount(amount, 5). To include effects properly and avoid errors, explicitly chain them using functions like Effect.map, Effect.flatMap, Effect.andThen, or Effect.tap.

Further Information on flatMap

Although many developers may recognize flatMap from its usage with arrays, in the Effect framework, it’s utilized to manage and resolve nested Effect structures. If your goal is to flatten nested arrays within an Effect (Effect<Array<Array<A>>>), this can be done using:

import { pipe, Effect, Array } from "effect"

const flattened = pipe(
  Effect.succeed([
    [1, 2],
    [3, 4]
  ]),
  Effect.map((nested) => Array.flatten(nested))
)

or using the standard Array.prototype.flat() method.

andThen

Both the Effect.map and Effect.flatMap functions serve to transform an effect into another effect in two different scenarios. In the first scenario, Effect.map is used when the transformation function does not return an effect, while in the second scenario, Effect.flatMap is used when the transformation function still returns an effect.

However, since both scenarios involve transformations, the Effect module also exposes a convenient all-in-one solution to use: Effect.andThen.

The Effect.andThen function executes a sequence of two actions, typically two effects, where the second action can depend on the result of the first action.

import { pipe, Effect } from "effect"

const transformedEffect = pipe(myEffect, Effect.andThen(anotherEffect))
// or
const transformedEffect = Effect.andThen(myEffect, anotherEffect)
// or
const transformedEffect = myEffect.pipe(Effect.andThen(anotherEffect))

The anotherEffect action can take various forms:

1. a value (i.e. same functionality of Effect.as)
2. a function returning a value (i.e. same functionality of Effect.map)
3. a Promise
4. a function returning a Promise
5. an Effect
6. a function returning an Effect(i.e. same functionality of Effect.flatMap)

Example

Let’s see an example where we can compare the use of Effect.andThen instead of Effect.map and Effect.flatMap:

import { pipe, Effect } from "effect"

// Function to apply a discount safely to a transaction amount
const applyDiscount = (
  total: number,
  discountRate: number
): Effect.Effect<number, Error> =>
  discountRate === 0
    ? Effect.fail(new Error("Discount rate cannot be zero"))
    : Effect.succeed(total - (total * discountRate) / 100)

// Simulated asynchronous task to fetch a transaction amount from database
const fetchTransactionAmount = Effect.promise(() => Promise.resolve(100))

// Using Effect.map, Effect.flatMap
const result1 = pipe(
  fetchTransactionAmount,
  Effect.map((amount) => amount * 2),
  Effect.flatMap((amount) => applyDiscount(amount, 5))
)

Effect.runPromise(result1).then(console.log) // Output: 190

// Using Effect.andThen
const result2 = pipe(
  fetchTransactionAmount,
  Effect.andThen((amount) => amount * 2),
  Effect.andThen((amount) => applyDiscount(amount, 5))
)

Effect.runPromise(result2).then(console.log) // Output: 190

It’s worth noting that Option and Either, types commonly used to handle optional values and simple error scenarios, are also compatible with Effect.andThen. However, it is important to understand that when these types are used, the operations are categorized under scenarios 5 and 6 as described previously, as both Option and Either operate as Effects in this context.

Example with Option

import { pipe, Effect, Option } from "effect"

// Simulated asynchronous task fetching a number from a database
const fetchNumberValue = Effect.promise(() => Promise.resolve(42))

// Although one might expect the type to be
// Effect<Option<number>, never, never>, it is actually
// Effect<number, NoSuchElementException, never>
const program = pipe(
  fetchNumberValue,
  Effect.andThen((x) => (x > 0 ? Option.some(x) : Option.none()))
)

Option As Effect

A value of type Option<A> is interpreted as an effect of type Effect<A, NoSuchElementException>.

Example with Either

import { pipe, Effect, Either } from "effect"

// Function to parse an integer from a string that can fail
const parseInteger = (input: string): Either.Either<number, string> =>
  isNaN(parseInt(input))
    ? Either.left("Invalid integer")
    : Either.right(parseInt(input))

// Simulated asynchronous task fetching a string from database
const fetchStringValue = Effect.promise(() => Promise.resolve("42"))

// Although one might expect the type to be
// Effect<Either<number, string>, never, never>, it is actually
// Effect<number, string, never>
const program = pipe(
  fetchStringValue,
  Effect.andThen((str) => parseInteger(str))
)

Either As Effect

A value of type Either<A, E> is interpreted as an effect of type Effect<A, E>.

tap

The Effect.tap API has a similar signature to Effect.flatMap, but the result of the transformation function is ignored. This means that the value returned by the previous computation will still be available for the next computation.

Example

import { pipe, Effect } from "effect"

// Function to apply a discount safely to a transaction amount
const applyDiscount = (
  total: number,
  discountRate: number
): Effect.Effect<number, Error> =>
  discountRate === 0
    ? Effect.fail(new Error("Discount rate cannot be zero"))
    : Effect.succeed(total - (total * discountRate) / 100)

// Simulated asynchronous task to fetch a transaction amount from database
const fetchTransactionAmount = Effect.promise(() => Promise.resolve(100))

const finalAmount = pipe(
  fetchTransactionAmount,
  Effect.tap((amount) =>
    Effect.sync(() => console.log(`Apply a discount to: ${amount}`))
  ),
  // `amount` is still available!
  Effect.flatMap((amount) => applyDiscount(amount, 5))
)

Effect.runPromise(finalAmount).then(console.log)
/*
Output:
Apply a discount to: 100
95
*/

Using Effect.tap allows us to execute side effects during the computation without altering the result. This can be useful for logging, performing additional actions, or observing the intermediate values without interfering with the main computation flow.

all

The Effect.all function is a powerful utility provided by Effect that allows you to combine multiple effects into a single effect that produces a tuple of results.

The syntax for Effect.all is as follows:

import { Effect } from "effect"

const combinedEffect = Effect.all([effect1, effect2, ...])

The Effect.all function will execute all these effects in sequence (to explore options for managing concurrency and controlling how these effects are executed, you can refer to the Concurrency Options documentation).

It will return a new effect that produces a tuple containing the results of each individual effect. Keep in mind that the order of the results corresponds to the order of the original effects passed to Effect.all.

Example

import { Effect } from "effect"

// Simulated function to read configuration from a file
const webConfig = Effect.promise(() =>
  Promise.resolve({ dbConnection: "localhost", port: 8080 })
)

// Simulated function to test database connectivity
const checkDatabaseConnectivity = Effect.promise(() =>
  Promise.resolve("Connected to Database")
)

// Combine both effects to perform startup checks
const startupChecks = Effect.all([webConfig, checkDatabaseConnectivity])

Effect.runPromise(startupChecks).then(([config, dbStatus]) => {
  console.log(
    `Configuration: ${JSON.stringify(config)}\nDB Status: ${dbStatus}`
  )
})
/*
Output:
Configuration: {"dbConnection":"localhost","port":8080}
DB Status: Connected to Database
*/

Applicability

The Effect.all function not only combines tuples but also works with iterables, structs, and records. To explore the full potential of Effect.all head over to the Introduction to Effect’s Control Flow Operators documentation.

Build your first pipeline

Now, let’s combine pipe, Effect.all and Effect.andThen to build a pipeline that performs a series of transformations:

import { Effect, pipe } from "effect"

// Function to add a small service charge to a transaction amount
const addServiceCharge = (amount: number) => amount + 1

// Function to apply a discount safely to a transaction amount
const applyDiscount = (
  total: number,
  discountRate: number
): Effect.Effect<number, Error> =>
  discountRate === 0
    ? Effect.fail(new Error("Discount rate cannot be zero"))
    : Effect.succeed(total - (total * discountRate) / 100)

// Simulated asynchronous task to fetch a transaction amount from database
const fetchTransactionAmount = Effect.promise(() => Promise.resolve(100))

// Simulated asynchronous task to fetch a discount rate
// from a configuration file
const fetchDiscountRate = Effect.promise(() => Promise.resolve(5))

// Assembling the program using a pipeline of effects
const program = pipe(
  Effect.all([fetchTransactionAmount, fetchDiscountRate]),
  Effect.flatMap(([transactionAmount, discountRate]) =>
    applyDiscount(transactionAmount, discountRate)
  ),
  Effect.map(addServiceCharge),
  Effect.map((finalAmount) => `Final amount to charge: ${finalAmount}`)
)

// Execute the program and log the result
Effect.runPromise(program).then(console.log)
// Output: "Final amount to charge: 96"

The pipe method

Effect provides a pipe method that works similarly to the pipe method found in rxjs. This method allows you to chain multiple operations together, making your code more concise and readable.

Here’s how the pipe method works:

const result = effect.pipe(func1, func2, ..., funcN)

This is equivalent to using the pipe function like this:

const result = pipe(effect, func1, func2, ..., funcN)

The pipe method is available on all effects and many other data types, eliminating the need to import the pipe function from the Function module and saving you some keystrokes.

Let’s rewrite the previous example using the pipe method:

import { Effect } from "effect"

const addServiceCharge = (amount: number) => amount + 1

const applyDiscount = (
  total: number,
  discountRate: number
): Effect.Effect<number, Error> =>
  discountRate === 0
    ? Effect.fail(new Error("Discount rate cannot be zero"))
    : Effect.succeed(total - (total * discountRate) / 100)

const fetchTransactionAmount = Effect.promise(() => Promise.resolve(100))

const fetchDiscountRate = Effect.promise(() => Promise.resolve(5))

const program = Effect.all([
  fetchTransactionAmount,
  fetchDiscountRate
]).pipe(
  Effect.flatMap(([transactionAmount, discountRate]) =>
    applyDiscount(transactionAmount, discountRate)
  ),
  Effect.map(addServiceCharge),
  Effect.map((finalAmount) => `Final amount to charge: ${finalAmount}`)
)

Cheatsheet

Let’s summarize the transformation functions we have seen so far:

API	Input	Output
map	Effect<A, E, R>, A => B	Effect<B, E, R>
flatMap	Effect<A, E, R>, A => Effect<B, E, R>	Effect<B, E, R>
andThen	Effect<A, E, R>, *	Effect<B, E, R>
tap	Effect<A, E, R>, A => Effect<B, E, R>	Effect<A, E, R>
all	[Effect<A, E, R>, Effect<B, E, R>, ...]	Effect<[A, B, ...], E, R>