Integrating Effectful and Persistent

Bellroy’s Tech Team carefully curates the set of libraries and techniques approved for general use in our codebases. We have a process for experimenting with and approving new libraries and techniques, and while we’ll write about that experimentation process another time, today we’re going to focus on the interaction between two of those experiments: the effectful effect-system library and the persistent database access library.

A short introduction to effectful

Effect system libraries allow the programmer to define “effects”: collections of related operations like “read-only access to a data store”. Abstract console I/O is the canonical example used in nearly every tutorial, and is represented in the Haskell effectful library like this:

data Console :: Effect where
  GetLine :: Console m Text
  PutLine :: Text -> Console m ()

-- `makeEffect` is from package `effectful-th`.
-- It automatically generates functions corresponding to each
-- constructor of an effect type. Here, it generates:
--
-- * getLine :: Console :> es => Eff es Text
-- * putLine :: Console :> es => Text -> Eff es ()
--
-- The `Console :> es` constraint indicates that these functions use
-- the `Console` effect. `Eff es` is a Monad, and the `es` type
-- variable stands for "effects" or "effect set".
--
-- Read the `:>` operator as "is in": "Console is in the effect set"
$(makeEffect ''Console)

Programmers using the effect write their code against these generated functions, and get type-level tracking of which effects their code uses:

greet :: Console :> es => Eff es ()
greet = do
  putLine "What is your name?"
  name <- getLine
  putLine $ "Hello, " <> name <> "!"

This greet function has two important features: it tracks that it uses the Console effect, while having no opinion about how the Console effect is implemented. We can actually provide multiple implementations (called “effect handlers”) for a single effect, and choose whichever one makes the most sense for our needs:

-- Discharge a `Console` effect by performing I/O actions against
-- standard input/ouput.
--
-- `IOE :> es` is `effectful`'s "arbitrary I/O" effect.
runConsoleIO :: (IOE :> es) => Eff (Console : es) a -> Eff es a

-- Purely discharge a `Console` effect by providing a list of lines to
-- use for input. Return a list of output lines along with the result.
-- This can be good for testing.
runConsolePure :: [Text] -> Eff (Console : es) a -> Eff es ([Text], a)

These features feed into the main benefits we hope to gain from an effect system:

• We’d like to write code that’s only aware of the abstract effects it needs to do its job; and
• We’d like to write better tests by providing alternative effect handlers for our testing code to use. We might want to turn off or capture logging, or replace a remote data store with an in-memory one.

Our experiment

Effect system libraries are an active research area in the Haskell world, and that means there are several promising libraries offering different tradeoffs between (at least) performance, developer ergonomics, scary type signatures, and learnability. We ran our experiment in a brand new project, and when we started that project in early 2023, effectful seemed like the most promising balance between those factors.

That project was also the first time we had to directly interface our Haskell code with a PostgreSQL database, and for that we decided to trial the persistent library. Combining this with effectful proved challenging because persistent’s typeclasses like PersistStoreRead are fundamentally coupled to MonadIO, and much of the value of using an effect system comes from tracking what effects are performed instead of settling for “pure functions” and “can perform arbitrary I/O”.

As part of the effectful experiment, we wanted to interpret domain-specific effects into a “Persistent effect”, and wanted an effect with the following properties:

1. Not needing to write wrappers for each individual persistent operation. At the experimentation stage, this is too much work for too little benefit.

2. Using a persistent action should not add an obvious IOE effect, which is effectful’s marker that the function can perform arbitrary IO. We’d like an effect to indicate that persistent-using functions are “doing database operations” without having to write IOE :> es everywhere.

3. It should be possible to share a persistent backend (e.g., SqlBackend) with specific subcomputations, so that we can provide backends either from a resource pool or share a single backend with the entire program.

4. Whether we use a resource pool should be invisible to code that uses our effect. We don’t want arbitrary functions to be able to manipulate the pool itself; our effect handler should be responsible for loaning out backend values from the resource pool.

To avoid getting bogged down writing an effectful-flavoured binding to all of persistent, we instead defined a slightly dodgy Persist effect that’s parameterised by the backend it looks for, and whose only job is to hold persistent actions:

data Persist backend :: Effect where
  LiftPersist :: ReaderT backend IO a -> Persist backend m a

type instance DispatchOf (Persist backend) = 'Dynamic

Because of the backend type parameter, the makeEffect Template Haskell macro won’t work, and so we manually define the operation that sends the request to a handler:

-- | Lift an action from the `persistent`
-- library into the effect system.
liftPersist ::
  forall backend a es.
  (Persist backend :> es) =>
  ReaderT backend IO a ->
  Eff es a
liftPersist action = send $ LiftPersist action

The final part of setting up an effect is providing ways to discharge it. We’ll need two — one to provide a backend directly, and one that sources it from a resource pool. runPersistDirect is quite straightforward, but runPersistFromPool requires us to unlift Eff es down to IO so that we can use Data.Pool.withResource:

-- | Discharge a @'Persist' backend@ effect by providing a backend to use.
runPersistDirect ::
  forall backend es a.
  (IOE :> es) =>
  backend ->
  Eff (Persist backend ': es) a ->
  Eff es a
runPersistDirect backend = interpret $ \_ -> \case
  LiftPersist m -> liftIO $ runReaderT m backend

-- | Discharge a @'Persist' backend@ effect by loaning a backend from
-- a resource pool. The backend is returned to the pool after use.
runPersistFromPool ::
  forall backend es a.
  (IOE :> es) =>
  Pool backend ->
  Eff (Persist backend ': es) a ->
  Eff es a
runPersistFromPool pool eff =
  withEffToIO (ConcUnlift Ephemeral Unlimited) $ \runInIO ->
    liftIO . withResource pool $ \backend ->
      runInIO $ runPersistDirect backend eff

Using the Persist backend effect

Actually using this effect is about as simple as any other effectful effect:

-- | Insert a single record, as a minimal example
persistSomeData ::
  Persist SqlBackend :> es =>
  SomeData ->
  Eff es SomeDataId
persistSomeData someData =
  liftPersist @SqlBackend $ Persist.insert someData

Conclusions

This approach has worked fairly well during our experiment, and has some benefits that made us think it was worth writing up:

• It captures the programmer’s intent quite well — a Persist backend :> es constraint clearly shows “database access happens here”;
• We didn’t need a perfectly idiomatic effect definition to begin writing useful code against the effect;
• After we worked through all the details and settled on the LiftPersist idea, it was reasonably quick to set up;
• This technique can work for a large class of libraries (even ones that are quite tightly coupled to IO); and
• Other developers didn’t need to touch the effect definition once it was written.

However, this quick-and-dirty approach has some pretty noticeable deficiencies that it’s worth being aware of:

• The biggest problem is that liftPersist . lift can smuggle an arbitrary IO a action, so the Persist backend effect isn’t making an honest “only database access happens here” promise. Our developers are well-intentioned, so this is fine for a prototype;
• The effect itself is opaque — the only possible interpretations are variations on “just run it”; and
• It’s very easy to let the Persist backend :> es constraint float to the outermost layer of your program and discharge it using runPersistFromPool, not realising this shares a single backend with the whole program and defeats the entire point of the resource pool.

As an experimental technique, we’re pretty happy overall with how this style of effect turned out. Using a simpler effect definition let us run a cheap experiment with persistent, and it’s always possible to refine the effect later. This is the kind of refactoring Haskell excels at — we can remove liftPersist whenever we want and chase down the type errors until everything compiles, defining more precise operations under the Persist backend effect as they’re needed.