Exploring Better Ways - Bellroy

Reflections on Bellroy's Tech Team 2025

Mike Webb — Fri, 13 Mar 2026 00:00:00 UT

Reflections on Bellroy's Tech Team 2025

Mike Webb 2026-03-13

Technology

We shipped more than ever in 2025, and by the end of the year we were struggling with the complexity of it all. In writing this post, I continue to be amazed at the amount the Bellroy Technology Team is able to achieve and support in a year with just 15 staff.

In 2024, we realised our processes had a blind spot. We could see internally-facing projects that were clearly valuable were not getting the attention they deserved, so we revised our project evaluation methodology to give these types of projects fairer weight based on the opportunity cost of not doing them. The result was that, in 2024 and 2025, we split our focus between process automation for other teams, company-wide tooling, infrastructure modernisation and customer-facing features. By the end of the year, a recurring theme in our team retrospectives was that the sheer breadth of what we were maintaining and building had become difficult for any one person to hold in their head, and yet there was an unspoken expectation that everyone was across everything.

Prior to each eight week cycle, we evaluate the list of projects that people all over the business had put forward. Every candidate project goes through a probabilistic NPV model we built internally, which estimates the value each project is likely to deliver over the next five years. The projects described below are the ones that scored highest - and even then, we weren’t always wholly successful at delivering their full value.

What we worked on

Our most complex project - integrating Stripe alongside Braintree as our payment provider - taught us the most about long-tail risk. The project touched every part of our checkout experience and our developers needed to deeply understand our legacy payment flow to be able to implement it as an A/B test. We had a long tail of edge cases that only emerged under real traffic; for example, asynchronous payment types such as AfterPay behave significantly differently in the wild from synchronous payment types, such as a credit card payment. This threw up some edge cases that we had not had to deal with our previous payment provider. Because these issues arose sometimes long after the project was completed, it either fell on the original project team to re-load the context to address the issues or it fell to other personnel to resolve the issue with far less experience with that domain.

Sometimes we understood the business objective and the technical requirements, but hadn’t thought about the staff-facing UX changes. We rolled out more aggressive CloudFront caching across our website, meaningfully improving page load times. While we achieved the desired performance increase, we didn’t sufficiently communicate the change in site behaviour to our content team and they were surprised to discover that their previous experience of “instant publishing” now came with a short lag (while the CloudFront cache was invalidated).

The Technology team have been using AI tools since the early days of Claude Code, and many other teams at Bellroy were buying per-seat licenses for Claude and ChatGPT on a as-needed basis. By mid-year these individual user accounts across ChatGPT and Claude were costing us upwards of USD$3,000 a month. In response to this, we deployed LobeChat as a self-hosted LLM client for the whole company, granting API access to the same models at a fraction of the cost. We ran roadshow sessions and regular open “AI assistance” sessions for the whole company as part of the rollout. Now responsible for keeping the tools up to date for a bigger audience, we were suddenly subjected to the breakneck pace of change in the tool itself and across 3 different model providers.

And those were just the big ticket items - on the customer-facing side, we expanded into new retail partnerships and marketplaces, and enabled bellroy.com orders to be fulfilled via Amazon in more countries. We continued our tradition of working closely with other teams to automate and improve their processes - inbound shipment management for Production, corporate sales inquiries for Wholesale, milestone staff rewards for People & Culture. We undertook a wholesale modernisation of our server infrastructure, migrating our configuration management from Chef to Puppet. We started evaluating Garnix as a replacement for our Hydra CI system, and explored containerisation for our Lambda-based services using Nix’s dockerTools, proving out a model we’re looking to adopt more broadly in 2026.

At the risk of repeating myself - it is staggering to me that a team of this size can achieve so much in a year. I feel very blessed to work with such a great crew.

The cost of doing it all at once

Reading the sections above, it would be easy to imagine these as neat, sequential projects handed off between well-defined teams. The reality was that all of these projects were achieved with teams working in parallel. We had developers rotating between projects every eight weeks based on availability, timezone and skillset. Within a given cycle, developers were focused on a single project - but at each rotation boundary they needed to ramp up on an entirely different system, understand the business context behind the work and build enough mental model to be productive. Over the course of the year, that ramp-up cost compounded.

Coordinating all of this fell disproportionately on a small number of people. The LobeChat rollout, for example, was happening alongside everything else - the roadshows, the support queries, the software updates - and was largely driven by one person (ahem) who was also trying to stay across every other active workstream. We needed to make sure that changes to shared infrastructure didn’t block an in-flight feature (and of course they did, multiple times), that deployments didn’t collide (ditto) and that the groups picking up a piece of work at the start of each cycle had enough context to be effective without lengthy handovers (I’ll give us a solid B+ on this).

By the second half of the year the cumulative weight of all these rotations was something the whole team felt. We got better at writing things down and leaving breadcrumbs for the next group, but the cognitive overhead of supporting this many concurrent workstreams across a team of our size was significant.

Looking ahead

Towards the end of the year one of the team leads suggested we read the Team Topologies book. After reading the book and having many long discussions, we made the decision to start 2026 with a new team structure, categorising our codebases and configuration sets into a “Customer Journey” business stream, an “Operations” business stream or an “Infrastructure” platform team. When we presented this to the team as a potential change, there was a palpable sense of relief and enthusiasm.

This comes at a good time - 2026 is shaping up to be an exciting year for Bellroy, with a fantastic team and an expansion of where we appear in the world and what we’re bringing to our customers. We’ve already seen some benefits with the change to the new structure: better relationships with stakeholders and our developers reporting a much less tumultuous project transition experience. But as with any change, there are trade-offs: our AI tooling and strategy still has no clear owner other than our CTO and there is a stream of work around developer tooling that we don’t currently have the resources to get done. We need to consider further hiring to be able to properly service these work streams.

Who knows, this time next year we might have 16 people!

Some Haskell idioms we like

Jack Kelly — Wed, 14 Jan 2026 00:00:00 UT

Some Haskell idioms we like

Jack Kelly 2026-01-14

Technology

Scaling up our Haskell usage at Bellroy has meant cultivating a “house style” or “engineering dialect”. While a house style obviously means making decisions about the traditional Haskell decision points — choosing a preferred set of libraries and language extensions — we also believe that a good engineering dialect means forming opinions about the way we express code. Haskell is a fairly simple language under all the syntax; the same idea can have many different expressions, some much clearer than others. In this post, we’ll share a few small idioms that we’ve adopted. They might not all be novel, but we think they’re valuable enough to document.

Explicit construction in concrete `Monad`s

Idiom: When working in a concrete monad like Maybe, Either e or [], use that type’s data constructors explicitly instead of calling pure.

Example:

-- This is a simplified example from one of our internal DSLs.

-- | AST for terms.
data Term
  = TmBool Bool
  | TmNumber Rational
  | TmAnd Term Term
  -- Plus some other constructors

-- | Enum of types.
data Type = TyBool | TyNumber -- Plus some others

data TypedTerm = TypedTerm Type Term

-- | Type-check a term.
--
-- If an expected type is provided, perform checking; if not
-- attempt to perform inference.
infer :: Maybe Type -> Term -> Either TypeError TypedTerm
infer expectedTy term = case term of
  TmBool _ -> Right $ TypedTerm TyBool term
  TmNumber _ -> Right $ TypedTerm TyNumber term
  TmAnd l r -> do
    TypedTerm TyBool _ <- infer (Just TyBool) l
    TypedTerm TyBool _ <- infer (Just TyBool) r
    Right $ TypedTerm TyBool term
--  ^^^^^
--  We use an explicit `Right` here to remind the reader that this
--  `do`-block only assembles data, and does not perform side effects.

Discussion: The primary use of Haskell’s Monad typeclass is to allow sequential description of operations “in a context” using do blocks. Although Haskell teaching material often demonstrates Monad instances for many data types before teaching IO, it is typically the “stateful” monads like State or IO that have complex enough control flow to warrant do blocks. This creates an implicit association in some readers’ minds between do blocks and sequential, side-effectful code, and most of our do blocks are of this type. Writing out the “success constructor” explicitly (e.g. Just, Right) indicates when this is not the case, and shows that no side effects are happening here. It is a small difference, but a clarifying habit, especially when there’s an outer do-block in an IO-like monad and a Maybe or Either value being constructed in a let or in a function argument.

Design modules for qualified import

Idiom: Choose names that read well when their containing module is imported qualified. Avoid contorting identifier names to disambiguate them from similar names in other modules.

Examples:

module Servant.Client.Handle where

-- | We say @Handle@ and not something like @ServantHandle@, so
-- that code which uses one handle can be kept more compact.
-- Code which needs to disambiguate between handles can consistently
-- refer to them as (e.g.) @Servant.Client.Handle@ or @DynamoDB.Handle@.
data Handle m = Handle {..}

addCaching :: CacheConfig -> Handle m -> Handle m
addCaching = undefined

-- | Sometimes moving an enum into its own module can help.
-- Here, we can @import qualified Bellroy.Shipping.Speed as Speed@
-- and then write (e.g.) @Speed.Regular@.
module Bellroy.Shipping.Speed (Speed(..)) where

data Speed = Regular | Expedited | Overnight

Discussion: As the codebase grows, it becomes almost inevitable that any given module will need to be imported qualified at some point. Attempts to work around this by adding context to identifier names (e.g. data ServantHandle m = ..., cacheServantHandle) create awkward identifiers like Servant.cacheServantHandle when a qualified import inevitably becomes necessary. Often the qualified import ends up being the same or similar length as a “contorted” name while reading more cleanly.

This idiom works best when applied to modules containing a primary type (such as a data structure or “handle”) and a collection of functions that operate mostly on that type.

Unpack large patterns using a `where` clause

Idiom: Keep function parameter declarations compact by unpacking larger patterns inside a where clause. This can also be done using let.

Examples:

-- When large patterns necessitate breaking the LHS of the `=`
-- over multiple lines, it's difficult to see where the body begins.
someFunction
  (SomeRecord {field1, field2, field3})
  anArg
  someOtherArg@(MorePatternNoise {..}) =
    body

-- Instead, move the pattern matches down into a `where`:
someFunction' someRecord anArg someOtherArg = body
  where
    SomeRecord {field1, field2, field3} = someRecord
    MorePatternNoise {..} = someOtherArg

Another (simplified) example from real code:

shipTo :: SalesOrder -> Either Error Logistics.Import.ShipTo
shipTo SalesOrder{..} = do
  -- Omitted: a bunch of code that parses and massages
  -- data into the form which downstream wants.

  Right
    Logistics.Import.ShipTo
      { name = fromMaybe fullName' orgName,
        attention = fullName' <$ orgName,
        address1,
        address2,
        address3,
        city = city',
        postalCode = postalCode',
        country = country',
        phone,
        email,
        consigneeTaxId = Nothing
      }
  where
    Address {city, country, postalCode, stateCode, stateName} =
      applyShippingAddressOverrides customerShippingAddress
    Attention {fullName, phone, email} = attention
 -- ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 -- Instead of bulking out the pattern-match on `SalesOrder{..}`,
 -- we match `Attention` here. This makes it much easier to skim
 -- the parameter declaration of a function and read through its body.

Use `inverseMap` liberally

Idiom: When writing paired injection/lookup functions (print/parse is our most common example), use relude’s inverseMap function (or an equivalent) to derive the “lookup” side from the “injection side”. The inverseMap function computes the partial inverse of the input function by enumerating its domain:

inverseMap :: (Bounded a, Enum a, Ord k) => (a -> k) -> k -> Maybe a

Example:

data ErrorCode = InsufficientCredits | InvalidParams | General
  deriving stock (Eq, Show, Generic, Enum, Bounded)

renderErrorCode :: ErrorCode -> Text
renderErrorCode = \case
  InsufficientCredits -> "insufficient_credits"
  InvalidParams -> "invalid_params"
  General -> "general"

parseErrorCode :: Text -> Maybe ErrorCode
parseErrorCode = inverseMap renderErrorCode
              -- ^^^^^^^^^^
              -- Avoids writing `parseErrorCode` by hand.

Discussion:

A common cause of bugs in print/parse function pairs comes from adding a new constructor to the data type and failing to update the parser. Writing the parser using inverseMap makes this impossible, because the parser is derived from the printer and GHC enforces the completeness of the printer’s case-match.

According to Hoogle, inverseMap is only provided by the (excellent) relude custom prelude, but is easy enough to define independently. The only subtlety is ensuring that the intermediate map is shared between calls:

-- Declare the `k` argument in a lambda so GHC thinks the function
-- is "fully applied" with one argument.
--
-- This could also use `enumerate :: (Bounded a, Enum a) => [a]`,
-- which arrives in base-4.22 (GHC 9.14).
inverseMap ::
  forall a k.
  (Bounded a, Enum a, Ord k) => (a -> k) -> k -> Maybe a
inverseMap f =
  let m = Map.fromList [(f a, a) | a <- [minBound .. maxBound] :: [a]]
   in \k -> Map.lookup k m

Takeaways

Haskell blogs often use the language’s expressive type system and other headline features to demonstrate some very spectacular things, but there’s also a lot of expressivity in its more mundane features and through careful choices of names. Idioms like the ones we’ve shared here tend not to get the blog post treatment, but instead percolate through less formal channels like code reviews, chat and forum posts. We felt it was worth surfacing some of the ones we’ve found, because they add clarity without adding much code, and don’t impose limits on the language features we use (although we have an evolving conversation about that too). We hope you also find them useful.

Free applicatives, the handle pattern, and remote systems

Jack Kelly — Mon, 13 Oct 2025 12:00:00 UT

Free applicatives, the handle pattern, and remote systems

Jack Kelly 2025-10-13

Technology

We recently refactored some gnarly code that manipulated customer and order records in our enterprise resource planning (ERP) system. That system had a few idiosyncrasies which complicated this task:

Creating new records required referencing other entities by internal ID, so we had to do a number of lookups before issuing “create record” requests;
For some entity types, we found it easiest to issue “search” API calls and extract the required IDs from the returned search results. This necessitated an extra parsing step between “we have a successful response” and “we have the ID we’re looking for”; and
Requests are often slow, but the marginal cost of additional requests in a batch was quite low. This meant that we could expect some good results from batching related requests together.

The benefits of batching led us to seek a solution that permitted static analysis. Applicative functors have a completely static control flow, and cannot express computations where one step depends on the result of a previous step. A well-chosen applicative would let us analyse the requests we need to send without executing any of them, batch queries together without worrying about data dependencies, and then route relevant results to each individual query to parse (if necessary). Our library users could ignore batching details but still gain the efficiency benefits of a batch query API.

In this post, we’ll look at how we’ve been using handles, what “free structures” are, how free applicatives elegantly solved some of our problems interfacing with a remote system, and how they interacted especially well with the “handle pattern”.

Handles, as Bellroy uses them

The “handle pattern” is a Haskell idiom that is similar to dependency injection in mainstream languages. Instead of directly writing in the side effects we want our code to perform, we accept a record of functions that we call a “handle”. (In an object-oriented language, we’d probably accept an object that implements an abstract interface instead of a record.) These handles can group related functions into a single record but often only contain one:

newtype Handle e m = Handle {
    performRequest :: ERP.Request -> m (Either e Aeson.Value)
  }

-- Plus some other handle-making functions e.g. for testing.
newHandle ::
  MonadIO m => ERP.Credentials -> m (Handle ERP.Error m)

Functions that consume handles generally look like this:

someFunction ::
  -- When all side effects come from handles,
  -- we rarely need anything stronger than `Monad`.
  Monad m =>
  -- First: Any handles we need
  FooHandle m -> BarHandle m ->
  -- Second: Other "normal" arguments
  Argument1 -> .. -> ArgumentN ->
  m Result

This idiom is a simpler, library-free alternative to effect system libraries like effectful, bluefin, heftia and polysemy. We previously wrote about an experiment with effectful, but we have still not committed to an effect system. Instead, we are refactoring towards handles as a way to encapsulate our side effects, and because it should be easy to convert handle-using code to an effect system if and when we choose one.

Because we have code written against other idioms (e.g. MTL-style classes), and because we often find it convenient to introduce an ExceptT e or MaybeT monad transformer in the body of our functions, we sometimes need to change the monad of a handle that we’ve been given. We do this by providing a hoistHandle function:

hoistHandle :: (forall x . f x -> g x) -> Handle f -> Handle g
hoistHandle f Handle {performRequest} =
  Handle {performRequest = f . performRequest}

That first argument, forall x . f x -> g x, is worth commenting on. A forall in a type signature explicitly introduces a type variable that is provided by the function’s caller. For a simpler example of how forall works here, let’s look at the map function on lists, but with explicit foralls:

map :: forall a b . (a -> b) -> [a] -> [b]

The caller of map gets to choose the types of a and b, and GHC is often smart enough to figure this out automatically:

-- GHC concludes that it needs to call
-- `map` with `Int` for `a` and `String` for `b`.
ghci> map show [1, 2, 3]
["1","2","3"]

In our hoistHandle function, we let the caller choose f and g, but they must provide us a function where we are allowed to choose x. The types force this function to convert f x into g x in a way that’s blind to what x actually is — guaranteeing that the conversion only changes structure, not wrapped values. It also ensures that we can write hoistHandle for a handle containing multiple functions, because we can choose a different x for each one.

Building our applicative

We want to build a structure that is essentially a syntax tree of the operations we want to perform. This means it needs to hold the requests we want to send, and because we want it to be an applicative, we’ll add constructors to represent pure and (<*>):

data Query a where
  QueryAndParse ::
    FromJSON x =>
    ERP.Request -> (x -> Either Text a) -> Query a

  -- Extra constructors to hold applicative structure
  Pure :: a -> Query a
  Ap :: Query (a -> b) -> Query a -> Query b

deriving stock instance Functor Query

instance Applicative Query where
  pure = Pure

  Pure f <*> Pure x = Pure $ f x
  QueryAndParse req f <*> Pure a =
    QueryAndParse req $ fmap ($ a) . f
  -- Plus another seven cases, being careful that
  -- each case obeys the applicative laws.

QueryAndParse is the only data constructor directly relevant to our problem. It captures the request we want to make against the ERP, a FromJSON x constraint so we can parse the raw response into some intermediate type representing an API response, and a function x -> Either Text a to extract just the data we want from that API response.

This design could work, but it’s a fair amount of boilerplate, and the next time we want an applicative like this we’d need to repeat most of it. In the next section, we’ll use a free applicative to separate the general “applicative” code from the specific “query and parse” code.

What is a “free structure”?

To understand how free applicatives help us with this problem, we need to have some idea what “freeness” means in this context. The Haskell community usually talks about taking “the free $class over $type” as a way to make $type an instance of $class, by adding just enough structure to construct a lawful instance of $class. Packages like free provide wrapping types that hold values of $type and provide instances of $class.

Let’s pare our Query type back to something much smaller: a type representing a single request against our ERP:

data OneQuery a where
  QueryAndParse ::
    FromJSON x =>
    ERP.Request -> (x -> Either Text a) -> OneQuery a

We will now re-write Query as the free Applicative over OneQuery. To make OneQuery into an Applicative, we’ll use the Ap wrapper from Control.Applicative.Free. Here is its interface:

-- `Ap f` is the free applicative over `f`. We never use its
-- constructors directly; instead, we use `liftAp` and the
-- `Applicative` interface (`pure`, `(<*>)`, `liftA2`, etc.)
data Ap f a

-- For *any* `f`, `Ap f` is an applicative.
instance Applicative (Ap f)

-- We can turn any `f a` into an `Ap f a`.
liftAp :: f a -> Ap f a

-- If we can turn our `f` into some applicative `g`, then we can turn
-- `Ap f a` into `g a` in a way that respects the Applicative laws:
--
-- runAp _ (pure x) = pure x
-- runAp f (x <*> y) = (runAp f x) <*> (runAp f y)
--
-- Similar to the `forall x. f x -> g x` in `hoistHandle` above,
-- this lets us turn each `f x` stored in the `Ap f a` into a
-- corresponding `g x`, while remaining ignorant of the specific
-- type `x`.
runAp :: Applicative g => (forall x. f x -> g x) -> Ap f a -> g a

We’ll skip the implementations because we won’t ever manually recurse through an Ap f a value; from a modularity perspective, we are only interested in the abstract interface. We declare Query as the free applicative over OneQuery, make it a newtype to establish an abstraction boundary between the query library and its callers, and use deriving newtype to avoid writing any applicative structure ourselves:

newtype Query a = Query (Free.Ap OneQuery a)
  deriving stock Functor
  deriving newtype Applicative

-- Helper functions to avoid building `Query` values by hand.

query ::
  FromJSON a => ERP.Request -> Query a
query req =
  Query . Free.liftAp $ QueryAndParse req Right

queryAndParse ::
  FromJSON a => ERP.Request -> (a -> Either Text b) -> Query b
queryAndParse req f =
  Query . Free.liftAp $ QueryAndParse req f

Building a `Query`

From this infrastructure, we can write functions representing individual queries. These are direct applications of the query and queryAndParse helpers:

queryLocationId ::
  ERP.Location.Name -> Query ERP.Location.Id
queryLocationId locationName =
  query $ ERP.lookupLocation locationName

queryOrderId ::
  ERP.Order.Name -> Query ERP.Order.Id
queryOrderId orderName =
  queryAndParse
    (ERP.searchOrders orderName) $ \case ->
      [order] -> Right order
      (_:_) -> Left "Multiple Orders in response"
      [] -> Left "No Orders in response"

From these functions we can build up complex queries using applicative operations:

queryOrderAndLocation ::
  ERP.Order.Name -> ERP.Location.Name ->
  Query (ERP.Order.Id, ERP.Location.Id)
queryOrderAndLocation orderName locationName =
  liftA2 (,) (queryOrderId orderName) (queryLocationId locationName)

Running a `Query`

We can run a Query by using runAp. Because we’re in an applicative context and we’re making requests that don’t alter the remote system, we can run every request and use a Validation to collect all failures:

data RunQueryError e = RequestError e | JsonError Text | ParseResultError Text
type RunQueryErrors e = NonEmpty (RunQueryError e)

runQuery :: forall e m a.
  Monad m =>
  ERP.Handle e m ->
  Query a ->
  m (Validation (RunQueryErrors e) a)
runQuery ERP.Handle{performRequest} (Query q) =
  getCompose $ Free.runAp (Compose . go) q
  where
    go :: OneQuery x -> m (Validation (RunQueryErrors e) x)
    go (QueryAndParse req parse) = performRequest req <&> \case
      Left reqErr -> Failure . NonEmpty.singleton $ RequestError reqErr
      Right value -> case Aeson.parseEither Aeson.parseJSON value of
        Left jsonErr -> Failure . NonEmpty.singleton . JsonError $ Text.pack jsonErr
        Right x -> case parse x of
          Left parseErr -> Failure . NonEmpty.singleton $ ParseResultError parseErr
          Right a -> Success a

The implementation can be mostly derived by following the types, but we’ll highlight some specifics:

Validation e a is a type that’s structurally isomorphic to Either e a, but provides an Applicative instance that accumulates errors:

-- From the validation-selective package.
instance Semigroup e => Applicative (Validation e) where
  pure = Success

  -- This asymmetric way of writing <*> maximises laziness.
  Failure e1 <*> b = Failure $ case b of
    Failure e2 -> e1 <> e2
    Success _  -> e1
  Success _ <*> Failure e = Failure e
  Success f <*> Success a = Success (f a)

Since the composition of any two applicatives is itself an applicative, Data.Functor.Compose lets us combine the m and Validation e applicatives into Compose m (Validation e), which executes actions in m and accumulates errors — exactly what we want.
Since we use the Compose constructor to wrap the result of go, Free.runAp will return a Compose m (Validation e) a which we must unwrap with getCompose.
The go function processes a single request held in a OneQuery x, and Free.runAp uses it to build up the applicative combination of each result.
We accept a handle telling us how to contact the ERP. This is the key location where the handle pattern and the free applicative interact, giving the library user a lot of power: the handle parameter frees us from being coupled to any particular monad and makes it easier to write tests for this code. We’ll see another way to construct a ERP.Handle very soon.
The caller of the Query interface has no idea that we’re building and consuming free structures under the hood. It’s an implementation detail that doesn’t distort the abstraction boundary at all.

Extracting requests

Now that we can execute queries, let’s explore the main benefit of free applicatives: the ability to analyse the applicative program without running it. We can extract a monoidal summary of any free applicative’s structure via runAp_:

runAp_ :: Monoid m => (forall x . f x -> m) -> Ap f a -> m

For an intuition why this is true, consider that the constant functor Const r has an Applicative instance whenever r is a monoid, because pure stores a mempty value and (<*>) combines the held values with (<>). For a fun exercise, implement runAp_ in terms of runAp and Const.

We can use runAp_ to extract a list of every request a Query a will send:

allRequests :: Query a -> [Request]
allRequests (Query q) = ordNub $ Free.runAp_ go q
  where
    go :: OneQuery x -> [Request]
    go (QueryAndParse req _) = [req]

Once we have the list of requests, we can look for ways to optimise them. De-duplicating the requests with ordNub is an easy optimisation, but if the remote API supports it, we could do more advanced optimisations like using a batch request API.

As a simple demonstration, we can perform all the lookup requests in advance and construct a Map Request Aeson.Value:

type SavedRequests = Map Request Aeson.Value

saveRequests ::
  forall e m.
  Monad m =>
  Handle e m -> [Request] -> m (Either e SavedRequests)
saveRequests Handle{performRequest} requests =
  runExceptT $ Map.fromList <$> traverse go requests
  where
    go :: Request -> ExceptT e m (Request, Aeson.Value)
    go req = (req,) <$> ExceptT $ performRequest req

Using a collection of saved results, we can construct a handle that returns the saved responses instead of performing real requests:

newtype UnsavedRequestError = UnsavedRequestError Request

newHandleFromSavedRequests ::
  (Applicative m) => SavedRequests -> Handle UnsavedRequestError m
newHandleFromSavedRequests requests =
  Handle
    { performRequest = \req ->
        pure . maybe (Left (UnsavedRequestError req)) Right $
          Map.lookup req requests
    }

This gives us a great story for testing. Since our runQuery works with any handle, we can capture some real requests to a file, redact any sensitive information, and create a pure handle built from saved requests. We can then use this handle to write test cases that run real code without performing side-effects.

If this example moved too quickly, or you want to see another application of free structures, Justin Le has a spectacular post on matching regular expressions using the free Alternative.

Payoffs and limitations

What have we achieved? We decided that we wanted an applicative to describe queries against our remote system. Instead of inventing a complicated data structure to represent the syntax tree of pure and (<*>) calls, we defined a type just to hold one request and took the free applicative over it. We also used the handle pattern to ask for only the side-effects that we needed. Both patterns are reasonably easy to implement, and in exchange we got some pretty neat benefits that would’ve been harder to realise with either technique alone:

We can analyse a Query without running it, and use the Query to inform the handle we do eventually use;
As a special case of (1), library users can code against a convenient interface and request individual records, but we can inspect the set of queries before we begin execution and issue optimised, parallelised, de-duplicated and batched requests in their place;
We don’t have to abort at the first failed request — we can collect and report every problem with a Query; and
We can record and replay requests, giving us a great testing story in the style of Ruby’s vcr library.

It’s not all roses, though. We lose a significant amount of expressive power by giving up the monadic interface. For APIs where we need to interleave pure queries and side-effecting requests, losing the Monad instance might be a bridge too far. Chris Penner suggests that Selective functors could be closer to the sweet spot, but then you lose the nice ergonomics of -XApplicativeDo. Chris Done identifies an “Applicative-wired Monad” pattern which uses a monad only to plumb together applicative values.

So where does this leave us? The handle pattern has been working well for us and we plan to continue refactoring code to use handles for the foreseeable future. In narrow contexts where we want to take advantage of static analysis, a well-chosen free applicative has given us a surprising amount of modularity, testability and opportunities for automatic optimisation. In the function that “runs” the free applicative, these two idioms interacted in a very satisfying way: the handle parameter gave us a lot of flexibility without asking library users to write a lot of boilerplate.

Designing for the different stages of a system's life

Mike Webb — Tue, 22 Jul 2025 12:00:00 UT

Designing for the different stages of a system's life

Mike Webb 2025-07-22

Technology

Every software engineer has their war stories — those moments when a seemingly brilliant architectural decision comes back to haunt them years later. Today, I want to share one of ours, because it reveals something important about how we think about system design: we often make good architectural choices for the things we expect to change but in ways that make it harder to adapt to the unexpected.

The Clean Architecture revelation

Like many engineering teams, we have a technical book club at Bellroy. Over the years, we’ve worked through the classics that have shaped our industry — The Pragmatic Programmer, Clean Code, and the book that would prove both transformative and, eventually, problematic: Clean Architecture.

When we first read Uncle Bob’s treatise on architectural patterns, it felt revelatory. Here was a framework that seemed to identify and solve many of the pain points our team was experiencing. As a Ruby on Rails shop experimenting with different infrastructure providers and integration partners, we’d been bitten repeatedly by tight coupling. Switching from one payment processor to another, or migrating from file-based storage to a database, always seemed to require changes scattered throughout our codebase.

Clean Architecture promised a better way. The core principle is simple: dependencies should point inward, toward your business logic, never outward toward implementation details. You achieve this through interfaces and gateways — abstract boundaries that let your core domain remain blissfully unaware of whether it’s talking to PostgreSQL or MongoDB, Stripe or PayPal.

We were sold. We printed out the famous Clean Architecture diagram and taped it to our office walls like an architectural commandment. We refactored our codebase to embrace the pattern, creating gateway interfaces for external services and ensuring our business logic lived in the pristine centre of our dependency circles.

The benefits were immediate and tangible. Switching from file-based storage to a relational database became trivial — we could just swap out the implementation behind the gateway interface. Our test suite became more focused and faster, since we could easily mock external dependencies. We felt like we’d discovered the secret to maintainable software.

The ultimate test

It was around this time that we decided we needed more than an accounting system, we needed an Enterprise Resource Planning (ERP) system. ERP migrations are legendary for the chaos they unleash on businesses. Our accounting system, which had served us well in our earlier days, had politely informed us that we’d grown too large for their infrastructure. They suggested we might be happier elsewhere — a diplomatic way of saying we were monopolizing their resources. We were suddenly on the clock.

A migration to an ERP would be the real test of Clean Architecture. We’d heard horror stories of companies losing weeks of productivity, data inconsistencies causing financial headaches, and integrations that limped along for months. We were determined to avoid that fate.

This is where Clean Architecture truly shone. Its gateway pattern gave us an elegant solution: we modified our accounting gateway to write to both our incumbent system and the sandbox environment of our new ERP. For three months, we ran this dual-write setup, comparing outputs at month-end and methodically tracking down every discrepancy.

The switchover itself was almost anticlimactic. At midnight on June 30, 2019 — the end of the Australian financial year — we deployed a single configuration change, updating our connection string to point to the production ERP environment. We ran both systems in parallel for another month as a safety net, then deployed one final change to stop writing to the old system entirely.

It was the smoothest integration I’ve ever been part of. We celebrated accordingly, convinced we’d mastered the art of system evolution.

When architecture becomes archaeology

But software has a way of humbling us, and our celebration proved premature. Not long after our ERP success, we made another significant decision: we committed to functional programming and began transitioning our entire codebase from Ruby to Haskell.

We adopted a pragmatic approach: maintain the Ruby code but only port functionality to Haskell when we could add meaningful value in the process. This meant our Ruby codebase gradually became legacy code, maintained but not actively developed. This transition — which we expected to take a couple of years — has now stretched into its seventh year.

Here’s where the Clean Architecture approach began to work against us. As we hired more Haskell-focused developers and our institutional knowledge of Ruby faded, those carefully crafted abstraction layers became archaeological puzzles. Reverse-engineering what a piece of code actually did — especially complex, multi-step operations with side effects — became a nightmare.

The very indirection that had made our ERP migration so smooth now made understanding our legacy code incredibly difficult. We developed a habit of first collapsing all the abstraction layers in any area we wanted to port, flattening the code so we could see what it actually accomplished. The comprehensive unit tests we’d written for each abstraction layer had to be collapsed too, accounting for all the edge cases across multiple interfaces.

What had once been our architectural strength became a significant liability, made worse as fewer team members retained working knowledge of Ruby.

The missing framework

This experience taught us something important about how we approach system design. We had excellent resources for many aspects of software engineering — Working Effectively with Legacy Code is brilliant for dealing with inherited technical debt, Refactoring gives teams techniques and a common language around good (safe!) refactoring technique, and books like Clean Architecture provide powerful patterns for active development of object-oriented codebases.

But we lacked frameworks that would have helped us think about the entire lifecycle of a system. We didn’t have good guidance on which architectural patterns are appropriate for different stages of a system’s life, or clear warnings about the long-term costs of our design decisions.

There are plenty of blog posts discussing the abstract costs of indirection and over-abstraction. What I’ve shared here is a concrete example of when and how a team actually pays those costs, and how difficult it was for us to anticipate them. Our Clean Architecture approach was absolutely the right choice for an actively developed Ruby system facing integration challenges. But it became the wrong choice for a legacy Ruby system being gradually replaced by Haskell.

Designing for tomorrow’s problems

The lesson isn’t that Clean Architecture is bad — it’s that every architectural decision involves tradeoffs across time. The abstraction layers that make active development and integration seamless can make legacy code maintenance significantly harder. The interfaces that provide flexibility during a system’s growth phase can become obstacles during its sunset phase.

Perhaps what we need is a more nuanced view of system architecture — one that acknowledges that the “right” design depends not just on current requirements, but on the likely evolution of the system over its entire lifespan. Sometimes the most maintainable code is the most obvious code, even if it’s less elegant or flexible. Sometimes in pursuing the honorable goal of DRYing (Don’t Repeat Yourself) up our code, we end up with a more complex system. We now talk about “complexity budgets” - the idea that a team of a particular size can only support a certain amount of complexity across our systems - and use that to guide our design decisions.

Are you:

designing for extensibility? Then a pattern like Clean Architecture is a great fit for object-oriented codebases, and effect systems provide similar benefits for functional programmers.
designing for maintainability? You’ll want to reduce the abstract concepts while maintaining separate, concrete classes/modules with a clear single responsibility.
designing for replacement? Then you want your code to read like an executable description of what the legacy system does right now, end-to-end, so that it can be easily ported.

Abstractions come with ongoing maintenance costs, but they only deliver value when you’re actively switching between the things they abstract. As systems evolve, maintainers must decide when these abstractions stop paying off. In our experience, legacy systems rarely benefit from keeping any of these abstractions.

As we migrate to Haskell, we’re putting this lesson into practice. We’re being more deliberate about when abstraction truly serves us — and when directness is the smarter long-term bet. The code we write today becomes tomorrow’s legacy system, and our future selves and colleagues deserve better than having to excavate meaning from layers of forgotten abstractions.

Integrating Effectful and Persistent

Jack Kelly — Thu, 17 Apr 2025 12:00:00 UT

Integrating Effectful and Persistent

Jack Kelly 2025-04-17

Technology

Bellroy’s Tech Team carefully curates the set of libraries and techniques approved for general use in our codebases. We have a process for experimenting with and approving new libraries and techniques, and while we’ll write about that experimentation process another time, today we’re going to focus on the interaction between two of those experiments: the effectful effect-system library and the persistent database access library.

A short introduction to `effectful`

Effect system libraries allow the programmer to define “effects”: collections of related operations like “read-only access to a data store”. Abstract console I/O is the canonical example used in nearly every tutorial, and is represented in the Haskell effectful library like this:

data Console :: Effect where
  GetLine :: Console m Text
  PutLine :: Text -> Console m ()

-- `makeEffect` is from package `effectful-th`.
-- It automatically generates functions corresponding to each
-- constructor of an effect type. Here, it generates:
--
-- * getLine :: Console :> es => Eff es Text
-- * putLine :: Console :> es => Text -> Eff es ()
--
-- The `Console :> es` constraint indicates that these functions use
-- the `Console` effect. `Eff es` is a Monad, and the `es` type
-- variable stands for "effects" or "effect set".
--
-- Read the `:>` operator as "is in": "Console is in the effect set"
$(makeEffect ''Console)

Programmers using the effect write their code against these generated functions, and get type-level tracking of which effects their code uses:

greet :: Console :> es => Eff es ()
greet = do
  putLine "What is your name?"
  name <- getLine
  putLine $ "Hello, " <> name <> "!"

This greet function has two important features: it tracks that it uses the Console effect, while having no opinion about how the Console effect is implemented. We can actually provide multiple implementations (called “effect handlers”) for a single effect, and choose whichever one makes the most sense for our needs:

-- Discharge a `Console` effect by performing I/O actions against
-- standard input/ouput.
--
-- `IOE :> es` is `effectful`'s "arbitrary I/O" effect.
runConsoleIO :: (IOE :> es) => Eff (Console : es) a -> Eff es a

-- Purely discharge a `Console` effect by providing a list of lines to
-- use for input. Return a list of output lines along with the result.
-- This can be good for testing.
runConsolePure :: [Text] -> Eff (Console : es) a -> Eff es ([Text], a)

These features feed into the main benefits we hope to gain from an effect system:

We’d like to write code that’s only aware of the abstract effects it needs to do its job; and
We’d like to write better tests by providing alternative effect handlers for our testing code to use. We might want to turn off or capture logging, or replace a remote data store with an in-memory one.

Our experiment

Effect system libraries are an active research area in the Haskell world, and that means there are several promising libraries offering different tradeoffs between (at least) performance, developer ergonomics, scary type signatures, and learnability. We ran our experiment in a brand new project, and when we started that project in early 2023, effectful seemed like the most promising balance between those factors.

That project was also the first time we had to directly interface our Haskell code with a PostgreSQL database, and for that we decided to trial the persistent library. Combining this with effectful proved challenging because persistent’s typeclasses like PersistStoreRead are fundamentally coupled to MonadIO, and much of the value of using an effect system comes from tracking what effects are performed instead of settling for “pure functions” and “can perform arbitrary I/O”.

As part of the effectful experiment, we wanted to interpret domain-specific effects into a “Persistent effect”, and wanted an effect with the following properties:

Not needing to write wrappers for each individual persistent operation. At the experimentation stage, this is too much work for too little benefit.
Using a persistent action should not add an obvious IOE effect, which is effectful’s marker that the function can perform arbitrary IO. We’d like an effect to indicate that persistent-using functions are “doing database operations” without having to write IOE :> es everywhere.
It should be possible to share a persistent backend (e.g., SqlBackend) with specific subcomputations, so that we can provide backends either from a resource pool or share a single backend with the entire program.
Whether we use a resource pool should be invisible to code that uses our effect. We don’t want arbitrary functions to be able to manipulate the pool itself; our effect handler should be responsible for loaning out backend values from the resource pool.

To avoid getting bogged down writing an effectful-flavoured binding to all of persistent, we instead defined a slightly dodgy Persist effect that’s parameterised by the backend it looks for, and whose only job is to hold persistent actions:

data Persist backend :: Effect where
  LiftPersist :: ReaderT backend IO a -> Persist backend m a

-- Generate the `liftPersist` function and other machinery:
--
-- liftPersist ::
--   Persist backend :> es =>
--   ReaderT backend IO a ->
--   Eff es a
--
-- EDIT(2025-04-22): A previous version of this post erroneously
-- claimed that you had to write `liftPersist` by hand,
-- because of the type parameter in the effect.
$(makeEffect ''Persist)

The other half of setting up an effect is providing ways to discharge it. We’ll need two — one to provide a backend directly, and one that sources it from a resource pool. runPersistDirect is quite straightforward, but runPersistFromPool requires us to unlift Eff es down to IO so that we can use Data.Pool.withResource:

-- | Discharge a @'Persist' backend@ effect by providing a backend to use.
runPersistDirect ::
  forall backend es a.
  (IOE :> es) =>
  backend ->
  Eff (Persist backend ': es) a ->
  Eff es a
runPersistDirect backend = interpret $ \_ -> \case
  LiftPersist m -> liftIO $ runReaderT m backend

-- | Discharge a @'Persist' backend@ effect by loaning a backend from
-- a resource pool. The backend is returned to the pool after use.
runPersistFromPool ::
  forall backend es a.
  (IOE :> es) =>
  Pool backend ->
  Eff (Persist backend ': es) a ->
  Eff es a
runPersistFromPool pool eff =
  withEffToIO (ConcUnlift Ephemeral Unlimited) $ \runInIO ->
    liftIO . withResource pool $ \backend ->
      runInIO $ runPersistDirect backend eff

Using the `Persist backend` effect

Actually using this effect is about as simple as any other effectful effect:

-- | Insert a single record, as a minimal example
persistSomeData ::
  Persist SqlBackend :> es =>
  SomeData ->
  Eff es SomeDataId
persistSomeData someData =
  liftPersist @SqlBackend $ Persist.insert someData

Conclusions

This approach has worked fairly well during our experiment, and has some benefits that made us think it was worth writing up:

It captures the programmer’s intent quite well — a Persist backend :> es constraint clearly shows “database access happens here”;
We didn’t need a perfectly idiomatic effect definition to begin writing useful code against the effect;
After we worked through all the details and settled on the LiftPersist idea, it was reasonably quick to set up;
This technique can work for a large class of libraries (even ones that are quite tightly coupled to IO); and
Other developers didn’t need to touch the effect definition once it was written.

However, this quick-and-dirty approach has some pretty noticeable deficiencies that it’s worth being aware of:

The biggest problem is that liftPersist . lift can smuggle an arbitrary IO a action, so the Persist backend effect isn’t making an honest “only database access happens here” promise. Our developers are well-intentioned, so this is fine for a prototype;
The effect itself is opaque — the only possible interpretations are variations on “just run it”; and
It’s very easy to let the Persist backend :> es constraint float to the outermost layer of your program and discharge it using runPersistFromPool, not realising this shares a single backend with the whole program and defeats the entire point of the resource pool.

As an experimental technique, we’re pretty happy overall with how this style of effect turned out. Using a simpler effect definition let us run a cheap experiment with persistent, and it’s always possible to refine the effect later. This is the kind of refactoring Haskell excels at — we can remove liftPersist whenever we want and chase down the type errors until everything compiles, defining more precise operations under the Persist backend effect as they’re needed.

Open Source at Bellroy: Supporting Old GHC Versions

Jack Kelly — Wed, 19 Mar 2025 12:00:00 UT

Open Source at Bellroy: Supporting Old GHC Versions

Jack Kelly 2025-03-19

Technology

There has been some good discussion on the Haskell.org Discourse recently about how many GHC versions it’s reasonable for a maintainer to support. Some maintainers minimise their maintenance burden by rotating out support for anything older than the three most recent GHC major releases; others drop old GHC versions with extreme reluctance.

Bellroy’s primary business is making and selling good things. To help the rest of the business make and sell these good things, Bellroy’s Tech Team builds and maintains software. We are always grateful for the massive ecosystem of open-source software that we build upon, and to the community of maintainers who keep it alive. Bellroy talks a lot about using business as a force for good; in the open-source realm, that means trying to “pay forward” the gifts we’ve built upon. Concretely, this means trying to work upstream-first wherever it makes sense, and releasing useful open-source packages of our own.

This puts us in a challenging position. We want our open-source releases to be broadly useful, and at the same time we cannot take on an unlimited maintenance burden. Tech Team plans and executes its work using a variant of the Shape Up process. We work in eight-week cycles, spending six weeks on project work followed by two weeks of “cool down”. Routine dependency updates and open-source maintenance all have to fit into the cool-down period.

We have an internal document describing how to maintain our open-source packages, and thought we’d share our current thinking on GHCs, library bounds and related issues. This is not the final word from us or even a general recommendation. It’s what we’re doing at the moment, it might change, and it is most definitely not an SLA.

Which GHCs to target?

Within those constraints, which GHCs do we support for an open-source release? First and foremost, we must maintain support for whatever GHC major release is used in our internal Haskell monorepo. Currently, that’s GHC 9.6.

We also aim to officially support any compiler marked “suitable for use” on the GHC GitLab’s wiki, but that’s less of a sure thing. We use Nix for developer shells, and for our open-source libraries, we need to keep the amount of Nix wrangling manageable. These packages use a Nix Flake that only provides GHC, and leaves package selection to the cabal command. If our package’s dependencies haven’t yet updated, or the newest GHC isn’t yet in nixpkgs, we might not be able to support a GHC recently marked “suitable for use”. In such instances, we offer help where we can, but usually have to leave the GHC upgrades for a future project cycle.

When we support a GHC version, we list it in the tested-with field of the package’s .cabal file. The get-tested tool reads this field and lets us run a GitHub Actions matrix of all the GHC versions we support.

Which old versions to remove?

Supporting multiple GHC versions sometimes means adding compatibility cruft to a package. The two most common forms of this are CPP directives in Haskell sources and if impl(ghc >=x.y) conditions in .cabal files. Such cruft might be necessary to support occasional major releases marked “suitable for use”, but can accumulate over a long range of GHC releases. We don’t want the maintenance burden of a package to grow without bound, but we also don’t want to needlessly remove support for older versions when it doesn’t cost additional effort.

The compromise we’ve settled on is this: we don’t remove old GHC versions from tested-with until supporting them requires more code than just supporting the “suitable for use” versions.

Which Haskell dialect to use?

The GHC202X language editions are extremely convenient. We use them throughout our monorepo, but open-source releases use Haskell2010 and explicit {-# LANGUAGE #-} pragmas. This is mostly to make things easier for the emerging MicroHs compiler, and to avoid raising the GHC lower bound just for a little convenience.

For similar reasons, we avoid convenience extensions such as -XBlockArguments (GHC 8.6.1) and -XImportQualifiedPost (GHC 8.10.1). These extensions only add alternative syntax, and although they’re generally old enough to not really factor into GHC bounds, it doesn’t seem worth excluding old versions or alternate compilers by requiring them.

What library bounds to use for dependencies?

Setting library bounds correctly still feels like a bit of an art. One helpful resource is the GHC wiki: its table of boot library versions, lists the version of base and other fundamental libraries (e.g., text) that each GHC release ships with. We set base bounds to admit every GHC version listed in tested-with, and ensure that other boot libraries have bounds which admit the library versions bundled with each GHC.

For other dependencies, we start with the version we actually used, and bound it above by the next major version (which is when the Package Version Policy allows the next breaking change). The cabal outdated command then warns us about new package versions we need to accommodate. If we have to make code changes, we cut a new release; otherwise we can revise the bounds on Hackage.

As with old GHCs, we can’t spend unlimited effort supporting expansive bounds, but for important libraries undergoing major change (e.g., the aeson-2.0 transition), we do try to support both versions at least for a while.

What dependencies are worth taking on?

We try to depend on the minimal set of libraries necessary to get the job done, because dragging in half of Hackage to solve a simple problem feels impolite. Concretely, this mostly means avoiding custom prelude packages and convenient-but-heavy libraries like string-interpolate (which needs haskell-src-exts), and using the microlens-* family of lens libraries instead of lens. In many cases, the generic-lens and generic-optics packages remove the need for library authors to provide lenses at all.

Conclusion

We currently maintain a small handful of open-source Haskell libraries and have more packages awaiting spin-out. Our experience so far is that these principles strike a good balance between maintenance effort and supporting a fair range of compilers and library versions. Dependency update days have generally been smooth, and we expect these processes to scale reasonably well as we release more software to Hackage. We’re looking forward to announcing those new packages when they’re ready.

And if we have to come up with something else, we’ll probably write about that too.

Bellroy Technology Team: 2024 in Review

Mike Webb — Fri, 14 Feb 2025 00:00:00 UT

Bellroy Technology Team: 2024 in Review

Mike Webb 2025-02-14

Technology

2024 has been a transformative year for our Technology Team as Bellroy continues to grow from its startup roots into a medium-sized company. We’ve tackled ambitious projects, celebrated significant wins, and learned valuable lessons – especially about how plans often evolve in unexpected ways when your company is scaling up.

One of our main focuses this year was streamlining internal operations through automation. Our Production team now has a much smoother product data process, saving hours of manual work each week. We’ve modernized our employee purchase program, letting our staff experience our customer journey exactly as other customers do – which has already given us great insights into our user experience. We’ve also automated much of our refunds processing, freeing up our Finance team to focus on more strategic work. While these improvements might not be immediately visible to customers, they’ve been crucial in keeping us nimble as we grow.

Our e-commerce platform saw some exciting improvements this year. We completely rebuilt our shopping cart, making it significantly faster and giving customers real-time updates about promotions and stock levels. We rolled out single-use promotion codes, letting us reward loyal customers with special offers without the usual worries about codes being shared widely on social media. We’ve also been continuously fine-tuning our checkout process through various experiments – though we’ll admit not every test yielded the results we expected! And of course, throughout the year we supported all of Bellroy’s product launches and campaigns, ensuring our platform delivered smooth experiences during our busiest and most exciting moments.

We faced our share of challenges too. Our attempt to improve our A/B testing capabilities proved trickier than anticipated, reminding us why maintaining our own platform can be harder than using off-the-shelf solutions. We attempted to migrate our legacy Rails solution to a more cutting-edge solution relying on AWS Cloudfront Edge Functions and Kafka. We underestimated the complexity in incrementally migrating away from the legacy system while implementing the new system, and were unable to complete the project in the six weeks that our Shape Up methodology dictates. We learned important lessons about what these technologies can and can’t do. Still, we remain confident that building our own technology gives us the flexibility and control we need to create the best possible experience for our customers; we’re taking another stab at a simpler solution in 2025 without the use of Edge Functions. It’s possible that if we’d persisted with our original plan we’d have succeeded, but be in a worse position architecturally. This is a major strength of using Shape Up - being able to intelligently adjust or stop work based on what you learn during development, rather than doggedly sticking to the plan.

We completed several projects this year where we aimed to automate manual processes for other teams at Bellroy. These were always challenging projects, as they required us to understand the intricacies of other teams’ workflows and systems. In helping us understand these processes, the teams often uncovered problems or inconsistencies that the teams themselves hadn’t noticed before the project started. While we think the projects added a huge amount of value, they almost inevitably produced outcomes that did not resemble the original plans. This was a good reminder that the best solutions often come from collaboration and iteration, rather than trying to predict everything up front.

Behind the scenes, we’ve made solid progress moving our legacy Ruby code to Haskell, resulting in better performance and stability. We’ve also improved how we handle technical documentation, setting up automated systems to keep our team’s wiki documentation in sync with our code. This might sound trivial, but it’s making a real difference in how efficiently we can maintain and improve our platform as it grows more complex.

We were proud to contribute to the Haskell open-source community in 2024, releasing the following libraries out into the wild:

servant-activeresource: Servant endpoints compatible with Rails’s ActiveResources
tasty-golden-extra: Additional golden test helpers for the tasty-golden package
unliftio-servant-server: Use MonadUnliftIO on servant APIs

Looking ahead to 2025, we see our role evolving alongside Bellroy’s growth. The Technology Team now splits its focus between enhancing customer experience and improving company operations. We’re expanding into new territory – from integrating with major Chinese e-commerce platforms to experimenting with alternative delivery methods in challenging markets. This year’s experiences have taught us to stay adaptable and embrace solutions that might look different from our initial plans. As Bellroy continues to grow, we’re excited to keep finding better ways to help both our customers carry better and our company work smarter.

Solving a ResourceT-related space leak in production

Zelin Feng — Thu, 12 Dec 2024 00:00:00 UT

Solving a ResourceT-related space leak in production

Zelin Feng 2024-12-12

Technology

Working with Haskell is fun, until you have space leaks — especially when they occur in AWS Lambda Functions. In this blog post, we focus on how we plugged a particularly nasty space leak. In a future post, we will introduce a profiling tool for Haskell applications running on AWS Lambda.

Problem

We observed one of our Lambda Functions dying of OOM (Out Of Memory) failures, seemingly at random. The function had 1GB of memory allocated, and there seemed to be a big disconnect between the work it was doing and its memory utilization.

Hunting down space leaks

In a docker container with memory constraints, the free command shows the memory size of the host system instead of the container’s memory constraints.

For AWS Lambda Functions, we suspected that the situation might be similar. If there is anything special that the runtime system of a programming language needs to do to detect the correct memory limit, it might be done in those official Lambda Function runtimes — Golang and Python, for example — but not in Haskell.

If GHC RTS (Runtime System) is reading incorrect memory constraints, it may decide to defer garbage collection because it thinks there is still available memory.

To verify or eliminate this hypothesis, we added the RTS option -M1000M and increased the Lambda Function memory to 1792MB. Under these conditions, RTS should try its best to GC before the heap reaches 1000MB.

However, the Lambda Function still ran Out of Memory after a while:

This result showed that a space leak existed, but we couldn’t reproduce this problem in our local or testing environments.

Locating the space leak using profiling

We developed a tool to profile Haskell in AWS Lambda Functions running Haskell binaries, by sending the eventlog to Amazon S3. Here is the generated eventlog (converted to HTML by eventlog2html):

The memory usage and heap size both kept increasing over more than 10 minutes.

Why was the eventlog collected over more than 10 minutes when the Lambda Function timeout was set to a smaller value?

To answer this question, we needed to understand the execution model of Lambda Function. When a request arrives, AWS starts a new instance of our Lambda Function if there isn’t one already running. Each instance handles one request at any point in time. When a response has been returned, AWS can still keep your Function instance alive for an arbitrary period of time and reuse it when a new request arrives. Thus, if a memory block is somehow held in the global environment (e.g. some top level monad) and isn’t freed between requests, the Lambda Function will eventually run Out of Memory if it keeps receiving requests.

In the area chart and “Detailed” tab in the HTML report generated by eventlog2html, we found decodeEventASN1Repr and many related functions consuming a lot of memory. We started to suspect amazonka, because in this Lambda Function we exclusively use amazonka to make network requests.

Since the profiling result suggested that many AWS responses couldn’t be freed by the garbage collector, we decided to take a closer look at the source code of amazonka.

Diving into the code

We examined all AWS calls in our Lambda Function that had the space leak. It turned out that the amazonka implementation of the AWS SendTaskSuccess API was different from others. In amazonka, this API is implemented in Amazonka.StepFunctions.SendTaskSuccess. All amazonka request types have to provide an instance of the AWSRequest typeclass. Here is how the AWSRequest instance is implemented on SendTaskSuccess:

instance Core.AWSRequest SendTaskSuccess where
  type
    AWSResponse SendTaskSuccess =
      SendTaskSuccessResponse
  request overrides =
    Request.postJSON (overrides defaultService)
  response =
    Response.receiveEmpty
      ( \s h x ->
          SendTaskSuccessResponse'
            Prelude.<$> (Prelude.pure (Prelude.fromEnum s))
      )

In response, it calls receiveEmpty, which is implemented as:

receiveEmpty ::
  MonadResource m =>
  (Int -> ResponseHeaders -> () -> Either String (AWSResponse a)) ->
  (ByteStringLazy -> IO ByteStringLazy) ->
  Service ->
  Proxy a ->
  ClientResponse ClientBody ->
  m (Either Error (ClientResponse (AWSResponse a)))
receiveEmpty f _ =
  stream $ \r s h _ ->
    liftIO (Client.responseClose r) $> f s h ()

Despite the complex parameters, the only relevant detail is that in receiveEmpty, responseClose is called without reading the entire HTTP response.

This seems innocent — and it is. The problem laid in http-conduit. amazonka calls the function http to implement its network requests. At the time of our investigation, the http function was implemented as:

http :: MonadResource m
     => Request
     -> Manager
     -> m (Response (ConduitM i S.ByteString m ()))
http req man = do
    (key, res) <- allocate (Client.responseOpen req man) Client.responseClose
    return res { responseBody = do
                   HCC.bodyReaderSource $ responseBody res
                   release key
               }

Note that allocate is called — it’s a function from ResourceT. Many Haskell developers may have been using ResourceT in their daily job without realizing what it does under the hood. ResourceT maintains a mutable Map of external resources with their cleanup functions. runResourceT automatically calls the cleanup functions right before it exits.

In many applications (including ours), ResourceT is added to the global application-level monad transformer stack, and runResourceT only runs once globally. Then, this global ResourceT becomes a global registry of resources. Any registered resource is released either explicitly by calling release or when the entire application exits. The garbage collector can’t free any heap object if it’s directly or indirectly referenced by a registered resource.

In the implementation of function http, release is called only when the full response body is consumed. But it’s not uncommon to only consume part of the response body. You could be making a POST or PUT request and the only thing you care in the response is if the status code is 200 or not. You could be reading a file from S3 and optionally discard the remaining content if your program decides to do so. In these scenarios, the resource won’t be released before the end of runResourceT, resulting in a space leak.

Creating a minimal reproducer

To prove the theory above, we created a minimal reproducer so that we wouldn’t need to test it in our production code.

main :: IO ()
main = do
  env <- Amazonka.newEnv Amazonka.discover
  void . Amazonka.runResourceT $
    for (replicate 1000 ()) $ \_ -> do
      randomStr <- fmap (T.pack . take 1000000) . lift $ getRandomRs ('a', 'z')
      void . Amazonka.send env $
        S3.newPutObject "bellroy-eventlog-test" "test" (Amazonka.toBody randomStr)

In this example, we make 1000 PutObject calls to Amazon S3, with random large request bodies. Like SendTaskSuccess, the amazonka implementation of S3 PutObject operation also uses receiveEmpty internally. runResourceT exits only after all 1000 requests are sent to S3. If our theory is correct, memory usage of this test program should keep increasing until it runs Out of Memory.

Here is the profiling result before we apply any fix:

There are two possible ways to fix this test program.

Call runResourceT on each Amazonka.send.
In http-client, call release when responseClose is called.

After implementing either fix, our new profiling output looks like this:

We submitted a PR to snoyberg/http-client so with subsequent versions of http-client, our minimal reproducer no longer has a space leak 🎉

Lessons to learn

By calling any function whose type includes MonadResource m => m a or ResourceT m a, you should be aware that some resource is registered to your current monad transformer stack and may hold large blocks of memory.

To release that resource, either:

Read the documentation of your library, and find out if there is any function that allows releasing the resource explicitly. Call that function. In http-client and http-conduit, the function to release a connection is responseClose.
Or, make your runResourceT scope smaller.

Save Effort: Build a Bash One-Liner

Luke Worth — Thu, 22 Aug 2024 00:00:00 UT

Save Effort: Build a Bash One-Liner

Luke Worth 2024-08-22

Technology

Preliminary disclaimer: the author firmly believes that an operating system makes for a perfectly good IDE; he won’t be taking questions about this.

The Bellroy tech team has a longstanding goal to eliminate our Ruby codebase¹. To help us reach that goal, the whole team can see a graph of LOC (Lines Of Code) currently stored in git for each programming language we use, plotted over time. This is updated once per week, and watching the Ruby line go downward encourages us to reduce the amount of Ruby we add each week, and (more importantly) to increase the amount we delete. However, like most metrics, it doesn’t quite directly measure progress toward our goal: if someone splits a complicated line of code into multiple lines the code becomes easier to understand, and thus easier to remove, but the total LOC moves away from zero. This problem was brought to the forefront recently when we instated automatic formatting of Ruby code using the syntax_tree tool (which, like all our other formatters, has almost no configuration and so avoids bike-shedding about code layout). If this tool can’t fit a single Ruby statement into one line it will usually break it into many lines: when we run it on every Ruby file of the bellroy.com codebase we gain about 6,700 lines of Ruby, which looks like we are adding Ruby code, which makes us sad.

To deal with this problem I looked for an alternative metric that would be easy to apply to our codebase, and robust to re-formatting. I quickly discovered the ABC Software Metric which is calculated from the number of Assignments, Branches, and Conditionals in the code, and which looked like a much better proxy for how much functionality lives in Ruby. And it turns out that RuboCop contains an implementation of this metric! But RuboCop is designed only to alert you when that metric is exceeded - at present it provides no easy way to display the raw ABC size.

At this point, the obvious thought crossed my mind: perhaps I could fork RuboCop, add the functionality I need, and submit a PR (because I am a good person). But another thought also crossed my mind: that sounds like hard work that I’d prefer to avoid. And of course, because I use UNIX² (the world’s best IDE), wouldn’t it be easier to use Rubocop as-is and use a bunch of duct tape and WD-40³ to get the outcome I want? The short answer is “yes, much easier.” Let’s work through this together!

First we must ask the most basic question: is the raw metric exposed by RuboCop at all? A quick search reveals what seems to be an error message format string, referencing something promisingly called complexity. Let’s see this error message in action:

$ nix run nixpkgs#rubocop -- --only Metrics/AbcSize config/application.rb
Inspecting 1 file
.

1 file inspected, no offenses detected

Ah, the documentation says default Max value is 17. Let’s see what happens if we set it to 0:

$ cat >.rubocop.yml <<<'Metrics/AbcSize: {Max: 0}'
$ nix run nixpkgs#rubocop -- --no-display-cop-names --only Metrics/AbcSize config/application.rb
...snip...
Offenses:

...snip...
config/application.rb:59:3: C: Assignment Branch Condition size for settings is too high. [<2, 4, 1> 4.58/0]
  def self.settings ...
  ^^^^^^^^^^^^^^^^^
...snip...

There it is! 4.58 is indeed greater than 0. RuboCop displays one of these messages for each method, and now the end is in sight. Now all we have to do is run this for every file in the codebase, extract all the complexity numbers, and sum them up.

POSIX comes with a number of raw text processing tools but, no matter how powerful they are, it is simply easier to deal with structured data. I’m talking about JSON, a data format that began life as a quick way to transmit data to and from web browsers, which is now the de-facto standard for encoding structured data in plaintext; and RuboCop knows how to produce it.

$ nix run nixpkgs#rubocop -- --no-display-cop-names --only Metrics/AbcSize --format json config/application.rb
{"metadata":{"rubocop_version":"1.62.1","ruby_engine":"ruby","ruby_version":"3.1.6","ruby_patchlevel":"260","ruby_platform":"arm64-darwin23"},"files":[{"path":"config/application.rb","offenses":[..snip..,{"severity":"convention","message":"Assignment Branch Condition size for settings is too high. [<2, 4, 1> 4.58/0]","cop_name":"Metrics/AbcSize","corrected":false,"correctable":false,"location":{"start_line":59,"start_column":3,"last_line":61,"last_column":5,"length":114,"line":59,"column":3}},..snip..]}],..snip..}

Looks worse to you and I, but for jq it is a delectable treat. Let’s pull out the bit we care about:

$ nix run nixpkgs#rubocop -- --no-display-cop-names --only Metrics/AbcSize --format json config/application.rb | jq -r '.files[].offenses[].message'
..snip..
Assignment Branch Condition size for settings is too high. [<2, 4, 1> 4.58/0]
..snip..

Getting closer, but it’s time for a short Q&A:

Q: What the heck is nix run nixpkgs#rubocop -- and why don’t you use bundle exec rubocop like a normal person?

A: nix is awesome. We use it all the time, on Linux, macOS, and Windows. Invoking RuboCop this way means I don’t have to install it, nor add it to the Gemfile, nor modify the code at all. Anyone can run this command on any Rails codebase and it’ll just work.
Q: What was that <<< thing before?

A: That is a here string. In Bash, this basically means “send the adjoining text to standard input”. Bash is full of hidden gems. I recommend periodically reading a random section of the reference manual.
Q: Isn’t that command getting long?

A: Yes, it is. That doesn’t matter because nobody else is going to see this (except in my case, where I’m publishing it onto the internet). We want to move quickly, not produce the “best” code.
Q: How good is jq?

A: I know, right?!

Unfortunately, for our ABC size calculator, we have eliminated all of the available JSON so jq is no longer useful; but some folks predicted this problem in the 1970s, and gave us AWK. Let’s try to pick out just the complexity number (which is the 4.58 we keep seeing):

$ MESSAGE='Assignment Branch Condition size for settings is too high. [<2, 4, 1> 4.58/0]'
$ awk '{print $13}' <<<"$MESSAGE"
4.58/0]

And let’s get rid of that /0]:

$ awk '{gsub(/\/0]/, "", $13); print $13}' <<<"$MESSAGE"
4.58

Putting it all together:

$ nix run nixpkgs#rubocop -- --no-display-cop-names --only Metrics/AbcSize --format json config/application.rb | jq -r '.files[].offenses[].message' | awk '{gsub(/\/0]/, "", $13); print $13}'
..snip..
6.16
3.74
4.58
3
3.74

That’s a list of ABC sizes of all methods in config/application.rb! We can use AWK to sum them up:

$ nix run nixpkgs#rubocop -- --no-display-cop-names --only Metrics/AbcSize --format json config/application.rb | jq -r '.files[].offenses[].message' | awk '{gsub(/\/0]/, "", $13); sum += $13} END {print sum}'
21.22

“Do it on the whole codebase,” you yell. I yell back, “OK, give me a minute” and run:

$ find . -name '*.rb'
..big list of every .rb file including vendored gems and other junk..

This is not quite what you wanted, and a bit unprincipled. We don’t want the files on the filesystem, we want the ones in the codebase. The canonical list of codebase files lives in Git, so let’s ask Git instead:

$ git ls-files | grep '\.rb$'
..exactly the list of Ruby files in our codebase..

Good. Let’s plumb this into our “program”:

$ nix run nixpkgs#rubocop -- --no-display-cop-names --only Metrics/AbcSize --format json $(git ls-files | grep '\.rb$') | jq -r '.files[].offenses[].message' | awk '{gsub(/\/0]/, "", $13); sum += $13} END {print sum}'
9910.94

Voilà! The ABC size of bellroy.com’s Rails backend⁴.

I want to share this one-liner so that anyone can run it in any Ruby codebase, but it still depends on the .rubocop.yml we deposited near the beginning. One way around this is to simply include the cat step we did earlier, but that would overwrite any existing .rubocop.yml, and would leave junk behind. Let us reach into our bag of fancy Bash features that everyone should know, and pull out process substitution. If you pass --config some_file.yml to RuboCop, it will read its configuration from some_file.yml instead of .rubocop.yml. If you pass --config <(echo 'Metrics/AbcSize: {Max: 0}'), RuboCop will read its configuration from a temporary named pipe which appears to contain the desired configuration. That lets us distribute a truly stand-alone tool that can run anywhere and leaves no trace, and we didn’t need to write any Ruby to do it. Here’s the final version, formatted for readability:

nix run nixpkgs#rubocop -- \
        --config <(echo 'Metrics/AbcSize: {Max: 0}') \
        --no-display-cop-names \
        --only Metrics/AbcSize \
        --format json \
        $(git ls-files | grep '\.rb$') \
        | jq -r '.files[].offenses[].message' \
        | awk '{gsub(/\/0]/, "", $13); sum += $13} END {print sum}'

We’ve written about this before:
- Our technology stack, and how we got here
- Technology stack transitions are hard
↩︎
I use macOS but anything vaguely POSIX-compatible should work.↩︎
Clearly this is a metaphor. In this post I’ll be using nix, bash, jq, and gawk, but if you want to follow along with actual duct tape and WD-40, be my guest.↩︎
Two weeks ago this number was 11574.6!↩︎

How We Use Both Process Orchestration and Process Choreography

Chad Musick — Thu, 08 Aug 2024 00:00:00 UT

How We Use Both Process Orchestration and Process Choreography

Chad Musick 2024-08-08

Data and Analytics

Message passing in software

At their most atomic, all software processes rely on message passing. Various architectures are constructed or (for those of us who like to anthropomorphize our work) arise naturally around the messengers and the messages to control their flow. In hardware, these messages may be stored in memory registers. In software, these messages are handled in a variety of ways according to the language and its operating model.

In online systems, distributed processes are typical. In an ideal example, you open a browser and come to Bellroy’s website. We serve you enticing images of our products and show you their features. You pick one or more and order. We accept your order, take payment, and instruct one of our warehouses to send your new carry goods to you.

There are lots of obvious message partners here:

User to/from their browser
The user’s browser to/from Bellroy’s website
Bellroy’s website to/from Bellroy’s internal applications
Bellroy’s internal applications to/from the payment service
Bellroy’s internal applications to/from the accounting system
Bellroy’s internal applications to/from the warehouses

Three basic models of message passing predominate in distributed systems:

Synchronous: The sender sends a message and waits for the recipient to fully respond
Asynchronous to queue: The sender sends a message, which enters a queue and is (maybe) processed later
Asynchronous to subscribers: The publisher sends a message, which it then forwards to a set of zero or more subscriber processes.

In practice, these are often combined. A browser might send a synchronous request to a website, which might synchronously submit an event to an event-stream platform like Apache Kafka, which might acknowledge that message and then publish it to a number of subscribers. It gets complicated. Without some kind of coordination, that complication makes systems hard to reason about and hard to build correctly.

Many different methods can be used to harness the complications of this distributed process without losing the power of the message-passing paradigm. Most such methods can be classified as orchestration or choreography (or both). Orchestration is well-suited to centrally directing the actions of systems known to the orchestrator, such as in workflows, and ill-suited to handling reactions from systems unknown to the orchestrator, such as auditing and analytics systems. In contrast, choreography is well-suited to handle reactions from arbitrary sytems but is typically slower than orchestration.

Orchestration: Centralised control

Orchestration acts in a way similar to historical software practices. There may be many different services (or functions), and these may be spread across multiple computers—and for orchestration, distributed systems are typically involved—but a centralised controller calls out to these services as needed to accomplish a specific task, end-to-end. For most purposes, an orchestrated system can be treated as a black box: you put your choices in, you get results or effects out.

There are different theoretical models for orchestration. Among these, the concept of a finite state machine is the most widely implemented. In a finite state machine, only a finite number of “states” are possible, and the machine transitions from one state to another as it runs. Additional information can be carried through the machine, but this information is not typically considered to be part of the state. A variety of open source and proprietary orchestration engines use this basic model for their operation. At Bellroy, we use AWS Step Functions for orchestration of some AWS services and Apache NiFi for much of our data orchestration.

Choreography: Independent actors deciding how to perform

In contrast with orchestration, choreography does not rely on a centralised understanding of the order in which things will happen. Instead, each component process listens for signals and decides how to respond to those signals; this can include ignoring them.

The granularity of choreography can differ greatly, from pixels in a diagram to the behavior of entire companies. At the most granular level of choreography are cellular automata, which behave by consistent rules that consider only their current state and the state of their neighbors. Conway’s Game of Life is a standard example of this type of system. Despite the simplicity of Conway’s rules, complex systems can arise from non-local effects of local interactions. In business, orchestration is the typical way for companies to interact with one another; companies make requests but don’t typically dictate how those requests are accomplished or concern themselves with the internal details of their vendors.

Where is the boundary?

Orchestration and choreography can look similar in their implementation details for a particular system, but there are important distinctions between them. In a purely orchestrated system, the orchestrator is concerned with all elements of execution, so replacing any element requires the system to be updated. In a purely choreographed system where all reacting systems are also purely choreographed, no work is performed because every actor is simply signalling others.

Nearly all choreography will be started by some kind of orchestration, and some choreographed responses will themselves be orchestrated. The boundary between the two types of control emerges primarily from the degree of control exerted over the exact process.

In the software context, publish–subscribe (pub/sub) systems are often used to define the boundaries between orchestration (direct invocation) and choreography (message publication). The publisher of a message does not know how many systems (if any) will respond to the message. The subscriber who receives a message does not know how many other subscribers (if any) are also responding to the message.

In the business context, there is often a defined set of steps to follow for a process. This is an orchestrated process. However, the business may pay for an outside vendor to achieve some outcome without caring how the other business does so. From the client perspective, this is a choreographed step.

How Bellroy uses both orchestration and choreography

Bellroy needs to accomplish several distinct but related tasks to effectively serve our current and future customers. Let’s look at how Bellroy does this at high level.

Serve our website: Primarily choreography

Bellroy’s website relies on a variety of different elements being loaded, such as images, text, scripts, and design specifications. The order in which these are loaded is not important. Bellroy’s servers respond to these requests independently. The browser requests elements as it learns about them, such as reading image URLs from HTML tags, and renders the page when it has enough information to do so.
Process orders: Primarily orchestration

Our order fulfilment process includes the following steps:
- Check that stock is available for the shipping address.
- Calculate the order total and verify that it matches what was displayed.
- Collect payment information in the front-end and relay it to our payment processor; then, record the results (which may take a very long time to come if manual verification is necessary).
- Send notices to our warehouses to ship the products.
- Record the transaction in our accounting system.
Perform internal processes: Primarily choreography
- Gather orders from other electronic and physical marketplaces and ship those that require shipping.
- Keep track of inventory levels.
- Analyze how changes to Bellroy’s website affect sales.
- Predict how much inventory to order from suppliers.

When the order of execution is unimportant, choreography allows for design efficiency at the cost of execution complexity; the same initial signal can result in many different routes for execution, including routes not known at the time of building the thing that generates the signal.

In many ways, orchestration is the micromanagement of processes. These are instructed to execute in a particular order, and the response status and result of each process is tracked with the intent of using the information to inform the next process. This is necessary when the order of execution matters (we don’t want to collect payment if we can’t fulfil an order, for example). Beyond this, it can be very efficient since it does not inherently require asynchronous message passing.

Used together, they can provide fast, flexible, reliable service that is easy to audit and expand without refactoring older systems. This requires conscious effort and an understanding of the tradeoffs involved in the choices of system types and boundaries between them.

Servant on AWS Lambda, and Two New Libraries

Jack Kelly — Mon, 08 Jul 2024 12:00:00 UT

Servant on AWS Lambda, and Two New Libraries

Jack Kelly 2024-07-08

Technology

In a previous post, I mentioned that we run most of our Haskell code on AWS Lambda. Today, I want to look inside those binaries, discuss the libraries we use to do “serverless” web services on AWS, and highlight a few new libraries we’ve recently open sourced.

Serverless Web Services on AWS…

Lambda is AWS’s “function-as-a-service” product. You upload your code to AWS (either as a zip file or a container image), and when it is “invoked”, AWS will find some hardware to run it on. Most invocations are done using an AWS SDK, where you provide a JSON payload and receive a JSON response.

It’s possible to build “serverless” web services out of Lambda Functions by integrating your functions with other AWS products. We currently use AWS API Gateway for this. API Gateways have several integration types, but the one we use most often is the AWS_PROXY integration. This integration has the API Gateway call a Lambda Function to handle the HTTP request, using JSON representations of the HTTP request and response.

…in Haskell

Hackage has a few different “Lambda runtime” packages, but the one we have settled on is hal. It provides functions that implement the request/response loop that Lambda functions are required to perform (requesting the next invocation from AWS, returning results to AWS, etc.) as well as marshalling types for common AWS events. Its maintainer has also been a delight to work with, and we’re grateful for his diligence and receptiveness to enhancements and bug fixes. For API Gateway, hal provides the AWS.Lambda.Events.ApiGateway.ProxyRequest and AWS.Lambda.Events.ApiGateway.ProxyResponse modules, describing the HTTP request/response JSON structures used by an API Gateway REST API.

To fit hal’s mRuntime function and implement our request handler, we need to provide a function ProxyRequest -> m ProxyResponse. This is morally very similar to the Application type provided by wai, the standard interface between Haskell web servers and web frameworks. wai is Haskell’s version of Ruby’s rack or Python’s wsgi: it provides an Application type for frameworks to provide and servers to consume and standard types for HTTP requests and responses, which together form a narrow waist between servers and clients.

If we can convert a hal ProxyRequest to a wai Request and a wai Response to a hal ProxyResponse, we can lift nearly any wai Application into a Lambda Function. One of our first open-source Haskell packages, wai-handler-hal, does exactly this. We have a working example on GitHub if you want to experiment with it, and we were very pleased to discover that the ZuriHac registration system moved off a custom Lambda runtime to hal+wai-handler-hal.

Building APIs: Servant

Now that we can run standard web application stacks on Lambda, which one do we use? Since we build a lot of APIs, servant is the obvious choice. It’s an excellent framework for specifying and building out APIs, but intermediate Haskellers often struggle to implement Servant servers using custom monads.

Many application monads just carry around a “context” of read-only data such as environment variables, and do not do any tricky control flow. Such monads admit a MonadUnliftIO instance, and Servant APIs built on top of them can be converted to wai Applications in a generic way. We recently released unliftio-servant-server, a package of helpers to do this for both traditional and record-based Servant APIs.

Serving ActiveResource APIs

Until recently, the majority of Bellroy’s systems were written in Ruby, often on Rails. As we’ve migrated more systems to Haskell, we’ve occasionally found the activeresource library from Rails to be quite helpful. activeresource is an object-relational mapper (ORM) for RESTful API endpoints: it represents individual instances of resources as Ruby objects with a similar interface to Rails’ database ORM, activerecord. When the database structures are simple enough (not too many joins), the similarity between activerecord and activeresource has let us carve out functionality from our Rails projects without extensive rewrites.

activeresource is a bit particular about the routes and HTTP status codes that it expects, so we developed servant-activeresource to capture the pattern. By encoding the conventions that activeresource expects, we are sure to get the Haskell side right every time.

Conclusions

This basic stack (APIs in Servant, wrapped in wai-handler-hal, running on AWS Lambda, behind an API Gateway) has proven very effective for us, and has become the default way we build services at Bellroy. It’s not a universal answer — some problems are better solved with traditional virtual machines or other persistent compute, Application Load Balancers become more cost-effective than API Gateways at scale, and CloudFront can now use Lambda URLs as origins — but it’s worked very well for us so far. We’ve been lucky that there’s so many good libraries in this space on Hackage, and we’re pleased that we’ve been able to give back with a few of our own.

Using Dhall To Manage GitHub Actions Workflows

Alvin Tang — Tue, 04 Jun 2024 00:00:00 UT

Using Dhall To Manage GitHub Actions Workflows

Alvin Tang 2024-06-04

Technology

In my previous post, I talked about how we use Github Actions to automate our workflows. As promised, I will show how we use Dhall to manage our GitHub Actions files.

Don’t Repeat Yourself

“Don’t Repeat Yourself” (DRY) is a programming principle that reduces repetition in code. As we created more actions in a growing number of repositories, we noticed that there were a lot of repeating steps, jobs, and even whole workflows. Github Actions has features to make workflows DRY such as composite actions and reusable workflows. However, these are useful only in certain situations:

Composite actions are useful for running a sequence of steps e.g. Docker setup, login, then fetch an image.
Reusable workflows are useful for running whole jobs e.g. building and testing a Ruby on Rails repository

Github Actions does not have an easy way to reuse steps or small groups of steps. Dhall gave us that flexibility.

Life before Dhall

Every time we create a new repository, workflow files are also added. The easiest thing to do is copy a workflow file from another repository and edit it. Since the workflow has to be specific to the repository, some steps have to be modified. Eventually, this leads to repetitiveness and consistency problems. The workflows look similar but not the same. We also encountered template format errors such as misaligned tabs and missing required keys.

As the number of repositories grows, so do the workflows. We needed a more reliable way of creating new workflows and maintaining old ones.

Enter the Dhall configuration language

Dhall is a programmable configuration language. It creates JSON and YAML files with the benefits of programming principles such as types, let expressions, imports, and functions. I won’t go into detail about all features of Dhall. The documentation provides an overview and some tutorials. In this post, I will just look at some of its features that help us manage our YAML files.

`github-actions-dhall`

We did not re-invent the wheel with Dhall and GitHub Actions. There’s already an open-source project for it in GitHub. We started with that and contributed along the way whenever we thought it was useful.

Let’s start with a simple Dhall file to write a workflow that builds Node.js. We want to produce the GitHub Actions workflow below:

name: Node.js CI
on:
  pull_request:
    branches:
      - main
  push:
    branches:
      - main
jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: "actions/checkout@v2"
      - name: Use Node.js
        uses: "actions/setup-node@v2"
        with:
          node-version: '21'
      - name: Install Dependencies
        run: npm install
      - name: Run Build
        run: npm run build

The Dhall file to produce the YAML above:

let GitHubActions = https://raw.githubusercontent.com/regadas/github-actions-dhall/HEAD/package.dhall

let checkoutStep = GitHubActions.Step::{
    , name = Some "Checkout"
    , uses = Some "actions/checkout@v2"
    }

let buildJob = GitHubActions.Job::{
    , runs-on = GitHubActions.RunsOn.Type.`ubuntu-latest`
    , steps = [
      Step,
      GitHubActions.Step::{
        , name = Some "Use Node.js"
        , uses = Some "actions/setup-node@v2"
        , `with` = Some (toMap {
          , node-version = "21"
          })
        },
      GitHubActions.Step::{
        , name = Some "Install Dependencies"
        , run = Some "npm install"
        },
      GitHubActions.Step::{
        , name = Some "Run Build"
        , run = Some "npm run build"
        }
      ]
    }

let workflow = GitHubActions.Workflow::{
    , name = "Node.js CI"
    , on = GitHubActions.On::{
      , push = Some GitHubActions.Push::{
        , branches = Some ["main"]
        }
      , pull_request = Some GitHubActions.PullRequest::{
        , branches = Some ["main"]
        }
      }
    , jobs = toMap { buildJob }
    }

in workflow

In this file, we define several blocks using the let expression. First, we import github-actions-dhall. github-actions-dhall has defined types for a step, job and workflow. We use Dhall’s record completion feature so we only fill in values that are not default. You can check the repository for the default values.

To produce a YAML file from Dhall, run the dhall-to-yaml executable:

dhall-to-yaml --file workflow1.dhall > workflow1.yaml

You’ll notice that Dhall produces a slightly different YAML file:

jobs:
  buildJob:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: "actions/checkout@v2"
      - name: Use Node.js
        uses: "actions/setup-node@v2"
        with:
          node-version: '21'
      - name: Install Dependencies
        run: npm install
      - name: Run Build
        run: npm run build
name: Node.js CI
on:
  pull_request:
    branches:
      - main
  push:
    branches:
      - main

Dhall orders the keys alphabetically. This takes some getting used to but this will work just the same. YAML doesn’t care about the order of keys.

The Dhall file is still longer than the YAML file, right? Let’s make the workflow a bit more complex by creating two jobs: one that builds node 21, and one that builds node 20. From the workflow file, we notice that only one step needs to change, actions/setup-node@v2. We need to change this to use node-version 20. In a YAML file, this means we copy the whole job then replace the value for that step only. In Dhall, I can refactor the common steps and reference the corresponding let expressions in the jobs that need it:

let GitHubActions = https://raw.githubusercontent.com/regadas/github-actions-dhall/HEAD/package.dhall

let checkoutStep = GitHubActions.Step::{
    , name = Some "Checkout"
    , uses = Some "actions/checkout@v2"
    }

let npmInstallStep =
      GitHubActions.Step::{
        , name = Some "Install Dependencies"
        , run = Some "npm install"
        }

let npmRunBuildStep =
      GitHubActions.Step::{
        , name = Some "Run Build"
        , run = Some "npm run build"
        }

let buildNode21Job = GitHubActions.Job::{
    , runs-on = GitHubActions.RunsOn.Type.`ubuntu-latest`
    , steps = [
      checkoutStep,
      GitHubActions.Step::{
        , name = Some "Use Node.js"
        , uses = Some "actions/setup-node@v2"
        , `with` = Some (toMap {
          , node-version = "21"
          })
        },
      npmInstallStep,
      npmRunBuildStep
      ]
    }

let buildNode20Job = GitHubActions.Job::{
    , runs-on = GitHubActions.RunsOn.Type.`ubuntu-latest`
    , steps = [
      checkoutStep,
      GitHubActions.Step::{
        , name = Some "Use Node.js"
        , uses = Some "actions/setup-node@v2"
        , `with` = Some (toMap {
          , node-version = "20"
          })
        },
      npmInstallStep,
      npmRunBuildStep
      ]
    }

let workflow = GitHubActions.Workflow::{
    , name = "Node.js CI"
    , on = GitHubActions.On::{
      , push = Some GitHubActions.Push::{
        , branches = Some ["main"]
        }
      , pull_request = Some GitHubActions.PullRequest::{
        , branches = Some ["main"]
        }
      }
    , jobs = toMap { buildNode21Job, buildNode20Job }
    }

in  workflow

The Dhall file is getting bigger. We can utilise the import feature to break this file into multiple smaller files. Create a steps.dhall file and let’s put all our step definitions there:

let GitHubActions = https://raw.githubusercontent.com/regadas/github-actions-dhall/HEAD/package.dhall

let checkoutStep = GitHubActions.Step::{
    , name = Some "Checkout"
    , uses = Some "actions/checkout@v2"
    }

let npmInstallStep =
      GitHubActions.Step::{
        , name = Some "Install Dependencies"
        , run = Some "npm install"
        }

let npmRunBuildStep =
      GitHubActions.Step::{
        , name = Some "Run Build"
        , run = Some "npm run build"
        }

in { checkoutStep, npmInstallStep, npmRunBuildStep }

The variables inside in { ... } are exported. We will then import the steps.dhall file in our original Dhall file:

let GitHubActions = https://raw.githubusercontent.com/regadas/github-actions-dhall/HEAD/package.dhall

let Steps = ./steps.dhall

let buildNode21Job = GitHubActions.Job::{
    , runs-on = GitHubActions.RunsOn.Type.`ubuntu-latest`
    , steps = [
      Steps.checkoutStep,
      GitHubActions.Step::{
        , name = Some "Use Node.js"
        , uses = Some "actions/setup-node@v2"
        , `with` = Some (toMap {
          , node-version = "21"
          })
        },
      Steps.npmInstallStep,
      Steps.npmRunBuildStep
      ]
    }

let buildNode20Job = GitHubActions.Job::{
    , runs-on = GitHubActions.RunsOn.Type.`ubuntu-latest`
    , steps = [
      Steps.checkoutStep,
      GitHubActions.Step::{
        , name = Some "Use Node.js"
        , uses = Some "actions/setup-node@v2"
        , `with` = Some (toMap {
          , node-version = "20"
          })
        },
      Steps.npmInstallStep,
      Steps.npmRunBuildStep
      ]
    }

let workflow = GitHubActions.Workflow::{
    , name = "Node.js CI"
    , on = GitHubActions.On::{
      , push = Some GitHubActions.Push::{
        , branches = Some ["main"]
        }
      , pull_request = Some GitHubActions.PullRequest::{
        , branches = Some ["main"]
        }
      }
    , jobs = toMap { buildNode21Job, buildNode20Job }
    }

in  workflow

We can do the same exercise for the job expressions to make it even shorter. For now, let’s keep it this way.

Now how about the Use node.js step? They look similar and only differ in the node version. In Dhall, we can write functions to handle this. We create a function useNodeVersionStep and pass a nodeVersion argument of type Text and it outputs a GitHubActions.Step.Type. Then we call this function where we want to add a step, then pass the node version that we’d like to use:

let GitHubActions = https://raw.githubusercontent.com/regadas/github-actions-dhall/HEAD/package.dhall

let Steps = ./steps.dhall

let useNodeVersionStep
    : Text → GitHubActions.Step.Type
    = λ(nodeVersion : Text) →
        GitHubActions.Step::{
        , name = Some "Use Node.js"
        , uses = Some "actions/setup-node@v2"
        , `with` = Some (toMap {
          , node-version = nodeVersion
          })
        }

let buildNode21Job =
    GitHubActions.Job::{
    , runs-on = GitHubActions.RunsOn.Type.`ubuntu-latest`
    , steps = [
      Steps.checkoutStep,
      (useNodeVersionStep "21"),
      Steps.npmInstallStep,
      Steps.npmRunBuildStep
      ]
    }

let buildNode20Job = GitHubActions.Job::{
    , runs-on = GitHubActions.RunsOn.Type.`ubuntu-latest`
    , steps = [
      Steps.checkoutStep,
      (useNodeVersionStep "20"),
      Steps.npmInstallStep,
      Steps.npmRunBuildStep
      ]
    }

let workflow = GitHubActions.Workflow::{
    , name = "Node.js CI"
    , on = GitHubActions.On::{
      , push = Some GitHubActions.Push::{
        , branches = Some ["main"]
        }
      , pull_request = Some GitHubActions.PullRequest::{
        , branches = Some ["main"]
        }
      }
    , jobs = toMap { buildNode21Job, buildNode20Job }
    }

in  workflow

The last three Dhall files that we made produce the same workflow YAML files:

jobs:
  buildNode20Job:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: "actions/checkout@v2"
      - name: Use Node.js
        uses: "actions/setup-node@v2"
        with:
          node-version: '20'
      - name: Install Dependencies
        run: npm install
      - name: Run Build
        run: npm run build
  buildNode21Job:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: "actions/checkout@v2"
      - name: Use Node.js
        uses: "actions/setup-node@v2"
        with:
          node-version: '21'
      - name: Install Dependencies
        run: npm install
      - name: Run Build
        run: npm run build
name: Node.js CI
on:
  pull_request:
    branches:
      - main
  push:
    branches:
      - main

How we use Dhall

Now we have a way to refactor common steps, jobs, and also create functions. We decided to put all our workflows in one repository and refactored them to make it more reusable. We have a folder structure like below:

dhall - contains all Dhall files
- deps - contains dhall dependencies such as github-actions-dhall
- libs - contains common Dhall definitions like functions and types
- steps - contains reusable steps or groups of steps
- jobs - contains reusable jobs
- composite - composite actions Dhall files with one directory per composite action
- workflows - workflow Dhall files with one directory for each repository managed
composite - the composite action YAML files with one directory for each composite action
workflows - the workflow YAML files with one directory for each repository managed

Generating the YAML files from the Dhall files is straightforward. We opted to use a simple Bash script instead of sophisticated build tools such as make since we don’t need to track dependencies.

#!/usr/bin/env bash
set -eou pipefail

echo -n "Cleanup files before generating..."
rm -rf composite/
rm -rf workflows/
echo "Done!"

echo -n "Generating composite actions..."
for COMPOSITE in $(ls dhall/composite)
do
  COMPOSITE_NAME=${COMPOSITE%.dhall}
  mkdir -p composite/$COMPOSITE_NAME
  dhall-to-yaml --file dhall/composite/$COMPOSITE >> composite/$COMPOSITE_NAME/action.yml
done
echo "Done!"

for REPOSITORY in $(ls dhall/workflows)
do
  REPOSITORY_NAME=${REPOSITORY%.dhall}
  echo -n "Generating workflows for $REPOSITORY_NAME..."
  mkdir -p workflows/$REPOSITORY_NAME
  for WORKFLOW in $(ls -p dhall/workflows/$REPOSITORY/ | grep -v /$)
  do
    dhall-to-yaml --file dhall/workflows/$REPOSITORY_NAME/$WORKFLOW >> workflows/$REPOSITORY_NAME/$WORKFLOW_NAME.yml
  done
  echo "Done!"
done

This will generate all our workflow files from the Dhall files. We then propagate these workflow files to their respective repositories using what else? GitHub Actions of course! That merits another post in the future.

How Dhall helped us

Dhall has provided us with flexibility and consistency in managing our GitHub Actions workflows.

Creating and updating workflows became less trivial because of importing and functions. Reusability of code means we are maintaining fewer lines of code.
It has helped us make fewer mistakes because of its type-checking. Something we’ve done by accident before in YAML is to delete a line and end up with completely broken YAML. In the example below, if you only delete the line with c: you accidentally put d: into a:. This is an easy mistake to make when a: and c: are both multiple pages long.

a:
  b: "b"
c:
  d: "d"

No more YAML formatting errors since the files are autogenerated. We commit YAML files alongside Dhall files, and mark them as generated with .gitattributes. This lets us fearlessly refactor because it will show no diffs in the YAML files.

If your GitHub Actions files are getting unwieldy, I suggest you start looking into Dhall to make your life easier!

Bellroy Technology Team: 2023 in Review

Mike Webb — Tue, 12 Mar 2024 00:00:00 UT

Bellroy Technology Team: 2023 in Review

Mike Webb 2024-03-12

Technology

Our journey through 2023 has been one of significant transformation and achievement. At the heart of our success is our commitment to Shape Up as our software development methodology, alongside a robust roadmap that outlines our long-term strategic goals and the immediate steps we can take towards each objective. Our roadmap provides us with a good litmus test when we’re shaping a project - either it moves toward our strategic objectives, or it signals a time to reassess those objectives. I’m thrilled to present an overview what our lil’ ol’ team of 13 was able to achieve last year.

Managing configuration as code has been best practice in our industry for many years, but I’m pleased to say that we’ve had great success introducing it to our non-technical configuration. For our team, this means allowing the rest of the company to directly manage the content and configuration they contribute to - we just provide the tools and processes that allow them to ship changes. We made significant strides in migrating more configuration datasets into our git repository, complemented by the development of internal tools designed to automate configuration deployment. This includes sophisticated tooling for unit-testing configuration changes. For example, we run golden tests against our discount calculation rules by running virtual carts through the rules and comparing the results. Our configuration contributors have developed a battery of tests for our configuration over time, allowing them to ship changes with confidence. Through improvements to our tooling - and migration of more configuration into our git repository - we’ve boosted company velocity in product releases and campaigns without having to provide technical oversight. The company as a whole publishes content and configuration changes into production 6-10 times a day.

Our dedication to providing world-class tooling support has redefined how content and front-end code is released and managed on our site. By streamlining our release processes and introducing user-friendly interfaces, we’ve empowered a broader spectrum of Bellrovians to contribute. For example, templated content - such as the job board on bellroy.com - is now managed using Markdown documents and FrontMatter, edited by our People & Culture team using Github Codespaces. Our front end developers did a huge amount of work in 2023 consolidating our front-end codebase and content to use a standardized component library. Now, anyone can ship great-looking content with much less need for a design review. 15% of Bellroy crew (outside the Technology Team) regularly make contributions to our website, which is a big step-up from 2022.

In 2023, Bellroy sent products to customers in 160 countries. Our goal to service our customers and partners around the globe demanded a system that could handle the complexities of global commerce, ensuring competitive shipping offerings and compliance with diverse tax requirements. We overhauled (and in some instances replaced) our legacy systems to handle these tasks, which also made it practical to integrate this functionality more tightly into more of our sales channels.

Our migration from Ruby on Rails monoliths to Haskell services, adhering to the Self Contained Systems (SCS) architecture, has enhanced the stability, maintainability, and cost-effectiveness of our internal systems. We aim to ensure that our technical stack mirrors our team makeup and that - while having a pager is a necessity in a 24/7/365 business - holding the pager is a trivial inconvenience shared by all. The reliability of our Haskell code has drastically reduced out-of-hours incidents, and our move from reserved EC2 instances to Lambda and DynamoDB has significantly cut operational costs. We’ve been able to reduce the size of the EC2 instances we use, and our monthly bills for each of our SCS services are a fraction of the cost of running an appropriately sized EC2 instance to perform the same function. On a related note, this migration has also encouraged us to evaluate and start using other AWS services - we put our first implementation of a Step Functions workflow into production in 2023. This AWS service has allowed us to better orchestrate our Lambda functions and has been a great addition to our toolset.

Our DevOps team has made great progress in enhancing our global continuous integration and deployment pipeline. By leveraging Github Actions, Nix, and Hydra, we’ve streamlined our development processes, particularly with our Haskell monorepo. The integration with Hydra has been crucial, ensuring efficient builds and deployments by rebuilding and deploying only what has changed since the last build/deploy. This approach has minimized CI/CD times, allowing our developers to focus on building great things instead of handholding changes into production. Adopting Hydra has not been without challenges - managing builds for multiple architectures and developing expertise has been difficult - but the benefits have been worth the effort.

In preparing this post and looking at our next slate of projects, it feels like we’re on a really good path. Our strategic directives still feel right and allow our developers to independently make principled design decisions. This is one of the benefits we were hoping to get from Shape Up, and I’m pleased to report that it continues to deliver. If I could wave a magic wand I’d convert all of our Ruby code to Haskell code tomorrow, but other than that I’m extremely happy with where we are as a team and as a company with technical capability as one of its core strengths. I look forward to sharing what we got up to in 2024.

Using Shape Up - What Works, What We've Changed, What's Next

Chris D'Aloisio — Fri, 09 Feb 2024 00:00:00 UT

Using Shape Up - What Works, What We've Changed, What's Next

Chris D'Aloisio 2024-02-09

Technology

Introduction

Project management isn’t just about keeping tasks on track to achieve a predefined outcome; it’s about delivering value to people. That’s where the Shape Up methodology, pioneered by Basecamp, stands out. It’s not your typical project management style – it breaks away from the conventional Agile frameworks such as Scrum, offering a fresh, focused way to steer projects.

Shape Up consists of six-week, time-boxed cycles, allowing teams to dive deep into their work without the distractions of ongoing sprints and endless backlogs. This approach brings a clarity of purpose to each cycle, making it easier for teams to produce impactful work. It’s a game-changer for those who want to see real progress without getting bogged down in ceremonies and task switching.

In this post, we’re going to explore how Shape Up has influenced the Tech Team’s ability to deliver value. We’ll give you a sneak peek into how we plan to evolve this approach further, aiming for better project outcomes in the future.

Understanding the Shape Up Method

Shape Up isn’t just a methodology; it’s a shift in mindset. At its heart, Shape Up champions projects of fixed time and flexible scope. Projects are tackled in six-week cycles, providing a clear endpoint and fostering a sense of urgency.

Unlike traditional project management that often stretches timelines to fit the scope, Shape Up flips the script – the scope is adjusted to fit the timeline. This ensures projects don’t overextend and results are delivered consistently.

The methodology also introduces ‘cool-down’ periods following each cycle. These two-week spans allow teams to recharge, reflect, and prepare for the upcoming cycle on their own time. This break from the grind is crucial, preventing burnout and keeping project approaches and solutions fresh.

Deep work periods are another cornerstone. By reducing meetings and daily standups common in methodologies like Scrum, Shape Up allows teams to immerse deeply in their tasks. This undisturbed focus leads to higher quality outputs and often, breakthrough innovations. It’s all about giving teams the room they need to think, create, and solve problems without the constant distraction of administrative overhead.

In Shape Up, autonomy is key. Teams are trusted to figure out the best way to tackle their projects. This empowerment not only boosts morale but also leads to more inventive and tailored solutions.

Customizing Shape Up to Fit Our Needs

Embracing Shape Up doesn’t mean rigidly sticking to its every rule. It’s about making the methodology work for you. At Bellroy, we’ve tailored Shape Up to better align with our dynamic needs, proving its adaptability and effectiveness.

One significant change we made is the introduction of a separate “roaming” team. Operating within the six-week cycles, this team has an allocated programme of work, but is deliberately left with capacity to deal with high-value, time-sensitive tasks that come up at short notice. This flexibility allows us to respond swiftly to urgent needs without disrupting the flow of ongoing projects.

This customization has significantly improved our responsiveness to other teams’ demands and schedules. The Tech Team, for instance, isn’t confined to a single product at Bellroy. We provide core services across the business, often in response to objectives set by various teams throughout the year. Other teams within Bellroy, such as our Logistics team, have to be able to respond to changing circumstances like closed shipping lanes with short or no notice and sometimes require our support to do so. The “roaming” team model enables us to address these needs promptly while maintaining our commitment to the Shape Up cycles for larger projects.

Additionally, we’ve incorporated more robust feedback loops within each cycle. Regular check-ins, albeit less frequent than in traditional methodologies, ensure that everyone stays aligned on goals and progress.

During check-ins, we have explicit cues to ensure that we are not focusing solely on technical objectives. We ask ourselves if we are on track to delivering the value that stakeholders are expecting, and that we do not leave things in an unfinished state at the end of the project.

These adaptations have allowed us to maintain the essence of Shape Up – focused work periods, autonomy, and regular shipping – while tweaking it to suit our unique workflow and company culture.

Impact on Productivity and Project Outcomes

The introduction of Shape Up in the Tech Team wasn’t just a procedural change; it brought a fundamental shift in how we approach productivity and overall project value. The flexibility in scope within fixed timelines has resulted in a more dynamic and responsive work environment, allowing us to pivot mid-project if required to deliver value where it matters most.

Perhaps the clearest impact has been on project completion rates. The six-week cycles, with their clear boundaries and goals, have led to a higher rate of on-time project deliveries. This improvement in delivery has cascaded across the organisation, improving our overall operational efficiency.

The cool-down periods have also played a vital role in maintaining a sustainable work pace. These breaks have become a time for reflection and learning, contributing to continuous improvement in our processes and approaches.

In essence, Shape Up has not just reshaped our project timelines; it has reinvigorated our teams and redefined our benchmarks for success.

Future Plans and Improvements

As we look ahead, the journey with Shape Up is far from over. Our plan is to further refine our approach, especially in capacity planning across the business.

We believe that a deeper understanding and application of Shape Up’s principle of fixed time, variable scope can significantly enhance cross-team project planning and resource management.

We’re also exploring ways to integrate more collaborative tools and techniques within the Shape Up framework. Our goal is to enhance communication and alignment among teams, ensuring that every cycle is as effective as possible.

Our experience with Shape Up has been transformative. It has not only improved how we manage projects but also how we view productivity and team dynamics. As we continue to evolve and adapt this methodology, we remain committed to pushing the boundaries of what’s possible in project management for our team and the rest of the business.

The History of Nix at Bellroy

Jack Kelly — Wed, 24 Jan 2024 12:00:00 UT

The History of Nix at Bellroy

Jack Kelly 2024-01-24

Technology

Bellroy relies heavily on Nix as an important part of our developer tooling. It provides us with reproducible environments for developer shells and CI runs, as well as a build environment for our statically linked Haskell code. Our tech team works in a moderately conservative Haskell dialect, so this level of Nix dependence might seem surprising and incongruent. In this post, I will explain how and why our Nix usage evolved in the way that it did, and point out useful tricks and tools at each stage of adoption.

Phase 1: Developer Shells via `shell.nix`

Nix has an intimidating learning curve, but most of this comes from writing Nix expressions. Developers can be easily taught to use Nix-based infrastructure once it’s been set up. Our first use of Nix was writing shell.nix files for use with nix-shell. Nix uses these files to create reproducible development environments containing correct versions of tools like ruby, ghc, etc., depending on the project. This is a great way to get started because it doesn’t ask developers to radically change their workflows, and allows them to trial Nix at their own pace. There are some subtleties to be aware of when setting up these shell expressions.

For true reproducibility, you need to store a reference to the version of nixpkgs used in each project’s source control. This is called “pinning nixpkgs”. We initially did this using the niv tool, and later by using Nix flakes.
As we have several developers using macOS, we pinned nixpkgs commits from nixpkgs-*-darwin branches. We found that this improved the cache hit rate for our macOS-using developers, and reduced the amount of software they had to build locally.
For Ruby and npm projects, we found it too difficult to capture all of their dependencies as Nix expressions. Packages in private repositories and on private package registries were the biggest challenge here, as many foo2nix tools only support public package repositories. As a workaround, our shells provide the language runtime (e.g., ruby) and its packaging tool (e.g., bundler), but leave fetching language-level dependencies to that language’s tool. We have found this to be a reasonable trade-off between correctness and practicality. The access-tokens setting in modern versions of Nix might help us, if we revisit this.

Phase 2: Building Haskell Deployment Packages using `haskell.nix`

We were comfortable just using Nix for developer shells for a fairly long time, until a confluence of several constraints forced us into a more elaborate Nix setup.

Most of our Haskell code is deployed to AWS Lambda. To build binaries for this environment, we originally used the lambci/lambda Docker container to build in an environment close to what AWS provides at runtime. This ceased to be viable once we started using Apache Kafka: the Haskell client we use (hw-kafka-client) binds to librdkafka, which is not provided by the AWS runtime environment. Instead of wrangling third-party RPM repositories or Lambda Layers, we used the excellent haskell.nix framework to build statically linked, UPX-compressed deployment packages. We published example Nix code code which does this, as part of our wai-handler-hal project.

Phase 3: Private Binary Cache using Amazon S3 and GitHub Actions

Nix + haskell.nix was a reliable way to generate deployment packages for our Haskell services, but even after adding IOG’s binary cache we would often have a lot of cache misses, leading to very long build times (particularly on macOS). It was time to bite the bullet and set up our own private cache. Nix links against the AWS SDK for C++ and can use S3-compatible object stores as binary caches, so an S3 Bucket was an obvious place to store our derivations. We needed a way to populate the cache, and weren’t ready to tackle Nix-native solutions like Hydra, so we built out a caching workflow using GitHub Actions’ hosted Linux and macOS runners. Behind this simple idea are a lot of details worth getting right, so we’ve tried to capture as many of them here as we possibly can.

Setting up the Bucket

The bucket is just a normal S3 bucket. Because Nix uses the S3 API, we can block all public access and leave website hosting turned off.
It might be worth creating the bucket in the region closest to most of your developers.
It is generally the case that many derivations stop being relevant shortly after they’ve been built.
- It might be worth considering an S3 Lifecycle Configuration to migrate old derivations from the “Standard” Storage Class to “Standard — Infrequent Access” and possibly even “Glacier Instant Retrieval”. Be careful of increased retrieval charges when using these storage classes.
- S3 Intelligent Tiering might also be worth considering. Be careful of its automation charges.
- It is possible to use a Lifecycle Configuration to delete very old derivations, but this can confuse the cache of Nix clients. It might also confuse Hydra (which keeps records of which derivations it has built).
The Nix manual provides example AWS Identity and Access Management (AWS IAM) Policy Documents for read-only and read/write access to an S3 Bucket. Actually providing credentials to Nix that have these permissions can be tricky, due to constraints imposed by Nix:
- We cannot use regular credentials to assume a more restricted role, because the C++ SDK that Nix uses does not support assume_role entries in ~/.aws/config.
- The Nix daemon runs as the root user, so we need to configure credentials in root’s home directory, and cannot use interactive ways of providing credentials.
- In the AWS cloud, Nix should be able to access credentials in the normal way (e.g., EC2 Instance Profiles).
- Nix on non-cloud machines (e.g., developer laptops) is more difficult. We are basically forced into using long-lived aws_access_key_id and aws_secret_access_key pairs. This is not best practice, so we don’t want these keypairs to be able to do too much. We recommend creating entirely separate IAM Users that can only access the cache bucket, and creating a separate User for each developer or server that needs access. Automating key rotation or setting up the access-keys-rotated rule in AWS Config can help ensure that keys are rotated regularly.
- The GitHub Actions Workflow that populates the cache will assume an AWS IAM Role with permissions to read and write the cache bucket. We don’t create an IAM User for the workflow, because GitHub Actions supports OpenID Connect and provides a guide for configuring OpenID Connect between GitHub and AWS.

Setting up Keys

Nix uses public/private key pairs to know which derivations to trust: our builder will sign derivations with the private key before uploading them to S3, and clients will know to trust the corresponding public key.

We generated a cache key pair following the recommendation in the Nix manual:

$ nix-store --generate-binary-cache-key example-nix-cache-1 key.private key.public

The private key was stored as as a GitHub Actions Secret.

The public key was set in the nixConfig setting of our flakes, which means that it applies to only our repositories. This speeds up cache checking for other builds, as Nix clients will only check our bucket when it makes sense:

{
  description = "A flake";
  inputs = ...;
  outputs = ...;

  nixConfig = {
    extra-substituters = [
      "s3://example-nix-cache?profile=bellroy"
      "https://cache.iog.io"
    ];
    extra-trusted-public-keys = [
      "example-nix-cache-1:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA="
      "hydra.iohk.io:f/Ea+s+dFdN+3Y/G+FDgSq+a5NEWhJGzdjvKNGv0/EQ="
    ];
  };
}

Setting up Clients

For multi-user Nix installations (the default), these AWS keys need to be loaded by the user running the Nix daemon (by default, this is root). These can be set by running commands like:

sudo -H aws configure --profile bellroy set aws_access_key_id AKYOURACCESSKEY
sudo -H aws configure --profile bellroy set aws_secret_access_key YOURSECRETKEY

macOS updates tend to remove files in ~root, including AWS config files. One way to permanently provide credentials to the Nix daemon is (thanks @lrworth):

Create AWS config and credential files in /etc/nix/aws/config and /etc/nix/aws/credentials.
Edit /Library/LaunchDaemons/org.nixos.nix-daemon.plist, adding the following lines under EnvironmentVariables:

<key>AWS_CONFIG_FILEkey>
<string>/etc/nix/aws/configstring>
<key>AWS_SHARED_CREDENTIALS_FILEkey>
<string>/etc/nix/aws/credentialsstring>

Run sudo -i sh -c 'launchctl remove org.nixos.nix-daemon && launchctl load /Library/LaunchDaemons/org.nixos.nix-daemon.plist' to restart the Nix daemon.

Setting up the Workflow

Here is a YAML description of a sample workflow, derived from the workflow that we previously used to update our cache:

name: Populate nix shell cache
on:
  schedule:
    - cron: "0 0 * * 0"
  workflow_dispatch: {}
jobs:
  populate-cache:
    strategy:
      fail-fast: false
      matrix:
        os:
          - ubuntu-latest
          - macos-latest
    runs-on: "${{ matrix.os }}"
    steps:
      - uses: "actions/checkout@93ea575cb5d8a053eaa0ac8fa3b40d7e05a33cc8"
      - uses: "aws-actions/configure-aws-credentials"
        with:
          aws-region: "${{ env.AWS_REGION }}"
          role-to-assume: "${{ secrets.AWS_OIDC_ROLE_ARN }}"
      - uses: "cachix/install-nix-action@daddc62a2e67d1decb56e028c9fa68344b9b7c2a"
        with:
          extra_nix_config: |
            post-build-hook = /etc/nix/upload-to-cache.sh
            substituters = https://cache.nixos.org/ https://cache.iog.io s3://example-nix-cache
            trusted-public-keys = cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY= hydra.iohk.io:f/Ea+s+dFdN+3Y/G+FDgSq+a5NEWhJGzdjvKNGv0/EQ= example-nix-cache-1:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=
          install_url: https://releases.nixos.org/nix/nix-2.7.0/install
          nix_path: nixpkgs=channel:nixpkgs-22.11-darwin
      - name: Set up nix signing key
        run: "echo ${{ secrets.NIX_CACHE_NIX_SIGNING_KEY }} | sudo tee /etc/nix/example-nix-cache.private > /dev/null"
      - name: Set up post-build hook
        run: |
          sudo tee /etc/nix/upload-to-cache.sh < /dev/null
          #!/bin/sh

          set -eu
          set -f # disable globbing
          export IFS=' '

          echo "Uploading paths" \$OUT_PATHS
          exec $(which nix) copy --to 's3://example-nix-cache?region=wherever&secret-key=/etc/nix/example-nix-cache.private&compression=zstd¶llel-compression=true' \$OUT_PATHS
          EOF
          sudo chmod u+x /etc/nix/upload-to-cache.sh
      - name: Restart nix-daemon
        run: |
          case $RUNNER_OS in
            Linux) sudo systemctl restart nix-daemon.service ;;
            macOS) sudo launchctl kickstart -k system/org.nixos.nix-daemon ;;
          esac
      - name: Install nix-build-uncached
        run: |
          nix-env -iE '_: import (builtins.fetchTarball {
            url = "https://github.com/Mic92/nix-build-uncached/archive/77fe5c8c4c5c7a1fa3f9baa042474b98f2456652.tar.gz";
            sha256 = "sha256:04hqiw3rhz01qqyz2x1q14aml1ifk3m97pldf4v5vhd5hg73k1zn";
          }) {}'
      - name: Build shells
        run: |
          nix-build-uncached -build-flags '-L --keep-going' -E '(import ./.).devShells.${builtins.currentSystem}'

As with the rest of this process, the basic idea is simple (run the workflow to build all derivations required by our development shells), but the devil is in the details:

A Nix post-build hook to sign and upload any derivations we build.
We used nix-build-uncached (now deprecated) to build only the derivations that we could not find in S3, preventing lots of redundant downloads. nix-build-uncached does not support flakes, so we invoked the build through a default.nix which uses flake-compat.

The deprecation notice in nix-build-uncached’s README.md suggests more modern alternatives:
- nix-fast-build has a --skip-cached flag. A comment on Nix issue #3946 says that nix-eval-jobs (which powers nix-fast-build) can be problematic when lots of import-from-derivation (IFD) is required, as in haskell.nix;
- The comments on Nix issue #3946 suggest that nix build --store $remote_store --builders auto might (eventually?) work.
If we were doing this again, we’d probably consider Determinate Systems’ Magic Nix Cache to evaluate Nix expressions more quickly, before the build begins.
We ran the workflow weekly as a trade-off between cache freshness and billable minutes, and enabled manual workflow dispatch for when we upgraded GHC versions or major packages.
We use Zstandard compression when we upload to S3, because it’s very light on CPU time and we found that XZ was very slow on large derivations.

Conclusion

You don’t have to adopt Nix all at once to get good value out of it. Simple development shells prevent a lot of headaches and tend to have good cache hit rates, which means that it’s fine to delay private caching until much later. Our initial shell.nix served us well for over a year before we started adding more sophisticated tooling, and we only did that because we were forced. Our moves to haskell.nix and GitHub-Actions-based caching were made in response to genuine needs, and we learned as we went. We did eventually move to a Hydra-based CI system, but that’s a story for another time.

Embracing automation with GitHub Actions

Alvin Tang — Thu, 30 Nov 2023 00:00:00 UT

Embracing automation with GitHub Actions

Alvin Tang 2023-11-30

Technology

Actions is a feature of GitHub for automating software workflows. It includes Continuous Integration and Delivery (CI/CD) so it allows us to build, test, and deploy our code. In this post, I’ll discuss some of the things we love about it and what we think it can improve on.

Things we love about Actions

Easy integration with GitHub

One major reason for using Actions is its built on top of GitHub. If a repository is already in GitHub, using Actions is as simple as pushing a workflow file in the correct directory. Workflow files are YAML files defining the steps to be done when an event is triggered. There is no need to run a separate application and/or server since GitHub provides it already.

In our organization, we have non-technical users working on content and configuration files. We moved these files into a git repository so they are version controlled. We taught users git basics including pushing commits and creating a pull request for their changes. With this move, we were also able to add workflows to make everyone’s lives easier. The workflows do several things like running tests, generating files, and verifying protected content; several workflows build updated assets and deploy them to the correct environment. The workflows also provide feedback by generating PR comments and Slack messages. All of these workflows run automatically or can be manually triggered. Over time, our users have come to understand what happens in these workflows and how to respond to different types of failures. And they are able to do this all within the familiar surrounds of Github.

Event triggers

CI/CD workflows are usually triggered by events. Actions provide a wide array of event triggers. The push and pull request events are the most common; you can use these to trigger builds and run tests on a repository.

We’ve found uses for other event triggers such as:

building a package for a draft release, triggered by the release event
running a workflow from an external script, triggered by the repository_dispatch event
merging a pull request upon approval, triggered by the pull_request_review event

Marketplace

There are workflow actions that are common to many repositories. These include actions like checking out a repository, building a Docker image, or setting up tools like node. Actions has a marketplace containing actions that other developers have published. GitHub also provides its own actions such as checkout, upload-artifact, or download-artifact. Aside from GitHub, a lot of third-party providers have published actions to support their products. If we want to integrate actions with other tools like Docker or Slack, we try searching the marketplace first!

Here are some of the third-party actions we use:

github/checkout - This action checks out a repository so we can work with its contents. This allows you to checkout multiple repositories. For example, we have a workflow that works with two other repositories. We can check out each of those repositories in their own directories so we can work with the files inside them.
docker/build-push-action - We use Docker containers for building GitHub codespaces. This Docker action builds a Docker image for our codespaces and pushes it to a registry.
slackapi/slack-github-action - We post workflow updates in Slack. This action allows us to send messages to Slack. It has formatting and even support for threaded messages!
cachix/install-nix-action - We use Nix in most of our repositories. This action installs Nix in GitHub-hosted runners so we can run code in our defined Nix environment.
aws-actions/configure-aws-credentials - This action configures a runner to access AWS. We use this to generate OIDC tokens so workflows can access AWS resources during build, test, and deployment.

Composite actions and reusable workflows

A few months into our Actions journey, workflows were getting repetitive. We started exploring how to DRY them up. Composite actions and reusable workflows address this problem. These allow us to group repetitive steps or jobs and call them in one step, instead of writing the steps all over again.

A sample composite action we have is called slack-update. This is a wrapper around slackapi/slack-github-action. We have predefined messages for certain events such as deployment success or failure messages. Using a composite action for this keeps our messaging consistent across different repositories. Here is a code snippet of this composite action:

description: Pinned slackapi/slack-github-action. Posts a message to the specified slack channel.
using: composite
name: slack update
inputs:
  authors:
    description: authors of the change
    required: true
  branch:
    description: branch name to put in messages
    required: true
  channel-id:
    description: slack channel ID where message will be posted
    required: true
  message-type:
    description: "type of message to be posted. Options are: `build-start`, `build-fail`, `build-success`, `open-pr`"
    required: true
  pr-url:
    default: ''
  ...
runs:
  steps:
    - if: "contains( inputs.message-type, 'open-pr' ) && inputs.branch == 'master'"
      name: Slack Notifications - Create PR payload
      run: |
        echo "PAYLOAD={\"text\":\":*${{ input.authors }}* has opened a PR: ${{ inputs.pr-url }}\"}" >> $GITHUB_ENV
      shell: bash
    - env:
        SLACK_BOT_TOKEN: "${{ inputs.slack-bot-token }}"
      name: Slack - Notifications - Post to slack
      uses: "slackapi/slack-github-action@e28cf165c92ffef168d23c5c9000cffc8a25e117" # see "Managing workflows" for more on why we use a SHA here
      with:
        channel-id: "${{ inputs.channel-id }}"
        payload: "${{ env.PAYLOAD }}"

And then we call the composite action as a step in workflows:

steps:
...
- if: "github.ref == 'refs/heads/master'"
  name: Post pull request link to Slack
  uses: "bellroy/workflows/composite/slack-update@master"
  with:
    branch: "${{ env.BRANCH }}"
    channel-id: "${{ env.SLACK_CHANNEL_ID }}"
    message-type: open-pr
    pr-url: "${{ env.PR_URL }}"
    slack-bot-token: "${{ secrets.SLACK_TOKEN }}"

Other examples of our composite actions:

get-tool - We build internal tools that are released through GitHub. These tools are used within our workflows so they have to be retrieved for the workflow to run successfully. We wrote an action to get the latest release of a tool, or we can retrieve a specific version.
install-nix - This is a wrapper around cachix/install-nix-action to customize Nix installations. We have predefined setups for different access levels to our private Nix cache.

Self-hosted runners

Self-hosted runners can be set up to run actions in our own server. We do this when workflows need access to protected resources. Our self-hosted runners run within our private network. Aside from that, we also customize the runner by adding the tools it needs to do its job. For example, one of our workflows need to produce messages to a Kafka topic in AWS Managed Streaming for Apache Kafka (MSK). We have provisioned a self-hosted runner in the same network as the MSK cluster and installed (kcat)[https://github.com/edenhill/kcat] as the Kafka client.

If all else fails, use bash scripts

The ability to use bash scripts in the workflows provides endless possibilities. We have scripts for output formatting, file manipulations, running tests, cleaning up after builds, etc. Combining this with a controlled environment (such as a self-hosted runner), we can do most of the automation we need.

Things we don’t love so much about Actions

Developing and testing workflows

The process of developing and testing actions is not the best. Some event triggers need the workflow change to be on the default branch to be executed. There are ways to work around this such as adding an event that can trigger even if it’s not on the default branch. But this still has limitations if we want to test the actual event, instead of the steps.

Workflows can also take a long time to run. A minor mistake such as a typographical error can cost us minutes before we get feedback from actions. This gets compounded if testing in self-hosted runners since the actions are queued.

Managing workflows

As the number of repositories grows, so does the number of workflows. Although there are mechanisms to DRY them up, it’s better to manage these workflows in one place. One benefit of this approach is that it makes managing our third-party actions much easier. As best practice, we pin third-party actions to commit SHAs (which can’t change), rather than release tags (which can), so we don’t get surprised by updates that can break our workflows or get exposed to supply chain attacks. This means we have to update these SHAs every so often. This becomes painful when there are a lot of repositories involved.

Overall, GitHub Actions is a great tool for automation and CI/CD. If you are already using GitHub, it provides easy integration and a wide variety of actions to use. It is also versatile enough to support custom actions specific to your needs.

Next time, I will discuss how we use Dhall to manage our workflow files for Github Actions.

Celebrating the release of Amazonka 2.0

Mike Webb — Tue, 12 Sep 2023 00:00:00 UT

Celebrating the release of Amazonka 2.0

Mike Webb 2023-09-12

Technology

For businesses which directly depend upon open source software, investing developer time and expertise can yield more tangible and enduring benefits than monetary sponsorship alone, fundamentally strengthening the technological infrastructure upon which your business’ success depends. Today, we at Bellroy are pleased to discuss our role in the the 2.0 release of Amazonka — a vital Haskell package providing bindings for the AWS (Amazon Web Services) infrastructure. This package stands as the primary tool in the Haskell ecosystem addressing these needs, and it is extremely satisfying to see a new major release on Hackage.

The Amazonka 2.0 release included a significant contribution from our dedicated staff engineer, Jack Kelly. Jack devoted considerable time and effort to the release, during and beyond his regular working hours, to ensure its successful delivery. His commitment is testament to the passion we share for open-source development and our desire to contribute back to this vibrant community.

We rely upon Amazonka in many of our production systems. Upon noticing that the project could greatly benefit from having a dedicated full-time developer, we supported Jack by allocating more than six weeks of full-time work to the project. Now that it’s released, we eagerly look forward to seeing the ripple effect this release (and our other contributions) will have in the Haskell community. It is our hope that Haskell ecosystem contributions such as these will inspire more developers to look favourably at Haskell when considering technologies for their next project. We also hope it encourages more companies to invest in open-source contributions and support the ecosystems in which they operate.

At Bellroy, we have always held a deep respect for open-source software. It significantly benefits our own endeavors, and underpins a significant portion of today’s digital infrastructure. Our journey from a proprietary e-commerce platform to a completely bespoke one would be unimaginable if not for the availability of open-source frameworks like Magento and Rails. We view contributing to the open-source community not as a nice-to-have, but as an obligation in keeping with our aim of using business as a force for good.

When we create technology that we believe could be useful to a wider audience — and it isn’t a source of competitive advantage — we generally consider open-sourcing it. Prior to adopting Haskell as our primary development language, we open-sourced multiple Ruby and Elm packages, such as:

dry-monads-sorbet Sorbet types for the dry-monads Ruby gem.
elm-email Parse email addresses safely in Elm.
elm-embed-youtube A simple way of embedding a Youtube player using an iframe, in Elm.
elm-imgix A wrapper around Imgix, in Elm.
elm-infinite-gallery A simple gallery that supports infinite scrolling, in Elm.
rspec-sorbet* A Ruby gem which makes it easier to use Sorbet & RSpec together.
sorbet-struct-comparable* Add equality testing to classes which inherit from Sorbet’s T::Struct class.

* A former Bellroy employee is now the primary maintainer of these packages.

Amazonka joins a growing list of open-source Haskell (and other FP) libraries we’ve contributed to, including:

hal A runtime environment for Haskell applications running on AWS Lambda.
haskell-cached-io Cache a single IO action (we’ve taken over maintainership of the original repository)
github-actions-dhall Typecheck, template and modularize your Github Action definitions with Dhall.

The following libraries are examples of where we saw a package-sized gap in the ecosystem, and decided to fill it ourselves.

aws-arn Type and optics for manipulating Amazon Resource Names (ARNs)
timeline A library for handling data that changes over time
wai-handler-hal Lets you run a WAI application as the backend Lambda of an AWS API Gateway REST API

We generally try to contribute upstream features and fixes first, even if doing so requires a little more initial digging. This approach is better for everyone in the long run: it reduces our maintenance burden, helps other people with their problems, and shores up the long-term viability of the Haskell ecosystem.

Bellroy’s mission is to become global leaders in carry, and we work towards this by “using business as a force for good” and “helping the world, and our crew, flourish”. This successful release reaffirms our “upstream-first” model of open-source contribution, and to release additional open-source packages — watch this space.

We’d like to say a big “thank you” to Brendan Hay for creating Amazonka, and everyone who has contributed to the above-listed projects over the years.

Humans rarely need to make business decisions on real-time data

Chad Musick — Tue, 15 Aug 2023 00:00:00 UT

Humans rarely need to make business decisions on real-time data

Chad Musick 2023-08-15

Data and Analytics

All other things being equal, fresher data are better data. However, it is almost never the case that all other things are equal.

Many data warehouses charge differently and more for streaming updates than for batch updates (see BigQuery ingestion pricing as an example)
Updates to some data may necessitate updates to other data, leading to a cascade of updates
Fresh data may be unreliable in isolation (see “how to handle piecemeal data” for handling issues such as referential integrity)
Initial trends may be misleading, such as in real-time web analytics

Every decision made at every company has some agent making that decision. Perhaps it is an artifical intelligence making it, or perhaps a deterministic business process, or sometimes, a human. Fundamentally, every decision is made at some level by a human, even if they decide to delegate it to a computer system. See Dr. Marshall’s book Data Conscience about the effects such delegation can have and how to consider them in better ways.

When deciding how fresh your data need to be, the key question to ask is this: “How quickly would new data change the actions of the decision-making agent?” This clarifies things immediately. As examples:

A fraud-detection system should prevent fraudulent orders, but in order to do so it must act fast enough to prevent the order from completing
Website personalisation should happen quickly enough to render the website
Stock-level updates should happen quickly enough to prevent selling stock that can’t be fulfilled

Strategic decisions – the kind that often rely on reports, dashboards, and other visualisations made by analysts using Business Intelligence software – are typically not made in seconds. Could new data arrive in the next five minutes that would change such decisions? Obviously, just as much as noting that we must get halfway to a destination before we can get to that destination. The chance of such information arriving is small, and waiting “just in case” for perfect certainty will result in an infinite wait. However, we are not flummoxed by Zeno’s paradoxes here, though they are some of the oldest recorded objections to real-time analytics.

Strategic decisions, then, can be made as effectively from old but representative data as from fresh but representative data. The important aspect of the data is that they are representative, not that they are arbitrarily new. Website conversion rates from two weeks ago may still be representative if nothing material has changed, and sales data from ten minutes ago may be unrepresentative if an item has gone out of stock. See our previous post “Keeping on-call calm” for applying this principle to operational issues.

Since all other things are rarely equal, decide the speed of data updates according to how those data will be used. Then, communicate the expected latencies in a place accessible to everyone who will rely on the data. Set alarms for critical conditions. Your decisions will be just as good, but your data/compute bills will be lower.

Technology stack transitions are hard

Mike Webb — Sat, 15 Jul 2023 00:00:00 UT

Technology stack transitions are hard

Mike Webb 2023-07-15

Technology

In today’s rapidly evolving technological landscape, businesses often face the challenge of transitioning between different technology stacks. In 2019, we made the difficult decision to move away from using Ruby and switch to Haskell as our primary stack. We have found this process to be complex and demanding, particularly when dealing with legacy systems. Retaining skilled developers who can maintain knowledge of both old and new stacks has been crucial to our transition. We’ve focused on making incremental progress, delivering tangible value at each stage. In this blog post, we will delve into the intricacies of managing the transition between technology stacks, addressing the importance of retaining talent, maximizing value, and effectively handling “leftover” code.

In a transition like this your legacy stack will inevitably become viewed as “lesser” and, once this happens, it is difficult to retain developers capable of maintaining it. Once we had made the decision to move to Haskell, we lost several developers who were committed to Ruby and did not see a transition to Haskell as their preferred career path. We believed that preserving knowledge of the legacy codebase was vital for a smooth transition. We identified developers that were willing to learn to develop in the new stack and ensured they were well-supported in their learning journey. We ran introductory workshops, book clubs and mentoring sessions, and assigned developers to projects building basic features in the new stack. This was an effective strategy for mitigating the risks associated with the transition and ensured continuity.

In a business like ours where software is not the “product” but the business is critically reliant upon it, we need to transition in safe increments and produce tangible value with each increment. In some instances the act of transitioning alone delivers sufficient value because the new stack is a better fit. With Haskell, we have been able to demonstrate a significant reduction in technical problems and outages as we migrate parts of our systems. In other instances we need to wait until a change is required in the legacy system and port that functionality in the process of delivering the change. We try to find ways to add value during this process. For example, we have many systems that use Domain Specific Language (DSL)-based configuration rules and Haskell is fantastic for DSLs. We’ve been able to re-use a common DSL for configuration across these systems, and by doing so, kept our internal-user interfaces consistent and the DSLs more powerful with each new release. Adding the ability to configure a system using this common DSL has been a great selling point for transitioning legacy modules.

A natural consequence of an incremental transition strategy is that it will take a long time before our transition is completed. Bellroy is roughly 3 years into this transition and, looking at the trend in our Ruby line count, we’re probably about 2 years away from completing it. For all of that time, we need to keep our dependencies up to date, fix the odd bug in the legacy code, and support the growth of the business. Given the way things have proceeded so far, we have found that Haskell offers significant improvements in reusability, stability and maintainability. The cumulative benefits of those improvements will mean that we can successfully complete the transition before the legacy codebase becomes too much of a burden.

Even with careful planning and execution, as we approach the end of the transition we’ll have a substantial amount of “leftover” legacy code. This is code that is critical to ongoing operations, but is difficult to improve or add value to during the transition to the new stack. Migrating this code may not directly contribute to Bellroy’s business objectives, making it challenging to allocate resources for the migration. However, leaving the code in its current state can have negative implications for future maintenance and scalability. Quantifying these intangible costs can help with making the case to allocate resources to migrate that code. At Bellroy, we have built a project evaluation engine (in Haskell, naturally) that lets us estimate the probabilistic Net Present Value (NPV) of a proposed project using textual descriptions of future cash flows. We use this engine to model the opportunity cost of maintaining legacy code (as opposed to building new features) and the risk of retaining legacy systems with a dwindling amount of specialised expertise in the team.

Now if you’ll excuse me - I need to go and think about which part of our systems we’ll supercharge next…

Getting Things Done, with Asana

Mike Webb — Mon, 01 May 2023 00:00:00 UT

Getting Things Done, with Asana

Mike Webb 2023-05-01

Technology

In May of 2023 I wrote a series of posts in our internal company newsletter, “The Owlet”, introducing the “Getting Things Done” (GTD) system and how it can be applied in our task management tool, Asana. The feedback I got from these posts was positive and broadly applicable across the many functional teams within Bellroy. I figured if it had such an effect on such a diverse bunch of people, it was likely worth sharing with the world. Enjoy!

Introducing “Getting Things Done”

GTD is a time management and productivity system created by David Allen. The system’s core idea is to capture all of the tasks that clutter our minds and to organise them in a structured way that allows for efficient and stress-free execution.

The system consists of five basic steps:

Capture: capture all of the tasks that come to mind. This can be done by writing them down in a notebook or using a digital tool such as a to-do list app.
Clarify: Once everything has been captured, clarify what each item actually means and whether it requires action. This involves determining whether an item is actionable, and if so, what specific action is needed.
Organise: Once all items have been clarified, they need to be organised into categories such as “Next”, “Blocked” or “Someday”. This step is important for ensuring that tasks are categorised/prioritised appropriately and that nothing falls through the cracks.
Reflect: regularly review all of the tasks that have been captured and organised, making any necessary updates or adjustments. Reflect on how your process is working and update that, too.
Engage: The final step is to actually engage the process, and do the work!

I’ve been using the GTD system for several years now in both my personal and professional life. I’ve found it to be extremely effective. It’s helped me to stay organised and stay focused on the tasks that are most important. For my personal task list I use Nirvana, a tool purpose-built for GTD. At work, I use Asana. While Asana is not a pure GTD tool, you can easily implement a GTD system within it.

In the following article, I’ll be sharing some tips on how to implement the GTD system using Asana.

Step 1 - the “Capture” step

The first step of the GTD productivity system is to “Capture” all of your tasks in a trusted system. Asana is a powerful tool that can help you do just that.

For this system to work for you, you need to have confidence that all of your tasks are captured in Asana. You need to go through all of your current task lists (including your mental ones) and add them to Asana. You may require an hour or two to do this, so set aside an appropriate amount of time. It’s worth it!

The “My Tasks” project will be the place where you can quickly add any new tasks that come to mind. You can add new tasks to your “My Tasks” project by clicking the + button in the top bar on any Asana page and selecting “Add task.” This will open a popup window, where you can describe the task and optionally assign it to a project.

You can also use the Asana mobile app to capture tasks and ideas on-the-go. Whether you’re waiting in line at the grocery store or sitting on the bus, you can quickly add a task to your “My Tasks”.

The Asana Slack integration allows you to turn Slack messages straight into Asana tasks, with a link back to your Slack conversation. This is useful when you want to revisit the conversation without being distracted from your current focus.

Asana’s Chrome browser extension is another powerful tool that can help you capture tasks and ideas quickly. With the extension installed, you can add tasks to your “My Tasks” directly from your browser. For example, if you come across an interesting article that you want to read later, you can quickly add it to your “My Tasks”.

Finally, it’s important to review your “My Tasks” regularly. I’ll talk more about best practices for this later. For now, set aside time each day to review your “Inbox” and your “My Tasks” board. I recommend doing this at the start of each day - I usually set aside the first 15 minutes of the day to review these and plan my day. If you know you’re going to spend this time each day, you start to get less distracted by things appearing in “Recently Assigned” and “Inbox” during the workday.

Step 2 - the “Clarify” step

The second step of the GTD productivity system is the “Clarify” step. In this step, you review all of the tasks you “Captured” in the previous step and determine whether they require action. This should be incorporated into a daily “My Tasks” and “Inbox” review process.

For each item in your “Inbox” or “Recently assigned” section of your “My Tasks” project, identify whether a task is actionable or not. If the task is not actionable, it may be a reference item or something that needs to be filed for future reference - you might move it into whichever system you use for reference information (I use Obsidian for this purpose; Evernote and Google Keep are also good tools for storing snippets of information). If the task is actionable, determine the specific next action required to move it forward. For example, if the task is “write an article”, the next action may be “create an outline for the article”.

Use Asana subtasks to lay out the bite-size tasks required to achieve the objective. Pro-tip: you might want to assign both the parent task and the subtasks to yourself so that you can prioritise subtasks independently.

For each task, determine the outcome you want to achieve and what context is required to achieve it. Determining the outcome will help you determine whether the task is actually necessary and whether it is aligned with your goals. Can the task be completed in your current context? For example, if you need to make a phone call to complete the task, you may need to be in a quiet place where you can concentrate. If you’re not in the right context, you may need to defer the task to a later time. Specifying context is also useful for batching tasks that require similar context.

Finally, for any tasks that can and should be delegated to someone else, delegate them during your daily review.

Step 3 - the “Organise” step

The third step of the GTD productivity system is the “Organise” step. In this step, you organise your “My Tasks” into a system that makes sense to you. Asana has multiple mechanisms for effectively organising your tasks.

Using Asana sections

One of the simplest and most effective organising mechanisms you can put in place is to use Asana rules to create due-date driven sections of your “My Tasks” project. With these rules in place, once you put a due date on a task you can be confident it will be surfaced to you at the right time.

My “My Tasks” board has five sections (in order):

Recently assigned - tasks recently assigned to me by myself and others
Blocked - tasks waiting on other tasks/people
Today - my daily plan
Next - tasks due in the next week
Someday - everything else

Here are the rules I have set up on my “My Tasks” project:

Move tasks due today to the “Today” section.
Move tasks due in one day to the “Next” section.
Move tasks due in three days to the “Next” section.
Move tasks due in one week to the “Next” section.

These last three rules are a bit of a hack, but they work well for me. Asana doesn’t have a way of saying “move tasks due between one and seven days from now into this section”, so I’ve had to use multiple rules to make sure everything gets caught.

Using tags

Asana has general-purpose tagging included in all projects by default. You can use tags to categorise your tasks by just about any criteria you can think of. For example, you might use tags to organise tasks by:

Effort (low, medium, high)
Impact (low, medium, high)
Priority (low, medium, high)
Context (required location e.g. “office”, required equipment e.g. “sketchpad”)

… and then use your categorisations to plan your day. For example, “if I don’t have any high priority tasks to do today while I’m in the office, I’ll schedule work on medium priority tasks that require the office”.

Using custom fields

The premium tier of Asana includes the ability to create custom fields. Custom fields are great for when you want a more finely tuned instrument for categorising or measuring your tasks in order to organise them. For example, you might add a custom field to include an estimate of how many minutes a task will take and use that to plan your day. Alternatively, you can create a custom currency field to estimate the financial benefit of a task and use that as a prioritisation guide.

Use what works for you!

For me, just diligently doing my daily review and having the due-date driven sections of “My Tasks” is enough to keep me organised. However, in other Tech Team projects we use a combination of tags and custom columns to prioritise tasks for the team. The important thing is to experiment and find what works for you. In the next step, you’ll incorporate reflection into your daily review process and use it to continually improve.

Step 4 - the “Reflect” step

The “Reflect” step is the fourth step of the GTD productivity system. In this step, you review and reflect on your tasks and process to ensure that they are still relevant and aligned with your goals. Setting aside regular time to review your “My Tasks” and “Inbox” is critical to your ongoing GTD success.

Every day

I start each day with a 15 minute review, during which I complete the following steps:

Review my “Inbox”. Is there anything that requires action from me? Create tasks for those actions if they don’t already exist.
Switch to “My Tasks” and review “Recently assigned” and do (anything two minutes or less), delegate (re-assign) or defer (set a due date and move to the appropriate section) everything in that section.
Review “Blocked” and see if the tasks in that section are still in fact blocked. If they are not, move them to the appropriate section. If they are, see if there is anything I can do to unblock them (sometimes a gentle poke is all that’s needed).
Plan my day - review “Today” and “Next” and put tasks in priority order. Check my calendar and double check whether my plan is practical given the time I have available outside of meetings. Reflect on the previous day and whether I under- or over-estimated the time I needed for each task that I had planned to do.
(If time) Review “Someday” and check that the tasks in those sections are still relevant and categorised appropriately.

Here’s another representation of a similar process, from Sam T. Davies’ summary of Getting Things Done (which is well worth a read).

Every week

It’s important to see what you have accomplished and celebrate your wins! Doing this gives you a lovely little dopamine hit that keeps you motivated and focused on your goals.

In Asana, you can filter your “My Tasks” by “Completed” to see what you have completed in the last week. I use this view every Friday when the Tech Team does our weekly demos, both to celebrate the work I’ve done and to pick something cool and/or interesting to show off to the team.

Step 5 - the “Engage” step

The “Engage” step is the fifth and final step of the GTD productivity system. In this step, you use your GTD system in Asana to do the work! This article has laid out, broadly, all the tools you need to Get Things Done.

Capture tasks in Asana as soon as you learn about them. Don’t stress too much about documenting every aspect - just record enough information for you to know what to do with it during your daily review.
Clarify the task in your daily review. Determine whether there’s a clear action you can take. Where possible, update the title and description so that someone else could easily pick it up and do it. And if it can be delegated, delegate it.
Organise your tasks using a system that makes sense to you. Use due dates, tags, sections… whatever you find most effective.
Reflect on your performance and your system each day. Are you managing to stick to your plan? If not, why not? You might need to try several organisational approaches before you find the right one. Be kind to yourself - there will be days where you grossly overestimate or underestimate your capacity or the work. Try to get better calibrated on what is realistic.
Engage with the GTD philosophy. Look for the best next action to take from all of your tasks. Feel better about the tasks you are consciously choosing to do, and the tasks you are consciously choosing not to do.

My subjective experience of prioritising and executing work in this way is that I am less distracted and more satisfied with my work at the end of each day. I feel less reactive and more proactive. Sometimes important things show up and demand your immediate attention, and that’s fine. That’s the nature of working in a fast-paced, high-impact environment. However, I find that I am able to make the decision to devote my immediate attention more mindfully. I’m able to quickly judge the importance of the task against my daily plan and be confident that I’m not missing something important.

In David Rock’s excellent book, “Your Brain At Work” (9m video summary of core message), the author describes task prioritisation as one of the most cognitively demanding activities the human mind engages in. He also states that the best time to perform this activity is at the start of your day, when mental energy levels are high. Having a disciplined practice of reviewing your tasks and planning your day each morning is a great way of aligning your activities for your energy levels.

Good luck, and let’s Get Things Done!

Keeping on-call calm

Cat Truscott — Mon, 20 Mar 2023 00:00:00 UT

Keeping on-call calm

Cat Truscott 2023-03-20

Technology

Being on-call can be stressful and disruptive. We need to have a reasonable on-call process that balances the needs of the business with the well-being of our team.

At Bellroy we have found the key elements to our on-call process are:

We treat our team with kindness We rotate on-call responsibilities to distribute the workload. We run a two week shift with at least a month between the end of one shift and the start of the next one. Team members get extra time off as compensation for the imposition of taking an on-call shift. If an out of hours response is necessary, extra time off is also allocated. Never expect someone who’s been up half the night dealing with fires to complete a full day of work the next day.
Our alerts are actually critical enough to disturb someone. Ensure that a critical severity alert is actually critical. Waking someone up at 3am for a failed background process that can wait for the next day should not happen. Paging someone should be for a problem that is causing (or could cause) significant business disruption. Every calculated alarm should have some fault tolerance in it; a single bad data point should not trigger a page event. Alarms should be clear, informative and actionable by the team member.
Our documentation is accessible and up to date Team members should have access to clear and up-to-date documentation so they can quickly identify and resolve issues. We use backstage and we’ve found that storing the documentation in the git repository with the code helps. Keeping documentation up to date is a struggle as it will drift from reality. We include reviewing and updating documentation into our regular dependency update cycle. We also maintain a knowledge base around incidents and alerts that we’ve seen more than once.
We have clear expectations and escalation paths Team members should understand their responsibilities and know how long they have to respond to an incident. In the case that an incident is not acknowledged or can’t be solved by the on-call person, there should be a clear escalation path.
We train our team Make sure all team members have access to adequate training so they are able to actually fix problems as they arise.
We work towards continuous improvement No process is perfect and there is always room for improvement. After each significant incident we write a post mortem to record the resolution and root cause. Then we make recommendations on how to mitigate similar events in the future.

On-call is a necessary evil, but we can make it much less of a burden. Like most online businesses we need support 24/7 to minimise the impact of incidents on our systems. By implementing these elements we have developed a fair process that ensures support for our critical systems and a healthy work-life balance for our team.

Our technology stack, and how we got here

Mike Webb — Wed, 01 Mar 2023 00:00:00 UT

Our technology stack, and how we got here

Mike Webb 2023-03-01

Technology

We’ve come a long way in terms of our technology stack since launching the first version of bellroy.com on Shopify, way back in 2010. In this blog post, we delve into Bellroy’s technical history, and the company’s journey through various platforms and languages. We also reveal some of the challenges we faced, and innovative solutions we employed to overcome them.

Bellroy’s initial foray into e-commerce was through the popular Shopify platform. Although the platform was easy to set up and use, we soon ran into limitations customising it to meet Bellroy’s unique needs. As our reputation grew internationally, we needed a solution with multi-currency and multi-language support without having to maintain separate configurations for each combination. Thus, the company decided to migrate to Magento, a platform that - at the time - seemed a better fit for our needs.

We soon discovered that Magento presented its own set of challenges. Particularly: finding skilled developers who could work with the platform, and smoothly migrating product configurations between staging and production environments once we’d tweaked them. These challenges led the company to explore other options, ultimately resulting in a full website build using Ruby on Rails and React in 2014. This decision happened to have a positive effect on hiring; because those were niche technologies at the time, they attracted a certain type of developer - the type who naturally fitted well with Bellroy’s culture and values. These were passionate hobbyists who loved to get better at their craft.

The new website initially served Bellroy well, but as the company continued to grow, the codebase became increasingly difficult to maintain, and - despite having a more capable team - prone to costly mistakes. It became clear that we needed an even more robust and reliable technology stack.

Bellroy’s first experience with functional programming came in 2018 through the development of a new version of our checkout using Elm, a statically typed functional programming language for front-end web development. Errors all but evaporated, and we could refactor and add new features with confidence. The positive experience with Elm inspired us to explore other statically typed functional programming languages. We began to experiment with the Sorbet type-system and dry-monads library for Ruby. We eventually committed to migrating significant parts of our codebase to Haskell in 2019. With Elm and Haskell we found again - as we had with Ruby and React - that the majority of developers who gravitated towards those languages were focused on continuous improvement and the craft of programming, which proved to be a good fit for Bellroy.

Haskell and Elm are not without their challenges. Haskell has a steep learning curve and - compared to Ruby - low penetration. A few of our Ruby developers decided to move elsewhere rather than learn it, and it took months for those who stayed to get to grips with it. Elm is a solid language but suffers from being perceived as abandoned by its creator, despite having an active community. However, these languages have positive characteristics that are unignorable. We regularly have the experience of releasing new tools and services that just work - first time - and we don’t have to touch for months or years at a time beyond package updates. The very first service we deployed is responsible for all of the product images on bellroy.com, and has remained functionally unchanged since 2019.

In Ruby, we would write and test code defensively, trying to cater for every likely possibility. In Haskell and Elm, we rely on type design, the compiler and property tests to give us rock-solid guarantees that the code does only what we want it to do. As we’ve deprecated parts of our Ruby codebase, the Haskell equivalent is a fraction of the size.

We decided to explore statically typed functional programming languages - like Elm and Haskell - for their reliability, scalability, and maintainability. By reducing the complexity of code and the potential for human error, programming languages like these can save companies time and resources, allowing them to focus on the growth and development of their business rather than fighting fires. This has certainly been our experience, and we hope that this post will encourage others to consider these languages for their next projects.

The unreasonable effectiveness of polymorphic records

Clément Delafargue — Tue, 01 Nov 2022 00:00:00 UT

The unreasonable effectiveness of polymorphic records

Clément Delafargue 2022-11-01

Technology

Record types are an important part of typed functional programming languages. Separating data from behaviour creates the need for a convenient representation of key-value pairs. Most of the time, such records are monomorphic: each field has a specific type. Using record types this way (forgetting Haskell record idiosyncrasies for a minute) should be familiar to most of you. Haskell is also well known for its pervasive use of parametric polymorphism; as always, digging at the intersection of two interesting ideas is time well spent. Let’s have a look at a few patterns combining records and polymorphism. It should be fun!

Step 0: Monomorphic Records

Most records are monorphic. Here, the record type Account has kind Type: it is a regular type, with no type parameters.

data Account = Account {
  accountId :: Int,
  fullName :: Text
}
deriving stock (Eq, Show, Generic)
deriving anyclass (FromJSON, ToJSON)

The Haskell ecosystem provides us with quite a lot of tooling: here we derive Eq and Show instances, as well as Generic. Generic acts as an extension point, allowing other libraries to gain access to structural information and provide more tooling for free (conditions may apply). In this case, Generic allows us to derive JSON codecs. aeson’s FromJSON and ToJSON provide conversion from/to JSON objects shaped like our record.

This kind of tooling is not specific to Haskell, and is becoming commonplace even in more mainstream languages like Java.

Let’s have a look at more sophisticated use cases, and see how far we can go.

Step 1: The Humble Wrapper

Some records are intended as metadata wrappers around other data. Let’s consider pagination: instead of returning a list of all Account values we want to return only a limited number of them, with pointers to more accounts. One naive way to do that would be the following:

data PaginatedAccounts = PaginatedAccounts {
  pages :: Int,
  previous :: Maybe Int,
  next :: Maybe Int,
  items :: [Account]
}
deriving stock (Eq, Show, Generic)
deriving anyclass (FromJSON, ToJSON)

This is unsatisfying: pagination is a cross-cutting concern that appears in a lot of places. Creating a new Paginated- variant for each type would require a lot of boilerplate code and would make room for inconsistencies. So we definitely want to be describe pagination once, and then use it across our application.

In that case, making the items field polymorphic will do just what we want. We can still derive typeclasses like Eq, Show, as well as, for instance, aeson classes like FromJSON and ToJSON. The compiler will make sure that, when using these instances, a is implementing them as well.

data Paginated a = Paginated {
  pages :: Int,
  previous :: Maybe Int,
  next :: Maybe Int,
  items :: [a]
}
deriving stock (Eq, Show, Generic)
deriving anyclass (FromJSON, ToJSON)

Note that a is the type of one paginated item, not of the list of items.

Supercharging extensions

Deriving Eq, Show or aeson instances feel natural (after all, many other languages provide similar mechanisms). But this is Haskell, and we shouldn’t be afraid to dream a little bigger. GHC can derive Functor, Foldable, and Traversable (with the help of DeriveTraversable, which is enabled by default in the GHC2021 extension set) for us. With them, the humble wrapper can be used in many more contexts.

  …
  deriving stock (…, Functor, Foldable, Traversable)

The humble wrapper makes no assumptions about its contents. It lets you derive typeclass instances with no hand-holding, happily threading dependencies for you. It is now also more versatile, thanks to the Traversable instance and its friends. It lets you work with intermediate structures inferred on the go, without the need to plan them in advance.

For cross-cutting concerns that are highly generic, simple polymorphic records like this are a great fit. The Paginated record is defined separately from the items it will hold, and typeclasses take care of encoding dependencies. Sadly, not all problems look like this. In some cases, there is more coupling between the record itself and the moving parts it contains.

Step 1-bis

Let’s look at a different example. Here we want to model a database record, before and after insertion in a database. Before insertion, the primary key does not exist yet. It is generated by the database server when creating the database row.

data DatabaseRecord pk =
  DatabaseRecord {
    recordId :: pk,
    name :: Text,
    dateOfBirth :: Date
  }
  deriving stock (Eq, Show, Generic, Functor, Foldable, Traversable)
  deriving anyclass (FromJSON, ToJSON)

Here, the idea is not to use the full breadth of available types for pk. It is intended to be used for two types: unit (()) and RecordId (a type modeling a primary key for the given database table). Instead of being a reusable wrapper, DatabaseRecord is only intended as being reused in two versions DatabaseRecord (), and DatabaseRecord RecordId: one version is meant to represent a row prior to insertion, when the primary key is not yet known. The other represents a full record after insertion, where we know the generated primary key.

This is merely a shift of focus from the first example. We’re still using a type parameter to make a field polymorphic. Doing so, we still get typeclasses instances for free. Note that the aeson instances, while available through generics, may not make sense in all contexts (ignoring for a minute the fact that directly serializing a database row to JSON is usually a bad idea in actual codebases): For DatabaseRecord RecordId, the automatic derivation will serialize it as a JSON object, deferring to ToJSON RecordId for the contents of recordId.

{
  "recordId": "01GJYP9AST5HSBJKBMJBNT28V2",
  "name": "Pink Floyd",
  "dateOfBirth": "1979-11-30"
}

For DatabaseRecord (), we would expect the recordId field to be either absent or set to null. Sadly, that is not what happens. Since recordId has type (), its JSON representation will be computed in accordance with ToJSON (), which is… []. This behaviour is surprising, but makes sense: a JSON array can be seen as a tuple, and the unit type can be seen as an empty tuple. So unit can be seen as an empty JSON array.

{
  "recordId": [],
  "name": "Pink Floyd",
  "dateOfBirth": "1979-11-30"
}

It is always possible to create a type that’s isomorphic to () and to give it a ToJSON instance that serializes to null, but that becomes tedious. Worse, the easiest thing to implement (just using () and generic-derivated instances) will give undesired behaviour.

The problem here is that we are using a very polymorphic type when we are only expecting to use it in two concrete implementations. The openness here is both a blessing and a curse: we get things for free, but we allow unwanted states to exist.

Step 2

Let’s take a step back and think about what we actually want. We don’t care about the actual type of the recordId field: when it is present, its shape is fixed. We mostly care about the context of whether it is there or not.

data DatabaseRecord (f :: Type -> Type) =
  DatabaseRecord {
    recordId :: f RecordId,
    name :: Text,
    dateOfBirth :: Date
  }

Here, we are not taking a regular type parameter, but rather a type constructor parameter (its kind is Type -> Type). This lets us pin the concrete RecordId type in the record definition. Now, what can f be? We want to cover two cases:

there is a recordId
there is no recordId

For the first case, we can use Identity (newtype Identity a = Identity { runIdentity :: a }). It acts as a wrapper around any a. More importantly, it fits the Type -> Type kind. So DatabaseRecord Identity provides us with what we want: a record carrying a primary key.

Now for the second case, we don’t want any value, but we still need the Type -> Type kind. For this, we can use the Proxy type. Defined as data Proxy a = Proxy, it carries a phantom type. a appears in the left-hand side of the declaration, but is absent from the right-hand side. This means that in Proxy a, the type a only appears at the type level. The actual value Proxy is a constructor with no arguments, so there is no value of type a actually carried at run time. This is used a lot with type-level programming, but here we are just trying to fullfil the Type -> Type kind without providing an actual value. So DatabaseRecord Proxy provides us with what we want: a record not carrying a primary key.

Another benefit of taking the context as a parameter instead of the actual type is that the context can be reused for different fields. Database rows usually carry a created_at field that is also populated when the row is inserted. With the f parameter, it can be added to the record definition with minimal fuss:

data DatabaseRecord (f :: Type -> Type) =
  DatabaseRecord {
    recordId :: f RecordId,
    name :: Text,
    dateOfBirth :: Date,
    createdAt :: f UTCTime
  }

Had we chosen the previous way to model DatabaseRecord, we would now have

data DatabaseRecord pk cat =
  DatabaseRecord {
    recordId :: pk,
    name :: Text,
    dateOfBirth :: Date,
    createdAt :: cat
  }

Using a type constructor as a parameter is a very convenient way to model the presence or absence of certain fields while still pinning the expected types of those fields, when they should be there. For one field, this is good because it makes the actual types clear, but it becomes really interesting when there are multiple fields with different types that share the same context.

The flip side, as always when moving with more constrained solutions, is that we reduce polymorphism and then cannot benefit from what it provides. Here it’s not possible to automatically derive typeclasses: we have to hold GHC’s hand:

{-# LANGUAGE StandaloneDeriving #-}
{-# LANGUAGE UndecidableInstances #-}

…

deriving instance (Show (f RecordId), Show (f UTCTime))  => Show (D f)

Also, we have pinned the concrete type of the primary key and the creation date. However, the f type constructor is still unconstrained, so we can still model DatabaseRecord [] or DatabaseRecord (ContT Void (Coyoneda (Cotambara (Tannen Identity (Kleisly IO))) Int)) (for instance).

Step 3: Trees That Grow (or just branches, maybe?)

This is where we bring out the big guns. The Trees That Grow paper describes a mechanism that lets us do exactly what we want: explicitly choosing types based on the context, at the small cost of using a few advanced haskell features.

The idea here is to start by listing the contexts we care about:

{-# LANGUAGE DataKinds #-}

data DbContext = BeforeInsertion | AfterInsertion

With the help of DataKinds, this humble enum is promoted to the realm of types: DbContext can be used as a kind, in addition to a type, and the constructors 'BeforeInsertion and 'AfterInsertion (note the leading quotes: while not strictly required, they let us make it explicit when using types promoted from data constructors) are created as types in addition to BeforeInsertion and AfterInsertion being terms.

So this lets us have an enum at the type level, describing the context we are in. So (slight spoiler) we will define DatabaseRecord as data DatabaseRecord (context :: DbContext) = …. This way, We can only talk about DatabaseRecord BeforeInsertion and DatabaseRecord AfterInsertion.

Next is to decide the actual types of the recordId and createdAt fields. We cannot do this inside the definition of DatabaseRecord. What we need to do is to define something like a function: DbContext -> Type, that operates at the type level. With the help of another extension (TypeFamilies), we can do just that:

type family RecordIdType (context :: DbContext) where
  RecordIdType BeforeInsertion = ()
  RecordIdType AfterInsertion = RecordId

type family CreatedAtType (context :: DbContext) where
  CreatedAtType BeforeInsertion = ()
  CreatedAtType AfterInsertion = UTCTime

And finally, we can define our brand new DatabaseRecord:

data DatabaseRecord (context :: DbContext) =
  DatabaseRecord {
    recordId :: RecordIdType context,
    name :: Text,
    dateOfBirth :: Date,
    createdAt :: CreatedAtType context
  }

deriving instance (Show (RecordIdType context), Show (CreatedAtType context)) => Show (DatabaseRecord context)

All done! This is obviously the perfect solution, so you should use this technique everytime. I promise nothing bad will happen.

More seriously, this encoding is the tightest implementation of the problem statement. As you can see it is also a fairly complex one, and like before it requires manually declaring deriving instance dependencies.

What did we learn?

In a not-so-surprising turn of events, it appears that advanced type-level features give us more expressive power while making things more complex and applicable in fewer contexts.

“Simple” polymorphic records crop up a lot in day-to-day web programming and are very effective at keeping APIs consistent. This pattern interacts nicely with automatic deriving, especially for JSON instances. For better or worse, it also doesn’t prevent us developers from modeling nonsensical types.

Taking type constructors (of kind Type -> Type) as a type parameter is surprisingly effective at modeling context-dependent data requirements. Higher kinded types have a scary reputation outside Haskell, but they can have concrete use-cases.

Finally, sophisticated techniques such as trees that grow are definitely not something one would use every day, but they can be extremely effective at reducing redundancy in complex data modeling. Seeing such advanced mechanisms all mesh together to give rise to a concrete use-case is part of what makes Haskell, Haskell.

Assembling reliable records from semi-reliable systems

Chad Musick — Mon, 31 Oct 2022 00:00:00 UT

Assembling reliable records from semi-reliable systems

Chad Musick 2022-10-31

Data and Analytics

Ensuring good data quality is fundamental to effective data governance, and this blog post describes one method of managing the effects of out-of-order arrival and latency, which any networked system can encounter.

Record-keeping in commerce is an old task. Some of the earliest written records were related to commerce, such as this stone payslip. These days, most companies will be storing these records on paper, in spreadsheets, or in databases. A few companies will have a cohesive data fabric that helps to both manage records and make effective use of them.

One of the technically trickier aspects of ensuring good data quality is that information may arrive from a set of outside sources, and those sources may supply information at different paces and different places wearing different faces. There are many ways to solve this problem. Here, we present one such method that provides good visibility into the current status of records and is robust against the order of arrival of information.

Let’s assume we have a very simple record with the following structure.

Order ID	Product	Package tracking	Delivered on	Customer rating
1	Sling Mini MIRUM® Edition

If everything goes exactly right, we will receive the package tracking information, then the delivery information, and then (we hope) the customer rating. Across millions of transactions, though, this process will probably break at least a few times.

Here’s one way it could break.

sequenceDiagram
Bellroy->>Warehouse: Let's ship Order 1
Warehouse->>Shipper: Here's Order 1 to ship
Shipper->>Warehouse: Here's your tracking id
Note right of Bellroy: Network error
Customer->>Bellroy: I love it! 5 stars
Bellroy->>Customer: We're glad you love it. Thanks for letting us know.
Bellroy-->>Warehouse: We're missing tracking
for Order 1
Warehouse->>Bellroy: Here it is
Bellroy->>Shipper: What's the delivery status?
Shipper->>Bellroy: We delivered last week

To construct this record requires information from multiple systems. That information may not arrive at all (e.g., due to a network error) or may arrive out of order (e.g., receiving feedback before the item is known to have been shipped). Any system that relies on 100% message arrival and 100% in-order arrival is likely to produce poor quality data. Fortunately, we can fix this by a simple process.

Document the ‘good’ states of a record.
Document the acceptable latency for information arrival.
Keep track of the timing of information arrival.
Check that the state of a record is in the ‘good’ state for a process before sending the record to that process.
Re-request late information or otherwise mitigate the problem, and possibly alert someone. This can be done via email or most workplace collaboration tools.

A typical toolchain for this will include a database, an ELT/ETL tool, an orchestration tool, and a messaging system. At present we use PostgreSQL as our transactional database, Apache NiFi for ELT and orchestration, Slack for many notifications and Zabbix for performance monitoring. These communicate via webhooks, database protocols, and Apache Kafka. Except for Slack, all of these are open-source products.

We’ll write in future posts about our affinity for open-source software and the contributions we make to maintaining and expanding the availability of useful open-source products.

Welcome!

Mike Webb — Wed, 19 Oct 2022 00:00:00 UT

Welcome!

Mike Webb 2022-10-19

Technology

Welcome. To the space where Bellroy’s crew gets technical. Here, we aim to share some of the ideas, tools and models that drive Bellroy behind the scenes.

When the team at Bellroy first considered starting this blog, we asked ourselves “what should this blog do that The Journal doesn’t?” Bellroy’s mission is to “…help you move through the world with ease and energy. While helping the world, and our crew, flourish”. Part of helping the world and our crew flourish is sharing the challenges and innovations we encounter along the way. And this blog allows us to provide deeper insight into those. We’re committed to helping our people learn, and one of the best ways to learn is by teaching. By having our crew share some of their eureka moments, we hope that it can solidify things for them, as well as help someone else out there solve a problem.

We’re proud of our products and the way we present them to the world. A hell of a lot of deliberate thought and effort goes into this. While our website presents the smooth veneer of the ‘duck above water’, below the water the legs (that is, our team) are working double-time and in all directions to keep things that way. We’re pleased to provide this little window into how Bellroy operates behind the scenes.

We’ve built this blog so that any team at Bellroy can contribute posts, and to start we have a great lineup of technical posts in our schedule. Over the next few months, we’ll publish posts on topics such as:

Our software engineering philosophy
Our systems architecture
Our infrastructure
Our machine-led and human-led processes
Our approaches to problem solving
… and the business drivers behind many of these

We hope you enjoy learning more about how we do what we do - as we continue to explore better ways.

One last technical note - we love functional programming and open-source software. We depend on open-source products and contribute back wherever we can. For this blog, we use Hakyll as our static site generator. We use Nix to manage our development and build environments. The blog repository was seeded using Robert Pearce’s excellent Hakyll + Nix template.

Exploring Better Ways - Bellroy

Reflections on Bellroy's Tech Team 2025

What we worked on

The cost of doing it all at once

Looking ahead

Some Haskell idioms we like

Explicit construction in concrete Monads

Design modules for qualified import

Unpack large patterns using a where clause

Use inverseMap liberally

Takeaways

Free applicatives, the handle pattern, and remote systems

Handles, as Bellroy uses them

Building our applicative

What is a “free structure”?

Building a Query

Running a Query

Extracting requests

Payoffs and limitations

Designing for the different stages of a system's life

The Clean Architecture revelation

The ultimate test

When architecture becomes archaeology

The missing framework

Designing for tomorrow’s problems

Integrating Effectful and Persistent

A short introduction to effectful

Our experiment

Using the Persist backend effect

Conclusions

Open Source at Bellroy: Supporting Old GHC Versions

Which GHCs to target?

Which old versions to remove?

Which Haskell dialect to use?

What library bounds to use for dependencies?

What dependencies are worth taking on?

Conclusion

Bellroy Technology Team: 2024 in Review

Solving a ResourceT-related space leak in production

Problem

Hunting down space leaks

Locating the space leak using profiling

Diving into the code

Creating a minimal reproducer

Lessons to learn

Save Effort: Build a Bash One-Liner

How We Use Both Process Orchestration and Process Choreography

Message passing in software

Orchestration: Centralised control

Choreography: Independent actors deciding how to perform

Where is the boundary?

How Bellroy uses both orchestration and choreography

Servant on AWS Lambda, and Two New Libraries

Serverless Web Services on AWS…

…in Haskell

Building APIs: Servant

Serving ActiveResource APIs

Conclusions

Using Dhall To Manage GitHub Actions Workflows

Don’t Repeat Yourself

Life before Dhall

Enter the Dhall configuration language

github-actions-dhall

How we use Dhall

How Dhall helped us

Bellroy Technology Team: 2023 in Review

Using Shape Up - What Works, What We've Changed, What's Next

Introduction

Understanding the Shape Up Method

Customizing Shape Up to Fit Our Needs

Impact on Productivity and Project Outcomes

Future Plans and Improvements

The History of Nix at Bellroy

Phase 1: Developer Shells via shell.nix

Phase 2: Building Haskell Deployment Packages using haskell.nix

Phase 3: Private Binary Cache using Amazon S3 and GitHub Actions

Setting up the Bucket

Setting up Keys

Setting up Clients

Setting up the Workflow

Explicit construction in concrete `Monad`s

Unpack large patterns using a `where` clause

Use `inverseMap` liberally

Building a `Query`

Running a `Query`

A short introduction to `effectful`

Using the `Persist backend` effect

`github-actions-dhall`

Phase 1: Developer Shells via `shell.nix`

Phase 2: Building Haskell Deployment Packages using `haskell.nix`