Functional Programming StrategiesIn Scala with Cats

By Noel Welsh

June 2024 Edition

Published by Inner Product

Preface

Some twenty years ago I started my first job in the UK. This job involved a commute by train, giving me about an hour a day to read without distraction. Around about the same time I first heard about Structure and Interpretation of Computer Programs, referred to as the “wizard book” and spoken of in reverential terms. It sounded like the just the thing for a recent graduate looking to become a better developer. I purchased a copy and spent the journey reading it, doing most of the exercises in my head. Structure and Interpretation of Computer Programs was already an old book at this time, and it’s programming style was archaic. However it’s core concepts were timeless and it’s fair to say it absolutely blew my mind, putting me on a path I’m still on today.

Another notable stop on this path occured some ten years ago when Dave and I started writing Scala with Cats. In Scala with Cats we attempted to explain the core type classes found in the Cats library, and their use in building software. I’m proud of the book we wrote together, but time and experience showed that type classes are only a small piece of the puzzle of building software in a functional programming style. We needed a much wider scope if we were to show people how to effectively build software with all the tools that functional programming provides. Still, writing a book is a lot of work, and we were busy with other projects, so Scala with Cats remained largely untouched for many years.

Around 2020 I got the itch to return to Scala with Cats. My initial plan was simply to update the book for Scala 3. Dave was busy with other projects so I decided to go alone. As the writing got underway I realized I really wanted to cover the additional topics I thought were missing. If Scala with Cats was a good book, I wanted to aim to write a great book; one that would contain almost everything I had learned about building software. The title Scala with Cats no longer fit the content, and hence I adopted a new name for what is largely a new book. The result, Functional Programming Strategies in Scala with Cats, is what you are reading now. I hope you find it useful, and I hope that just maybe some young developer will find this book inspiring the same way I found Structure and Interpretation of Computer Programs inspiring all those years ago.

Preface from Scala with Cats

The aims of this book are two-fold: to introduce monads, functors, and other functional programming patterns as a way to structure program design, and to explain how these concepts are implemented in Cats.

Monads, and related concepts, are the functional programming equivalent of object-oriented design patterns—architectural building blocks that turn up over and over again in code. They differ from object-oriented patterns in two main ways:

This generality means they can be difficult to understand. Everyone finds abstraction difficult. However, it is generality that allows concepts like monads to be applied in such a wide variety of situations.

In this book we aim to show the concepts in a number of different ways, to help you build a mental model of how they work and where they are appropriate. We have extended case studies, a simple graphical notation, many smaller examples, and of course the mathematical definitions. Between them we hope you’ll find something that works for you.

Ok, let’s get started!

Versions

This book is written for Scala 3.3.4 and Cats 2.10.0. Here is a minimal build.sbt containing the relevant dependencies and settings1:

scalaVersion := "3.3.4"

libraryDependencies +=
  "org.typelevel" %% "cats-core" % "2.10.0"

scalacOptions ++= Seq(
  "-Xfatal-warnings"
)

Template Projects

For convenience, we have created a Giter8 template to get you started. To clone the template type the following:

$ sbt new scalawithcats/cats-seed.g8

This will generate a sandbox project with Cats as a dependency. See the generated README.md for instructions on how to run the sample code and/or start an interactive Scala console.

The cats-seed template is very minimal. If you’d prefer a more batteries-included starting point, check out Typelevel’s sbt-catalysts template:

$ sbt new typelevel/sbt-catalysts.g8

This will generate a project with a suite of library dependencies and compiler plugins, together with templates for unit tests and documentation. See the project pages for catalysts and sbt-catalysts for more information.

Conventions Used in This Book

This book contains a lot of technical information and program code. We use the following typographical conventions to reduce ambiguity and highlight important concepts:

Typographical Conventions

New terms and phrases are introduced in italics. After their initial introduction they are written in normal roman font.

Terms from program code, filenames, and file contents, are written in monospace font. Note that we do not distinguish between singular and plural forms. For example, we might write String or Strings to refer to java.lang.String.

References to external resources are written as hyperlinks. References to API documentation are written using a combination of hyperlinks and monospace font, for example: scala.Option.

Source Code

Source code blocks are written as follows. Syntax is highlighted appropriately where applicable:

object MyApp extends App {
  println("Hello world!") // Print a fine message to the user!
}

Most code passes through mdoc to ensure it compiles. mdoc uses the Scala console behind the scenes, so we sometimes show console-style output as comments:

"Hello Cats!".toUpperCase
// res0: String = "HELLO CATS!"

Callout Boxes

We use two types of callout box to highlight particular content:

Tip callouts indicate handy summaries, recipes, or best practices.

Advanced callouts provide additional information on corner cases or underlying mechanisms. Feel free to skip these on your first read-through—come back to them later for extra information.

License

This work is licensed under CC BY-SA 4.0. To view a copy of this license, visit http://creativecommons.org/licenses/by-sa/4.0/

Portions of this work are based on Scala with Cats by Dave Pereira-Gurnell and Noel Welsh, which is licensed under CC BY-SA 3.0.

1 Functional Programming Strategies

This is a book on strategies for creating code in a functional programming (FP) style, seen through a Scala lens. If you understand most of the mechanics of Scala, but feel there is something missing in your understanding of how to use the language effectively, this book is for you. If you don’t know so much Scala, but are prepared to learn it as part of learning about functional programming, this book is also for you. It covers the usual functional programming abstractions like monads and monoids, but more than that it tries to teach you how to think like a functional programmer. It’s a book as much about process as it is about the code that results from process, and in particular it focuses on what I call metacognitive programming strategies.

I would guess most programmers would struggle to describe the process they use to write code. Some might mention “test driven development” and perhaps “pair programming”, but I wouldn’t expect much more from the general programming population. Both the above techniques come from eXtreme Programming, which dates to the late 90s, and you would hope our field had added new knowledge in that time. But it’s not really the fault of the developers—most of them haven’t been taught any explicit process. Our industry certainly likes to talk about process, in the form of agile, kanban boards, and so on, and in recent times a tremendous effort has spent on expanding those who are taught programming. However the actual programming—the bit that produces the code that is the whole point of the endeavour—is still largely treated as magic. It doesn’t have to be that way.

Functional programmers love fancy words for simple ideas, so it’s no surprise I’m drawn to metacognitive programming strategies. Let’s unpack that phrase to see what it means. Metacognition means thinking about thinking. A lot of research has shown the benefits of metacognition in learning, and that it is an important part of developing expertise. Metacognition is not just one thing—it’s not sufficient to just tell someone to think about their thinking. Rather we should expect metacognition to be a collection of different strategies, some of which are general and some of which are domain specific. From this we get the idea of metacognitive programming strategies—explicitly naming and describing different thinking strategies that proficient programmers use.

I believe metacognitive programming strategies are useful for both beginners and experts. For beginners we can make programming a more systematic and repeatable process. Producing code no longer requires magic in the majority of cases, but rather the application of some well defined steps. For experts, the benefit is exactly the same. At least that is my experience (and I believe I’ve been programming long enough to call myself an expert.) By having an explicit process I can run it exactly the same way every day, which makes my code simpler to write and read, and saves my brain cycles for more important problems. In some ways this is an attempt to bring to programming the benefit that process and standardization has brought to manufacturing, particularly the “Toyota Way”. In Toyota’s process individuals are expected to think about how their work is done and how it can be improved. This is, in effect, metacognition for assembly lines. This is only possible if the actual work itself does not require their full attention. The dramatic improvements in productivity and quality in car manufacturing that Toyota pioneered speak to the effectiveness of this approach. Software development is more varied than car manufacturing but we should still expect some benefit, particularly given the primitive state of our current industry.

The question then becomes: what metacognitive strategies can programmers use? I believe that functional programming is particularly well suited to answer this question. A major theme in functional programming research is finding and naming useful code structures. Once we have discovered a useful abstraction we can get the programmer to ask themselves “would this abstraction solve this problem?” This is essentially what the design patterns community did, also back in the nineties, but there is an important difference. The academic FP community strongly values formal models, which means that the building blocks of FP have a precision that design patterns lack. However there is more to process than categorizing the output. There is also the actual process of how the code comes to be. Code doesn’t usually spring fully formed from our keyboard, and in the iterative refinement of code we also find structure. Here the academic FP community has less to say, but there is a strong folklore of techniques such as “type driven development”

Over the last ten or so years of programming and teaching programming I’ve collected a wide range of strategies. Some come from others (for example, How to Design Programs and its many offshoots remain very influential for me) and some I’ve found myself. Ultimately I don’t think anything here is new; rather my contribution is in collecting and presenting these strategies as one coherent whole.

1.1 Three Levels for Thinking About Code

Let’s start thinking about thinking about programming, with a model that describes three different levels that we can use to think about code. The levels, from highest to lowest, are paradigm, theory, and craft. Each level provides guidance for the ones below.

The paradigm level refers to the programming paradigm, such as object-oriented or functional programming. You’re probably familiar with these terms, but what exactly is a programming paradigm? To me, the core of a programming paradigm is a set of principles that define, usually somewhat loosely, the properties of good code. A paradigm is also, implicitly, a claim that code that follows these principles will be better than code that does not. For functional programming I believe these principles are composition and reasoning. I’ll explain these shortly. Object-oriented programmers might point to, say, the SOLID principles as guiding their coding decisions.

The importance of the paradigm is that it provides criteria for choosing between different implementation strategies. There are many possible solutions for any programming problem, and we can use the principles in the paradigm to decide which approach to take. For example, if we’re a functional programmer we can consider how easily we can reason about a particular implementation, or how composable it is. Without the paradigm we have no basis for making a choice.

The theory level translates the broad principles of the paradigm to specific well defined techniques that apply to many languages within the paradigm. We are still, however, at a level above the code. Design patterns are an example in the object-oriented world. Algebraic data types are an example in functional programming. Most languages that are in the functional programming paradigm, such as Haskell and O’Caml, support algebraic data types, as do many languages that straddle multiple paradigms, such as Rust, Scala, and Swift.

The theory level is where we find most of our programming strategies.

At the craft level we get to actual code, and the language specific nuance that goes into it. An example in Scala is the implementation of algebraic data types in terms of sealed trait and final case class in Scala 2, or enum in Scala 3. There are many concerns at this level that are important for writing idiomatic code, such as placing constructors on companion objects in Scala, that are not relevant at the higher levels.

In the next section I’ll describe the functional programming paradigm. The remainder of this book is primarily concerned with theory and craft. The theory is language agnostic but the craft is firmly in the world of Scala. Before we move onto the functional programming paradigm are two points I want to emphasize:

  1. Paradigms are social constructs. They change over time. Object-oriented programming as practiced today differs from the style originally used in Simula and Smalltalk, and functional programming today is very different from the original LISP code.

  2. The three level organization is just a tool for thought. In the real world it is more complicated.

1.2 Functional Programming

This is a book about the techniques and practices of functional programming (FP). This naturally leads to the question: what is FP and what does it mean to write code in a functional style? It’s common to view functional programming as a collection of language features, such as first class functions, or to define it as a programming style using immutable data and pure functions. (Pure functions always return the same output given the same input.) This was my view when I started down the FP route, but I now believe the true goals of FP are enabling local reasoning and composition. Language features and programming style are in service of these goals. Let me attempt to explain the meaning and value of local reasoning and composition.

1.2.1 What Functional Programming Is

I believe that functional programming is a hypothesis about software quality: that it is easier to write and maintain software that can be understood before it is run, and is built of small reusable components. The first property is known as local reasoning, and the second as composition. Let’s address each in turn.

Local reasoning means we can understand pieces of code in isolation. When we see the expression 1 + 1 we know what it means regardless of the weather, the database, or the current status of our Kubernetes cluster. None of these external events can change it. This is a trivial and slightly silly example, but it illustrates the point. A goal of functional programming is to extend this ability across our code base.

It can help to understand local reasoning by looking at what it is not. Shared mutable state is out because relying on shared state means that other code can change what our code does without our knowledge. It means no global mutable configuration, as found in many web frameworks and graphics libraries for example, as any random code can change that configuration. Metaprogramming has to be carefully controlled. No monkey patching, for example, as again it allows other code to change our code in non-obvious ways. As we can see, adapting code to enable local reasoning can mean quite some sweeping changes. However if we work in a language that embraces functional programming this style of programming is the default.

Composition means building big things out of smaller things. Numbers are compositional. We can take any number and add one, giving us a new number. Lego is also compositional. We compose Lego by sticking it together. In the particular sense we’re using composition we also require the original elements we combine don’t change in any way when they are composed. When we create by 2 by adding 1 and 1 we get a new result that doesn’t change what 1 means.

We can find compositional ways to model common programming tasks once we start looking for them. React components are one example familiar to many front-end developers: a component can consist of many components. HTTP routes can be modelled in a compositional way. A route is a function from an HTTP request to either a handler function or a value indicating the route did not match. We can combine routes as a logical or: try this route or, if it doesn’t match, try this other route. Processing pipelines are another example that often use sequential composition: perform this pipeline stage and then this other pipeline stage.

1.2.1.1 Types

Types are not strictly part of functional programming but statically typed FP is the most popular form of FP and sufficiently important to warrant a mention. Types help compilers generate efficient code but types in FP are as much for the programmer as they are the compiler. Types express properties of programs, and the type checker automatically ensures that these properties hold. They can tell us, for example, what a function accepts and what it returns, or that a value is optional. We can also use types to express our beliefs about a program and the type checker will tell us if those beliefs are correct. For example, we can use types to tell the compiler we do not expect an error at a particular point in our code and the type checker will let us know if this is the case. In this way types are another tool for reasoning about code.

Type systems push programs towards particular designs, as to work effectively with the type checker requires designing code in a way the type checker can understand. As modern type systems come to more languages they naturally tend to shift programmers in those languages towards a FP style of coding.

1.2.2 What Functional Programming Isn’t

In my view functional programming is not about immutability, or keeping to “the substitution model of evaluation”, and so on. These are tools in service of the goals of enabling local reasoning and composition, but they are not the goals themselves. Code that is immutable always allows local reasoning, for example, but it is not necessary to avoid mutation to still have local reasoning. Here is an example of summing a collection of numbers.

def sum(numbers: List[Int]): Int = {
  var total = 0
  numbers.foreach(x => total = total + x)
  total
}

In the implementation we mutate total. This is ok though! We cannot tell from the outside that this is done, and therefore all users of sum can still use local reasoning. Inside sum we have to be careful when we reason about total but this block of code is small enough that it shouldn’t cause any problems.

In this case we can reason about our code despite the mutation, but the Scala compiler can determine that this is ok. Scala allows mutation but it’s up to us to use it appropriately. A more expressive type system, perhaps with features like Rust’s, would be able to tell that sum doesn’t allow mutation to be observed by other parts of the system2. Another approach, which is the one taken by Haskell, is to disallow all mutation and thus guarantee it cannot cause problems.

Mutation also interferes with composition. For example, if a value relies on internal state then composing it may produce unexpected results. Consider Scala’s Iterator. It maintains internal state that is used to generate the next value. If we have two Iterators we might want to combine them into one Iterator that yields values from the two inputs. The zip method does this.

This works if we pass two distinct generators to zip.

val it = Iterator(1, 2, 3, 4)

val it2 = Iterator(1, 2, 3, 4)
it.zip(it2).next()
// res0: Tuple2[Int, Int] = (1, 1)

However if we pass the same generator twice we get a surprising result.

val it3 = Iterator(1, 2, 3, 4)
it3.zip(it3).next()
// res1: Tuple2[Int, Int] = (1, 2)

The usual functional programming solution is to avoid mutable state but we can envisage other possibilities. For example, an effect tracking system would allow us to avoid combining two generators that use the same memory region. These systems are still research projects, however.

So in my opinion immutability (and purity, referential transparency, and no doubt more fancy words that I have forgotten) have become associated with functional programming because they guarantee local reasoning and composition, and until recently we didn’t have the language tools to automatically distinguish safe uses of mutation from those that cause problems. Restricting ourselves to immutability is the easiest way to ensure the desirable properties of functional programming, but as languages evolve this might come to be regarded as a historical artifact.

1.2.3 Why It Matters

I have described local reasoning and composition but have not discussed their benefits. Why are they are desirable? The answer is that they make efficient use of knowledge. Let me expand on this.

We care about local reasoning because it allows our ability to understand code to scale with the size of the code base. We can understand module A and module B in isolation, and our understanding does not change when we bring them together in the same program. By definition if both A and B allow local reasoning there is no way that B (or any other code) can change our understanding of A, and vice versa. If we don’t have local reasoning every new line of code can force us to revisit the rest of the code base to understand what has changed. This means it becomes exponentially harder to understand code as it grows in size as the number of interactions (and hence possible behaviours) grows exponentially. We can say that local reasoning is compositional. Our understanding of module A calling module B is just our understanding of A, our understanding of B, and whatever calls A makes to B.

We introduced numbers and Lego as examples of composition. They have an interesting property in common: the operations that we can use to combine them (for example, addition, subtraction, and so on for numbers; for Lego the operation is “sticking bricks together”) give us back the same kind of thing. A number multiplied by a number is a number. Two bits of Lego stuck together is still Lego. This property is called closure: when you combine things you end up with the same kind of thing. Closure means you can apply the combining operations (sometimes called combinators) an arbitrary number of times. No matter how many times you add one to a number you still have a number and can still add or subtract or multiply or…you get the idea. If we understand module A, and the combinators that A provides are closed, we can build very complex structures using A without having to learn new concepts! This is also one reason functional programmers tend to like abstractions such a monads (beyond liking fancy words): they allow us to use one mental model in lots of different contexts.

In a sense local reasoning and composition are two sides of the same coin. Local reasoning is compositional; composition allows local reasoning. Both make code easier to understand.

1.2.4 The Evidence for Functional Programming

I’ve made arguments in favour of functional programming and I admit I am biased—I do believe it is a better way to develop code than imperative programming. However, is there any evidence to back up my claim? There has not been much research on the effectiveness of functional programming, but there has been a reasonable amount done on static typing. I feel static typing, particularly using modern type systems, serves as a good proxy for functional programming so let’s look at the evidence there.

In the corners of the Internet I frequent the common refrain is that static typing has neglible effect on productivity. I decided to look into this and was surprised that the majority of the results I found support the claim that static typing increases productivity. For example, the literature review in this dissertation (section 2.3, p16–19) shows a majority of results in favour of static typing, in particular the most recent studies. However the majority of these studies are very small and use relatively inexperienced developers—which is noted in the review by Dan Luu that I linked. My belief is that functional programming comes into its own on larger systems. Furthermore, programming languages, like all tools, require proficiency to use effectively. I’m not convinced very junior developers have sufficient skill to demonstrate a significant difference between languages.

To me the most useful evidence of the effectiveness of functional programming is that industry is adopting functional programming en masse. Consider, say, the widespread and growing adoption of Typescript and React. If we are to argue that FP as embodied by Typescript or React has no value we are also arguing that the thousands of Javascript developers who have switched to using them are deluded. At some point this argument becomes untenable.

This doesn’t mean we’ll all be using Haskell in five years. More likely we’ll see something like the shift to object-oriented programming of the nineties: Smalltalk was the paradigmatic example of OO, but it was more familiar languages like C++ and Java that brought OO to the mainstream. In the case of FP this probably means languages like Scala, Swift, Kotlin, or Rust, and mainstream languages like Javascript and Java continuing to adopt more FP features.

1.2.5 Final Words

I’ve given my opinion on functional programming—that the real goals are local reasoning and composition, and programming practices like immutability are in service of these. Other people may disagree with this definition, and that’s ok. Words are defined by the community that uses them, and meanings change over time.

Functional programming emphasises formal reasoning, and there are some implications that I want to briefly touch on.

Firstly, I find that FP is most valuable in the large. For a small system it is possible to keep all the details in our head. It’s when a program becomes too large for anyone to understand all of it that local reasoning really shows its value. This is not to say that FP should not be used for small projects, but rather that if you are, say, switching from an imperative style of programming you shouldn’t expect to see the benefit when working on toy projects.

The formal models that underlie functional programming allow systematic construction of code. This is in some ways the reverse of reasoning: instead of taking code and deriving properties, we start from some properties and derive code. This sounds very academic but is in fact very practical, and how I develop most of my code.

Finally, reasoning is not the only way to understand code. It’s valuable to appreciate the limitations of reasoning, other methods for gaining understanding, and using a variety of strategies depending on the situation.

In this first part of the book we’re building the foundational strategies on which the rest of the book will build and elaborate. In Chapter 2 we look at algebraic data types, which are our main way of modelling data. We turn to codata in Chapter 3, which is the opposite, or dual, or algebraic data. Type classes are the focus on Chapter 4, while fundamentals of interpreters are discussed in Chapter 5. These four strategies all describe code artifacts. For example, we can label part of code as an algebraic data type or a type class. We’ll also see strategies that help us write code but don’t necessarily end up directly reflected in it, such as following the types.

2 Algebraic Data Types

This chapter has our first example of a programming strategy: algebraic data types. Any data we can describe using logical ands and logical ors is an algebraic data type. Once we recognize an algebraic data type we get three things for free:

The key point is this: from an implementation independent representation of data we can automatically derive most of the interesting implementation specific parts of working with that data.

We’ll start with some examples of data, from which we’ll extract the common structure that motivates algebraic data types. We will then look at their representation in Scala 2 and Scala 3. Next we’ll turn to structural recursion for transforming algebraic data types, followed by structural corecursion for constructing them. We’ll finish by looking at the algebra of algebraic data types, which is interesting but not essential.

2.1 Building Algebraic Data Types

Let’s start with some examples of data from a few different domains. These are simplified description but they are all representative of real applications.

A user in a discussion forum will typically have a screen name, an email address, and a password. Users also typically have a specific role: normal user, moderator, or administrator, for example. From this we get the following data:

A product in an e-commerce store might have a stock keeping unit (a unique identifier for each variant of a product), a name, a description, a price, and a discount.

In two-dimensional vector graphics it’s typical to represent shapes as a path, which is a sequence of actions of a virtual pen. The possible actions are usually straight lines, Bezier curves, or movement that doesn’t result in visible output. A straight line has an end point (the starting point is implicit), a Bezier curve has two control points and an end point, and a move has an end point.

What is common between all the examples above is that the individual elements—the atoms, if you like—are connected by either a logical and or a logical or. For example, a user is a screen name and an email address and a password and a role. A 2D action is a straight line or a Bezier curve or a move. This is the core of algebraic data types: an algebraic data type is data that is combined using logical ands or logical ors. Conversely, whenever we can describe data in terms of logical ands and logical ors we have an algebraic data type.

2.1.1 Sums and Products

Being functional programmers we can’t let a simple concept go without attaching some fancy jargon:

So algebraic data types consist of sum and product types.

2.1.2 Closed Worlds

Algebraic data types are closed worlds, which means they cannot be extended after they have been defined. In practical terms this means we have to modify the source code where we define the algebraic data type if we want to add or remove elements.

The closed world property is important because it gives us guarantees we would not otherwise have. In particular, it allows the compiler to check that we handle all possible cases when we use an algebraic data type. This is known as exhaustivity checking. This is an example of how functional programming prioritizes reasoning about code—in this case automated reasoning by the compiler—over other properties such as extensibility. We’ll learn more about exhaustivity checking soon.

2.2 Algebraic Data Types in Scala

Now we know what algebraic data types are, we will turn to their representation in Scala. The important point here is that the translation to Scala is entirely determined by the structure of the data; no thinking is required! This means the work is in finding the structure of the data that best represents the problem at hand. Work out the structure of the data and the code directly follows from it.

As algebraic data types are defined in terms of logical ands and logical ors, to represent algebraic data types in Scala we must know how to represent these two concepts. Scala 3 simplifies the representation of algebraic data types compared to Scala 2, so we’ll look at each language version separately.

I’m assuming that you’re familiar with the language features we use to represent algebraic data types in Scala, so I won’t be going over them.

2.2.1 Algebraic Data Types in Scala 3

In Scala 3 a logical and (a product type) is represented by a final case class. If we define a product type A is B and C, the representation in Scala 3 is

final case class A(b: B, c: C)

Not everyone makes their case classes final, but they should. A non-final case class can still be extended by a class, which breaks the closed world criteria for algebraic data types.

A logical or (a sum type) is represented by an enum. For the sum type A is B or C, the Scala 3 representation is

enum A {
  case B
  case C
}

There are a few wrinkles to be aware of.

If we have a sum of products, such as:

the representation is

enum A {
  case B(d: D, e: E)
  case C(f: F, g: G)
}

In other words we don’t write final case class inside an enum. You also can’t nest enum inside enum. Nested logical ors can be rewritten into a single logical or containing only logical ands (known as disjunctive normal form) so this is not a limitation in practice. However the Scala 2 representation is still available in Scala 3 should you want more expressivity.

2.2.2 Algebraic Data Types in Scala 2

A logical and (product type) has the same representation in Scala 2 as in Scala 3. If we define a product type A is B and C, the representation in Scala 2 is

final case class A(b: B, c: C)

A logical or (a sum type) is represented by a sealed abstract class. For the sum type A is a B or C the Scala 2 representation is

sealed abstract class A
final case class B() extends A
final case class C() extends A

Scala 2 has several little tricks to defining algebraic data types.

Firstly, instead of using a sealed abstract class you can use a sealed trait. There isn’t much practical difference between the two. When teaching beginners I’ll often use sealed trait to avoid having to introduce abstract class. I believe sealed abstract class has slightly better performance and Java interoperability, but I haven’t tested this. I also think sealed abstract class is closer, semantically, to the meaning of a sum type.

For extra style points we can extend Product with Serializable from sealed abstract class. Compare the reported types below with and without this little addition.

Let’s first see the code without extending Product and Serializable.

sealed abstract class A
final case class B() extends A
final case class C() extends A
val list = List(B(), C())
// list: List[A extends Product with Serializable] = List(B(), C())

Notice how the type of list includes Product and Serializable.

Now we have extending Product and Serializable.

sealed abstract class A extends Product with Serializable
final case class B() extends A
final case class C() extends A
val list = List(B(), C())
// list: List[A] = List(B(), C())

Much easier to read!

You’ll only see this in Scala 2. Scala 3 has the concept of transparent traits, which aren’t reported in inferred types, so you’ll see the same output in Scala 3 no matter whether you add Product and Serializable or not.

Finally, we can use a case object instead of a case class when we’re defining some type that holds no data. For example, reading from a text stream, such as a terminal, can return a character or the end-of-file. We can model this as

sealed abstract class Result
final case class Character(value: Char) extends Result
case object Eof extends Result

As the end-of-file indicator Eof has no associated data we use a case object. There is no need to mark the case object as final, as objects cannot be extended.

2.2.3 Examples

Let’s make the discussion above more concrete with some examples.

2.2.3.1 Role and User

In the discussion forum example, we said a role is normal, moderator, or administrator. This is a logical or, so we can directly translate it to Scala using the appropriate pattern. In Scala 3 we write

enum Role {
  case Normal
  case Moderator
  case Administrator
}

In Scala 2 we write

sealed abstract class Role extends Product with Serializable
case object Normal extends Role
case object Moderator extends Role
case object Administrator extends Role

The cases within a role don’t hold any data, so we used a case object in the Scala 2 code.

We defined a user as a screen name, an email address, a password, and a role. In both Scala 3 and Scala 2 this becomes

final case class User(
  screenName: String,
  emailAddress: String,
  password: String,
  role: Role
)

I’ve used String to represent most of the data within a User, but in real code we might want to define distinct types for each field.

2.2.3.2 Paths

We defined a path as a sequence of actions of a virtual pen. The possible actions are straight lines, Bezier curves, or movement that doesn’t result in visible output. A straight line has an end point (the starting point is implicit), a Bezier curve has two control points and an end point, and a move has an end point.

This has a straightforward translation to Scala. We can represent paths as the following in both Scala 3 and Scala 2.

final case class Path(actions: Seq[Action])

An action is a logical or, so we have different representations in Scala 3 and Scala 2. In Scala 3 we’d write

enum Action {
  case Line(end: Point)
  case Curve(cp1: Point, cp2: Point, end: Point)
  case Move(end: Point)
}

where Point is a suitable representation of a two-dimensional point.

In Scala 2 we have to go with the more verbose

sealed abstract class Action extends Product with Serializable 
final case class Line(end: Point) extends Action
final case class Curve(cp1: Point, cp2: Point, end: Point)
  extends Action
final case class Move(end: Point) extends Action

2.2.4 Representing ADTs in Scala 3

We’ve seen that the Scala 3 representation of algebraic data types, using enum, is more compact than the Scala 2 representation. However the Scala 2 representation is still available. Should you ever use the Scala 2 representation in Scala 3? There are a few cases where you may want to:

Exercise: Tree

To gain a bit of practice defining algebraic data types, code the following description in Scala (your choice of version, or do both.)

A Tree with elements of type A is:

We can directly translate this binary tree into Scala. Here’s the Scala 3 version.

enum Tree[A] {
  case Leaf(value: A)
  case Node(left: Tree[A], right: Tree[A])
}

In the Scala 2 encoding we write

sealed abstract class Tree[A] extends Product with Serializable
final case class Leaf[A](value: A) extends Tree[A]
final case class Node[A](left: Tree[A], right: Tree[A]) extends Tree[A]

2.3 Structural Recursion

Structural recursion is our second programming strategy. Algebraic data types tell us how to create data given a certain structure. Structural recursion tells us how to transform an algebraic data types into any other type. Given an algebraic data type, the transformation can be implemented using structural recursion.

As with algebraic data types, there is distinction between the concept of structural recursion and the implementation in Scala. This is more obvious because there are two ways to implement structural recursion in Scala: via pattern matching or via dynamic dispatch. We’ll look at each in turn.

2.3.1 Pattern Matching

I’m assuming you’re familiar with pattern matching in Scala, so I’ll only talk about how to implement structural recursion using pattern matching. Remember there are two kinds of algebraic data types: sum types (logical ors) and product types (logical ands). We have corresponding rules for structural recursion implemented using pattern matching:

  1. For each branch in a sum type we have a distinct case in the pattern match; and
  2. Each case corresponds to a product type with the pattern written in the usual way.

Let’s see this in code, using an example ADT that includes both sum and product types:

which we represent (in Scala 3) as

enum A {
  case B(d: D, e: E)
  case C(f: F, g: G)
}

Following the rules above means a structural recursion would look like

anA match {
  case B(d, e) => ???
  case C(f, g) => ???
}

The ??? bits are problem specific, and we cannot give a general solution for them. However we’ll soon see strategies to help create them.

2.3.2 The Recursion in Structural Recursion

At this point you might be wondering where the recursion in structural recursion comes from. This is an additional rule for recursion: whenever the data is recursive the method is recursive in the same place.

Let’s see this in action for a real data type.

We can define a list with elements of type A as:

This is exactly the definition of List in the standard library. Notice it’s an algebraic data type as it consists of sums and products. It is also recursive: in the pair case the tail is itself a list.

We can directly translate this to code, using the strategy for algebraic data types we saw previously. In Scala 3 we write

enum MyList[A] {
  case Empty()
  case Pair(head: A, tail: MyList[A])
}

Let’s implement map for MyList. We start with the method skeleton specifying just the name and types.

enum MyList[A] {
  case Empty()
  case Pair(head: A, tail: MyList[A])
  
  def map[B](f: A => B): MyList[B] = 
    ???
}

Our first step is to recognize that map can be written using a structural recursion. MyList is an algebraic data type, map is transforming this algebraic data type, and therefore structural recursion is applicable. We now apply the structural recursion strategy, giving us

enum MyList[A] {
  case Empty()
  case Pair(head: A, tail: MyList[A])
  
  def map[B](f: A => B): MyList[B] = 
    this match {
      case Empty() => ???
      case Pair(head, tail) => ???
    }
}

I forgot the recursion rule! The data is recursive in the tail of Pair, so map is recursive there as well.

enum MyList[A] {
  case Empty()
  case Pair(head: A, tail: MyList[A])
  
  def map[B](f: A => B): MyList[B] = 
    this match {
      case Empty() => ???
      case Pair(head, tail) => ??? tail.map(f)
    }
}

I left the ??? to indicate that we haven’t finished with that case.

Now we can move on to the problem specific parts. Here we have three strategies to help us:

  1. reasoning independently by case;
  2. assuming the recursion is correct; and
  3. following the types

The first two are specific to structural recursion, while the final one is a general strategy we can use in many situations. Let’s briefly discuss each and then see how they apply to our example.

The first strategy is relatively simple: when we consider the problem specific code on the right hand side of a pattern matching case, we can ignore the code in any other pattern match cases. So, for example, when considering the case for Empty above we don’t need to worry about the case for Pair, and vice versa.

The next strategy is a little bit more complicated, and has to do with recursion. Remember that the structural recursion strategy tells us where to place any recursive calls. This means we don’t have to think through the recursion. Instead we assume the recursive call will correctly compute what it claims, and only consider how to further process the result of the recursion. The result is guaranteed to be correct so long as we get the non-recursive parts correct.

In the example above we have the recursion tail.map(f). We can assume this correctly computes map on the tail of the list, and we only need to think about what we should do with the remaining data: the head and the result of the recursive call.

It’s this property that allows us to consider cases independently. Recursive calls are the only thing that connect the different cases, and they are given to us by the structural recursion strategy.

Our final strategy is following the types. It can be used in many situations, not just structural recursion, so I consider it a separate strategy. The core idea is to use the information in the types to restrict the possible implementations. We can look at the types of inputs and outputs to help us.

Now let’s use these strategies to finish the implementation of map. We start with

enum MyList[A] {
  case Empty()
  case Pair(head: A, tail: MyList[A])
  
  def map[B](f: A => B): MyList[B] = 
    this match {
      case Empty() => ???
      case Pair(head, tail) => ??? tail.map(f)
    }
}

Our first strategy is to consider the cases independently. Let’s start with the Empty case. There is no recursive call here, so reasoning about recursion doesn’t come into play. Let’s instead use the types. There is no input here other than the Empty case we have already matched, so we cannot use the input types to further restrict the code. Let’s instead consider the output type. We’re trying to create a MyList[B]. There are only two ways to create a MyList[B]: an Empty or a Pair. To create a Pair we need a head of type B, which we don’t have. So we can only use Empty. This is the only possible code we can write. The types are sufficiently restrictive that we cannot write incorrect code for this case.

enum MyList[A] {
  case Empty()
  case Pair(head: A, tail: MyList[A])
  
  def map[B](f: A => B): MyList[B] = 
    this match {
      case Empty() => Empty()
      case Pair(head, tail) => ??? tail.map(f)
    }
}

Now let’s move to the Pair case. We can apply both the structural recursion reasoning strategy and following the types. Let’s use each in turn.

The case for Pair is

case Pair(head, tail) => ??? tail.map(f)

Remember we can consider this independently of the other case. We assume the recursion is correct. This means we only need to think about what we should do with the head, and how we should combine this result with tail.map(f). Let’s now follow the types to finish the code. Our goal is to produce a MyList[B]. We already the following available:

We could return just Empty, matching the case we’ve already written. This has the correct type but we might expect it is not the correct answer because it does not use the result of the recursion, head, or f in any way.

We could return just tail.map(f). This has the correct type but we might expect it is not correct because we don’t use head or f in any way.

We can call f on head, producing a value of type B, and then combine this value and the result of the recursive call using Pair to produce a MyList[B]. This is the correct solution.

enum MyList[A] {
  case Empty()
  case Pair(head: A, tail: MyList[A])
  
  def map[B](f: A => B): MyList[B] = 
    this match {
      case Empty() => Empty()
      case Pair(head, tail) => Pair(f(head), tail.map(f))
    }
}

If you’ve followed this example you’ve hopefully see how we can use the three strategies to systematically find the correct implementation. Notice how we interleaved the recursion strategy and following the types to guide us to a solution for the Pair case. Also note how following the types alone gave us three possible implementations for the Pair case. In this code, and as is usually the case, the solution was the implementation that used all of the available inputs.

2.3.3 Exhaustivity Checking

Remember that algebraic data types are a closed world: they cannot be extended once defined. The Scala compiler can use this to check that we handle all possible cases in a pattern match, so long as we write the pattern match in a way the compiler can work with. This is known as exhaustivity checking.

Here’s a simple example. We start by defining a straight-forward algebraic data type.

// Some of the possible units for lengths in CSS
enum CssLength {
  case Em(value: Double)
  case Rem(value: Double)
  case Pt(value: Double)
}

If we write a pattern match using the structural recursion strategy, the compiler will complain if we’re missing a case.

import CssLength.*

CssLength.Em(2.0) match {
  case Em(value) => value
  case Rem(value) => value
}
// -- [E029] Pattern Match Exhaustivity Warning: ----------------------------------
// 1 |CssLength.Em(2.0) match {
//   |^^^^^^^^^^^^^^^^^
//   |match may not be exhaustive.
//   |
//   |It would fail on pattern case: CssLength.Pt(_)
//   |
//   | longer explanation available when compiling with `-explain`

Exhaustivity checking is incredibly useful. For example, if we add or remove a case from an algebraic data type, the compiler will tell us all the pattern matches that need to be updated.

2.3.4 Dynamic Dispatch

Using dynamic dispatch to implement structural recursion is an implementation technique that may feel more natural to people with a background in object-oriented programming.

The dynamic dispatch approach consists of:

  1. defining an abstract method at the root of the algebraic data types; and
  2. implementing that abstract method at every leaf of the algebraic data type.

This implementation technique is only available if we use the Scala 2 encoding of algebraic data types.

Let’s see it in the MyList example we just looked at. Our first step is to rewrite the definition of MyList to the Scala 2 style.

sealed abstract class MyList[A] extends Product with Serializable
final case class Empty[A]() extends MyList[A]
final case class Pair[A](head: A, tail: MyList[A]) extends MyList[A]

Next we define an abstract method for map on MyList.

sealed abstract class MyList[A] extends Product with Serializable {
  def map[B](f: A => B): MyList[B]
}
final case class Empty[A]() extends MyList[A]
final case class Pair[A](head: A, tail: MyList[A]) extends MyList[A]

Then we implement map on the concrete subtypes Empty and Pair.

sealed abstract class MyList[A] extends Product with Serializable {
  def map[B](f: A => B): MyList[B]
}
final case class Empty[A]() extends MyList[A] {
  def map[B](f: A => B): MyList[B] = 
    Empty()
}
final case class Pair[A](head: A, tail: MyList[A]) extends MyList[A] {
  def map[B](f: A => B): MyList[B] =
    Pair(f(head), tail.map(f))
}

We can use exactly the same strategies we used in the pattern matching case to create this code. The implementation technique is different but the underlying concept is the same.

Given we have two implementation strategies, which should we use? If we’re using enum in Scala 3 we don’t have a choice; we must use pattern matching. In other situations we can choose between the two. I prefer to use pattern matching when I can, as it puts the entire method definition in one place. However, Scala 2 in particular has problems inferring types in some pattern matches. In these situations we can use dynamic dispatch instead. We’ll learn more about this when we look at generalized algebraic data types.

Exercise: Methods for Tree

In a previous exercise we created a Tree algebraic data type:

enum Tree[A] {
  case Leaf(value: A)
  case Node(left: Tree[A], right: Tree[A])
}

Or, in the Scala 2 encoding:

sealed abstract class Tree[A] extends Product with Serializable
final case class Leaf[A](value: A) extends Tree[A]
final case class Node[A](left: Tree[A], right: Tree[A]) extends Tree[A]

Let’s get some practice with structural recursion and write some methods for Tree. Implement

Use whichever you prefer of pattern matching or dynamic dispatch to implement the methods.

I chose to use pattern matching to implement these methods. I’m using the Scala 3 encoding so I have no choice.

I start by creating the method declarations with empty bodies.

enum Tree[A] {
  case Leaf(value: A)
  case Node(left: Tree[A], right: Tree[A])
  
  def size: Int = 
    ???

  def contains(element: A): Boolean =
    ???
    
  def map[B](f: A => B): Tree[B] =
    ???
}

Now these methods all transform an algebraic data type so I can implement them using structural recursion. I write down the structural recursion skeleton for Tree, remembering to apply the recursion rule.

enum Tree[A] {
  case Leaf(value: A)
  case Node(left: Tree[A], right: Tree[A])
  
  def size: Int = 
    this match { 
      case Leaf(value)       => ???
      case Node(left, right) => left.size ??? right.size
    }

  def contains(element: A): Boolean =
    this match { 
      case Leaf(value)       => ???
      case Node(left, right) => left.contains(element) ??? right.contains(element)
    }
    
  def map[B](f: A => B): Tree[B] =
    this match { 
      case Leaf(value)       => ???
      case Node(left, right) => left.map(f) ??? right.map(f)
    }
}

Now I can use the other reasoning techniques to complete the method declarations. Let’s work through size.

def size: Int = 
  this match { 
    case Leaf(value)       => 1
    case Node(left, right) => left.size ??? right.size
  }

I can reason independently by case. The size of a Leaf is, by definition, 1.

def size: Int = 
  this match { 
    case Leaf(value)       => 1
    case Node(left, right) => left.size ??? right.size
  }

Now I can use the rule for reasoning about recursion: I assume the recursive calls successfully compute the size of the left and right children. What is the size then of the combined tree? It must be the sum of the size of the children. With this, I’m done.

def size: Int = 
  this match { 
    case Leaf(value)       => 1
    case Node(left, right) => left.size + right.size
  }

I can use the same process to work through the other two methods, giving me the complete solution shown below.

enum Tree[A] {
  case Leaf(value: A)
  case Node(left: Tree[A], right: Tree[A])
  
  def size: Int = 
    this match { 
      case Leaf(value)       => 1
      case Node(left, right) => left.size + right.size
    }

  def contains(element: A): Boolean =
    this match { 
      case Leaf(value)       => element == value
      case Node(left, right) => left.contains(element) || right.contains(element)
    }
    
  def map[B](f: A => B): Tree[B] =
    this match { 
      case Leaf(value)       => Leaf(f(value))
      case Node(left, right) => Node(left.map(f), right.map(f))
    }
}

2.3.5 Folds as Structural Recursions

Let’s finish by looking at the fold method as an abstraction over structural recursion. If you did the Tree exercise above, you will have noticed that we wrote the same kind of code again and again. Here are the methods we wrote. Notice the left-hand sides of the pattern matches are all the same, and the right-hand sides are very similar.

def size: Int = 
  this match { 
    case Leaf(value)       => 1
    case Node(left, right) => left.size + right.size
  }

def contains(element: A): Boolean =
  this match { 
    case Leaf(value)       => element == value
    case Node(left, right) => left.contains(element) || right.contains(element)
  }
  
def map[B](f: A => B): Tree[B] =
  this match { 
    case Leaf(value)       => Leaf(f(value))
    case Node(left, right) => Node(left.map(f), right.map(f))
  }

This is the point of structural recursion: to recognize and formalize this similarity. However, as programmers we might want to abstract over this repetition. Can we write a method that captures everything that doesn’t change in a structural recursion, and allows the caller to pass arguments for everything that does change? It turns out we can. For any algebraic data type we can define at least one method, called a fold, that captures all the parts of structural recursion that don’t change and allows the caller to specify all the problem specific parts.

Let’s see how this is done using the example of MyList. Recall the definition of MyList is

enum MyList[A] {
  case Empty()
  case Pair(head: A, tail: MyList[A])
}

We know the structural recursion skeleton for MyList is

def doSomething[A](list: MyList[A]) =
  list match {
    case Empty()          => ???
    case Pair(head, tail) => ??? doSomething(tail)
  } 

Implementing fold for MyList means defining a method

def fold[A, B](list: MyList[A]): B =
  list match {
    case Empty() => ???
    case Pair(head, tail) => ??? fold(tail)
  }

where B is the type the caller wants to create.

To complete fold we add method parameters for the problem specific (???) parts. In the case for Empty, we need a value of type B (notice that I’m following the types here).

def fold[A, B](list: MyList[A], empty: B): B =
  list match {
    case Empty() => empty
    case Pair(head, tail) => ??? fold(tail, empty)
  }

For the Pair case, we have the head of type A and the recursion producing a value of type B. This means we need a function to combine these two values.

def foldRight[A, B](list: MyList[A], empty: B, f: (A, B) => B): B =
  list match {
    case Empty() => empty
    case Pair(head, tail) => f(head, foldRight(tail, empty, f))
  }

This is foldRight (and I’ve renamed the method to indicate this). You might have noticed there is another valid solution. Both empty and the recursion produce values of type B. If we follow the types we can come up with

def foldLeft[A,B](list: MyList[A], empty: B, f: (A, B) => B): B =
  list match {
    case Empty() => empty
    case Pair(head, tail) => foldLeft(tail, f(head, empty), f)
  }

which is foldLeft, the tail-recursive variant of fold for a list. (We’ll talk about tail-recursion in a later chapter.)

We can follow the same process for any algebraic data type to create its folds. The rules are:

Returning to MyList, it has:

Exercise: Tree Fold

Implement a fold for Tree defined earlier. There are several different ways to traverse a tree (pre-order, post-order, and in-order). Just choose whichever seems easiest.

I start by add the method declaration without a body.

enum Tree[A] {
  case Leaf(value: A)
  case Node(left: Tree[A], right: Tree[A])
  
  def fold[B]: B =
    ???
}

Next step is to add the structural recursion skeleton.

enum Tree[A] {
  case Leaf(value: A)
  case Node(left: Tree[A], right: Tree[A])
  
  def fold[B]: B =
    this match {
      case Leaf(value)       => ???
      case Node(left, right) => left.fold ??? right.fold
    }
}

Now I follow the types to add the method parameters. For the Leaf case we want a function of type A => B.

enum Tree[A] {
  case Leaf(value: A)
  case Node(left: Tree[A], right: Tree[A])
  
  def fold[B](leaf: A => B): B =
    this match {
      case Leaf(value)       => leaf(value)
      case Node(left, right) => left.fold ??? right.fold
    }
}

For the Node case we want a function that combines the two recursive results, and therefore has type (B, B) => B.

enum Tree[A] {
  case Leaf(value: A)
  case Node(left: Tree[A], right: Tree[A])
  
  def fold[B](leaf: A => B)(node: (B, B) => B): B =
    this match {
      case Leaf(value)       => leaf(value)
      case Node(left, right) => node(left.fold(leaf)(node), right.fold(leaf)(node))
    }
}

Exercise: Using Fold

Prove to yourself that you can replace structural recursion with calls to fold, by redefining size, contains, and map for Tree using only fold.

enum Tree[A] {
  case Leaf(value: A)
  case Node(left: Tree[A], right: Tree[A])
  
  def fold[B](leaf: A => B)(node: (B, B) => B): B =
    this match {
      case Leaf(value)       => leaf(value)
      case Node(left, right) => node(left.fold(leaf)(node), right.fold(leaf)(node))
    }
    
  def size: Int = 
    this.fold(_ => 1)(_ + _)

  def contains(element: A): Boolean =
    this.fold(_ == element)(_ || _)
    
  def map[B](f: A => B): Tree[B] =
    this.fold(v => Leaf(f(v)))((l, r) => Node(l, r))
}

2.4 Structural Corecursion

Structural corecursion is the opposite—more correctly, the dual—of structural recursion. Whereas structural recursion tells us how to take apart an algebraic data type, structural corecursion tells us how to build up, or construct, an algebraic data type. Whereas we can use structural recursion whenever the input of a method or function is an algebraic data type, we can use structural corecursion whenever the output of a method or function is an algebraic data type.

Duality in Functional Programming

Two concepts or structures are duals if one can be translated in a one-to-one fashion to the other. Duality is one of the main themes of this book. By relating concepts as duals we can transfer knowledge from one domain to another.

Duality is often indicated by attaching the co- prefix to one of the structures or concepts. For example, corecursion is the dual of recursion, and sum types, also known as coproducts, are the dual of product types.

Structural recursion works by considering all the possible inputs (which we usually represent as patterns), and then working out what we do with each input case. Structural corecursion works by considering all the possible outputs, which are the constructors of the algebraic data type, and then working out the conditions under which we’d call each constructor.

Let’s return to the list with elements of type A, defined as:

In Scala 3 we write

enum MyList[A] {
  case Empty()
  case Pair(head: A, tail: MyList[A])
}

We can use structural corecursion if we’re writing a method that produces a MyList. A good example is map:

enum MyList[A] {
  case Empty()
  case Pair(head: A, tail: MyList[A])
  
  def map[B](f: A => B): MyList[B] = 
    ???
}

The output of this method is a MyList, which is an algebraic data type. Since we need to construct a MyList we can use structural corecursion. The structural corecursion strategy says we write down all the constructors and then consider the conditions that will cause us to call each constructor. So our starting point is to just write down the two constructors, and put in dummy conditions.

enum MyList[A] {
  case Empty()
  case Pair(head: A, tail: MyList[A])
  
  def map[B](f: A => B): MyList[B] = 
    if ??? then Empty()
    else Pair(???, ???)
}

We can also apply the recursion rule: where the data is recursive so is the method.

enum MyList[A] {
  case Empty()
  case Pair(head: A, tail: MyList[A])
  
  def map[B](f: A => B): MyList[B] = 
    if ??? then Empty()
    else Pair(???, ???.map(f))
}

To complete the left-hand side we can use the strategies we’ve already seen:

In short order we arrive at the correct solution

enum MyList[A] {
  case Empty()
  case Pair(head: A, tail: MyList[A])
  
  def map[B](f: A => B): MyList[B] = 
    this match {
      case Empty() => Empty()
      case Pair(head, tail) => Pair(f(head), tail.map(f))
    }
}

There are a few interesting points here. Firstly, we should acknowledge that map is both a structural recursion and a structural corecursion. This is not always the case. For example, foldLeft and foldRight are not structural corecursions because they are not constrained to only produce an algebraic data type. Secondly, note that when we walked through the process of creating map as a structural recursion we implicitly used the structural corecursion pattern, as part of following the types. We recognised that we were producing a List, that there were two possibilities for producing a List, and then worked out the correct conditions for each case. Formalizing structural corecursion as a separate strategy allows us to be more conscious of where we apply it. Finally, notice how I switched from an if expression to a pattern match expression as we progressed through defining map. This is perfectly fine. Both kinds of expression achieve the same effect. Pattern matching is a little bit safer due to exhaustivity checking. If we wanted to continue using an if we’d have to define a method (for example, isEmpty) that allows us to distinguish an Empty element from a Pair. This method would have to use pattern matching in its implementation, so avoiding pattern matching directly is just pushing it elsewhere.

2.4.1 Unfolds as Structural Corecursion

Just as we could abstract structural recursion as a fold, for any given algebraic data type we can abstract structural corecursion as an unfold. Unfolds are much less commonly used than folds, but they are still a nice tool to have.

Let’s work through the process of deriving unfold, using MyList as our example again.

enum MyList[A] {
  case Empty()
  case Pair(head: A, tail: MyList[A])
}

The corecursion skeleton is

if ??? then MyList.Empty()
else MyList.Pair(???, recursion(???))

Our starting point is writing the skeleton for unfold. It’s a little bit unusual in that I’ve added a parameter seed. This is the information we use to create an element. We’ll need this, but we cannot derive it from our strategies, so I’ve added it in here as a starting assumption.

def unfold[A, B](seed: A): MyList[B] =
  ???

Now we start using our strategies to fill in the missing pieces. I’m using the corecursion skeleton and I’ve applied the recursion rule immediately in the code below, to save a bit of time in the derivation.

def unfold[A, B](seed: A): MyList[B] =
  if ??? then MyList.Empty()
  else MyList.Pair(???, unfold(seed))

We can abstract the condition using a function from A => Boolean.

def unfold[A, B](seed: A, stop: A => Boolean): MyList[B] =
  if stop(seed) then MyList.Empty()
  else MyList.Pair(???, unfold(seed, stop))

Now we need to handle the case for Pair. We have a value of type A (seed), so to create the head element of Pair we can ask for a function A => B

def unfold[A, B](seed: A, stop: A => Boolean, f: A => B): MyList[B] =
  if stop(seed) then MyList.Empty()
  else MyList.Pair(f(seed), unfold(???, stop, f))

Finally we need to update the current value of seed to the next value. That’s a function A => A.

def unfold[A, B](seed: A, stop: A => Boolean, f: A => B, next: A => A): MyList[B] =
  if stop(seed) then MyList.Empty()
  else MyList.Pair(f(seed), unfold(next(seed), stop, f, next))

At this point we’re done. Let’s see that unfold is useful by declaring some other methods in terms of it. We’re going to declare map, which we’ve already seen is a structural corecursion, using unfold. We will also define fill and iterate, which are methods that construct lists and correspond to the methods with the same names on List in the Scala standard library.

To make this easier to work with I’m going to declare unfold as a method on the MyList companion object. I have made a slight tweak to the definition to make type inference work a bit better. In Scala, types inferred for one method parameter cannot be used for other method parameters in the same parameter list. However, types inferred for one method parameter list can be used in subsequent lists. Separating the function parameters from the seed parameter means that the value inferred for A from seed can be used for inference of the function parameters’ input parameters.

I have also declared some destructor methods, which are methods that take apart an algebraic data type. For MyList these are head, tail, and the predicate isEmpty. We’ll talk more about these a bit later.

Here’s our starting point.

enum MyList[A] {
  case Empty()
  case Pair(_head: A, _tail: MyList[A])

  def isEmpty: Boolean =
    this match {
      case Empty() => true
      case _       => false
    }
    
  def head: A =
    this match {
      case Pair(head, _) => head
    }
    
  def tail: MyList[A] =
    this match {
      case Pair(_, tail) => tail
    }
}
object MyList {
  def unfold[A, B](seed: A)(stop: A => Boolean, f: A => B, next: A => A): MyList[B] =
    if stop(seed) then MyList.Empty()
    else MyList.Pair(f(seed), unfold(next(seed))(stop, f, next))
}

Now let’s define the constructors fill and iterate, and map, in terms of unfold. I think the constructors are a bit simpler, so I’ll do those first.

object MyList {
  def unfold[A, B](seed: A)(stop: A => Boolean, f: A => B, next: A => A): MyList[B] =
    if stop(seed) then MyList.Empty()
    else MyList.Pair(f(seed), unfold(next(seed))(stop, f, next))
    
  def fill[A](n: Int)(elem: => A): MyList[A] =
    ???
    
  def iterate[A](start: A, len: Int)(f: A => A): MyList[A] =
    ???
}

Here I’ve just added the method skeletons, which are taken straight from the List documentation. To implement these methods we can use one of two strategies:

Let’s talk about each in turn.

You might have noticed that the parameters to unfold are almost exactly those you need to create a for-loop in a language like Java. A classic for-loop, of the for(i = 0; i < n; i++) kind, has four components:

  1. the initial value of the loop counter;
  2. the stopping condition of the loop;
  3. the statement that advances the counter; and
  4. the body of the loop that uses the counter.

These correspond to the seed, stop, next, and f parameters of unfold respectively.

Loop variants and invariants are the standard way of reasoning about imperative loops. I’m not going to describe them here, as you have probably already learned how to reason about loops (though perhaps not using these terms). Instead I’m going to discuss the second reasoning strategy, which relates writing unfold to something we’ve already discussed: structural recursion.

Our first step is to note that natural numbers (the integers 0 and larger) are conceptually algebraic data types even though the implementation in Scala—using Int—is not. A natural number is either:

It’s the simplest possible algebraic data type that is both a sum and a product type.

Once we see this, we can use the reasoning tools for structural recursion for creating the parameters to unfold. Let’s show how this works with fill. The n parameter tells us how many elements there are in the List we’re creating. The elem parameter creates those elements, and is called once for each element. So our starting point is to consider this as a structural recursion over the natural numbers. We can take n as seed, and stop as the function x => x == 0. These are the standard conditions for a structural recursion over the natural numbers. What about next? Well, the definition of natural numbers tells us we should subtract one in the recursive case, so next becomes x => x - 1. We only need f, and that comes from the definition of how fill is supposed to work. We create the value from elem, so f is just _ => elem

object MyList {
  def unfold[A, B](seed: A)(stop: A => Boolean, f: A => B, next: A => A): MyList[B] =
    if stop(seed) then MyList.Empty()
    else MyList.Pair(f(seed), unfold(next(seed))(stop, f, next))
    
  def fill[A](n: Int)(elem: => A): MyList[A] =
    unfold(n)(_ == 0, _ => elem, _ - 1)
    
  def iterate[A](start: A, len: Int)(f: A => A): MyList[A] =
    ???
}

We should check that our implementation works as intended. We can do this by comparing it to List.fill.

List.fill(5)(1)
// res6: List[Int] = List(1, 1, 1, 1, 1)
MyList.fill(5)(1)
// res7: MyList[Int] = MyList(1, 1, 1, 1, 1)

Here’s a slightly more complex example, using a stateful method to create a list of ascending numbers. First we define the state and method that uses it.

var counter = 0
def getAndInc(): Int = {
  val temp = counter
  counter = counter + 1
  temp 
}

Now we can create it to create lists.

List.fill(5)(getAndInc())
// res8: List[Int] = List(0, 1, 2, 3, 4)
counter = 0
MyList.fill(5)(getAndInc())
// res10: MyList[Int] = MyList(0, 1, 2, 3, 4)

Exercise: Iterate

Implement iterate using the same reasoning as we did for fill. This is slightly more complex than fill as we need to keep two bits of information: the value of the counter and the value of type A.

object MyList {
  def unfold[A, B](seed: A)(stop: A => Boolean, f: A => B, next: A => A): MyList[B] =
    if stop(seed) then MyList.Empty()
    else MyList.Pair(f(seed), unfold(next(seed))(stop, f, next))
    
  def fill[A](n: Int)(elem: => A): MyList[A] =
    unfold(n)(_ == 0)(_ => elem, _ - 1)
    
  def iterate[A](start: A, len: Int)(f: A => A): MyList[A] =
    unfold((len, start)){
      (len, _) => len == 0,
      (_, start) => start,
      (len, start) => (len - 1, f(start))
    }
}

We should check that this works.

List.iterate(0, 5)(x => x - 1)
// res11: List[Int] = List(0, -1, -2, -3, -4)
MyList.iterate(0, 5)(x => x - 1)
// res12: MyList[Int] = MyList(0, -1, -2, -3, -4)

Exercise: Map

Once you’ve completed iterate, try to implement map in terms of unfold. You’ll need to use the destructors to implement it.

def map[B](f: A => B): MyList[B] =
  MyList.unfold(this)(
    _.isEmpty,
    pair => f(pair.head),
    pair => pair.tail
  )
List.iterate(0, 5)(x => x + 1).map(x => x * 2)
// res13: List[Int] = List(0, 2, 4, 6, 8)
MyList.iterate(0, 5)(x => x + 1).map(x => x * 2)
// res14: MyList[Int] = MyList(0, 2, 4, 6, 8)

Now a quick discussion on destructors. The destructors do two things:

  1. distinguish the different cases within a sum type; and
  2. extract elements from each product type.

So for MyList the minimal set of destructors is isEmpty, which distinguishes Empty from Pair, and head and tail. The extractors are partial functions in the conceptual, not Scala, sense; they are only defined for a particular product type and throw an exception if used on a different case. You may have also noticed that the functions we passed to fill are exactly the destructors for natural numbers.

The destructors are another part of the duality between structural recursion and corecursion. Structural recursion is:

Structural corecursion instead is:

One last thing before we leave unfold. If we look at the usual definition of unfold we’ll probably find the following definition.

def unfold[A, B](in: A)(f: A => Option[(A, B)]): List[B]

This is equivalent to the definition we used, but a bit more compact in terms of the interface it presents. We used a more explicit definition that makes the structure of the method clearer.

2.5 The Algebra of Algebraic Data Types

A question that sometimes comes up is where the “algebra” in algebraic data types comes from. I want to talk about this a little bit and show some of the algebraic manipulations that can be done on algebraic data types.

The term algebra is used in the sense of abstract algebra, an area of mathematics. Abstract algebra deals with algebraic structures. An algebraic structure consists of a set of values, operations on that set, and properties that those operations must maintain. An example is the set of integers, the operations addition and multiplication, and the familiar properties of these operations such as associativity, which says that a + (b+c) = (a+b) + c. The abstract in abstract algebra means that it doesn’t deal with concrete values like integers—that would be far too easy to understand—and instead with abstractions with wacky names like semigroup, monoid, and ring. The example of integers above is an instance of a ring. We’ll see a lot more of these soon enough!

Algebraic data types also correspond to the algebraic structure called a ring. A ring has two operations, which are conventionally written + and ×. You’ll perhaps guess that these correspond to sum and product types respectively, and you’d be absolutely correct. What about the properties of these operations? We’ll they are similar to what we know from basic algebra:

So far, so abstract. Let’s make it concrete by looking at actual examples in Scala.

Remember the algebraic data types work with types, so the operations + and × take types as parameters. So Int × String is equivalent to

final case class IntAndString(int: Int, string: String)

We can use tuples to avoid creating lots of names.

type IntAndString = (Int, String)

We can do the same thing for +. Int + String is

enum IntOrString {
  case IsInt(int: Int)
  case IsString(string: String)
}

or just

type IntOrString = Either[Int, String]

Exercise: Identities

Can you work out which Scala type corresponds to the identity 1 for product types?

It’s Unit, because adding Unit to any product doesn’t add any more information. So, Int contains exactly as much information as Int × Unit (written as the tuple (Int, Unit) in Scala).

What about the Scala type corresponding to the identity 0 for sum types?

It’s Nothing, following the same reasoning as products: a case of Nothing adds no further information (and we cannot even create a value with this type.)

What about the distribution law? This allows us to manipulate algebraic data types to form equivalent, but perhaps more useful, representations. Consider this example of a user data type.

final case class Person(name: String, permissions: Permissions)
enum Permissions {
  case User
  case Moderator
}

Written in mathematical notation, this is

Person = String × Permissions Permissions = User + Moderator

Performing substitution gets us

Person = String × (User+Moderator)

Applying distribution results in

Person = (String×User) + (String×Moderator)

which in Scala we can represent as

enum Person {
  case User(name: String)
  case Moderator(name: String)
}

Is this representation more useful? I can’t say without the context of where the data is being used. However I can say that knowing this manipulation is possible, and correct, is useful.

There is a lot more that could be said about algebraic data types, but at this point I feel we’re really getting into the weeds. I’ll finish up with a few pointers to other interesting facts:

2.6 Conclusions

We have covered a lot of material in this chapter. Let’s recap the key points.

Algebraic data types allow us to express data types by combining existing data types with logical and and logical or. A logical and constructs a product type while a logical or constructs a sum type. Algebraic data types are the main way to represent data in Scala.

Structural recursion gives us a skeleton for transforming any given algebraic data type into any other type. Structural recursion can be abstracted into a fold method.

We use several reasoning principles to help us complete the problem specific parts of a structural recursion:

  1. reasoning independently by case;
  2. assuming recursion is correct; and
  3. following the types.

Following the types is a very general strategy that is can be used in many other situations.

Structural corecursion gives us a skeleton for creating any given algebraic data type from any other type. Structural corecursion can be abstracted into an unfold method. When reasoning about structural corecursion we can reason as we would for an imperative loop, or, if the input is an algebraic data type, use the principles for reasoning about structural recursion.

Notice that the two main themes of functional programming—composition and reasoning—are both already apparent. Algebraic data types are compositional: we compose algebraic data types using sum and product. We’ve seen many reasoning principles in this chapter.

I haven’t covered everything there is to know about algebraic data types; I think doing so would be a book in its own right. Below are some references that you might find useful if you want to dig in further, as well as some biographical remarks.

Algebraic data types are standard in introductory material on functional programming. Structural recursion is certainly extremely common in functional programming, but strangely seems to rarely be explicitly defined as I’ve done here. I learned about both from How to Design Programs [Felleisen et al. 2018].

I’m not aware of any approachable yet thorough treatment of either algebraic data types or structural recursion. Both seem to have become assumed background of any researcher in the field of programming languages, and relatively recent work is caked in layers of mathematics and obtuse notation that I find difficult reading. The infamous Functional Programming with Bananas, Lenses, Envelopes and Barbed Wire [Meijer et al. 1991] is an example of such work. I suspect the core ideas of both date back to at least the emergence of computability theory in the 1930s, well before any digital computers existed.

The earliest reference I’ve found to structural recursion is Proving Properties of Programs by Structural Induction [Burstall 1969]. Algebraic data types don’t seem to have been fully developed, along with pattern matching, until NPL in 1977. NPL was quickly followed by the more influential language Hope, which spread the concept to other programming languages.

Corecursion is a bit better documented in the contemporary literature. How to Design Co-Programs [Gibbons 2021] covers the main ideas we have looked at here, while Gibbons and Jones [1998] discusses uses of unfold.

The Derivative of a Regular Type is its Type of One-Hole Contexts [McBride 2001] describes the derivative of algebraic data types.

3 Objects as Codata

In this chapter we will look at codata, the dual of algebraic data types. Algebraic data types focus on how things are constructed. Codata, in contrast, focuses on how things are used. We define codata by specifying the operations that can be performed on the type. This is very similar to the use of interfaces in object-oriented programming, and this is the first reason that we are interested in codata: codata puts object-oriented programming into a coherent conceptual framework with the other strategies we are discussing.

We’re not only interested in codata as a lens to view object-oriented programming. Codata also has properties that algebraic data does not. Codata allows us to create structures with an infinite number of elements, such as a list that never ends or a server loop that runs indefinitely. Codata has a different form of extensibility to algebraic data. Whereas we can easily write new functions that transform algebraic data, we cannot add new cases to the definition of an algebraic data type without changing the existing code. The reverse is true for codata. We can easily create new implementations of codata, but functions that transform codata are limited by the interface the codata defines.

In the previous chapter we saw structural recursion and structural corecursion as strategies to guide us in writing programs using algebraic data types. The same holds for codata. We can use codata forms of structural recursion and corecursion to guide us in writing programs that consume and produce codata respectively.

We’ll begin our exploration of codata by more precisely defining it and seeing some examples. We’ll then talk about representing codata in Scala, and the relationship to object-oriented programming. Once we can create codata, we’ll see how to work with it using structural recursion and corecursion, using an example of an infinite structure. Next we will look at transforming algebraic data to codata, and vice versa. We will finish by examining differences in extensibility.

A quick note about terminology before we proceed. We might expect to use the term algebraic codata for the dual of algebraic data, but conventionally just codata is used. I assume this is because data is usually understood to have a wider meaning than just algebraic data, but codata is not used outside of programming language theory. For simplicity and symmetry, within this chapter I’ll just use the term data to refer to algebraic data types.

3.1 Data and Codata

Data describes what things are, while codata describes what things can do.

We have seen that data is defined in terms of constructors producing elements of the data type. Let’s take a very simple example: a Bool is either True or False. We know we can represent this in Scala as

enum Bool {
  case True
  case False
}

The definition tells us there are two ways to construct an element of type Bool. Furthermore, if we have such an element we can tell exactly which case it is, by using a pattern match for example. Similarly, if the instances themselves hold data, as in List for example, we can always extract all the data within them. Again, we can use pattern matching to achieve this.

Codata, in contrast, is defined in terms of operations we can perform on the elements of the type. These operations are sometimes called destructors (which we’ve already encountered), observations, or eliminators. A common example of codata is a data structures such as a set. We might define the operations on a Set with elements of type A as:

In Scala we could implement this definition as

trait Set[A] {
  
  /** True if this set contains the given element */
  def contains(elt: A): Boolean
  
  /** Construct a new set containing all elements in this set and the given element */
  def insert(elt: A): Set[A]
  
  /** Construct the union of this and that set */
  def union(that: Set[A]): Set[A]
}

This definition does not tell us anything about the internal representation of the elements in the set. It could use a hash table, a tree, or something more exotic. It does, however, tell us what we can do with the set. We know we can take the union but not the intersection, for example.

If you come from the object-oriented world you might recognize the description of codata above as programming to an interface. In some ways codata is just taking concepts from the object-oriented world and presenting them in a way that is consistent with the rest of the functional programming paradigm. However, this does not mean adopting all the features of object-oriented programming. We won’t use state, which is difficult to reason about. We also won’t use implementation inheritance either, for the same reason. In our subset of object-oriented programming we’ll either be defining interfaces (which may have default implementations of some methods) or final classes that implement those interfaces. Interestingly, this subset of object-oriented programming is often recommended by advocates of object-oriented programming.

Let’s now be a little more precise in our definition of codata, which will make the duality between data and codata clearer. Remember the definition of data: it is defined in terms of sums (logical ors) and products (logical ands). We can transform any data into a sum of products. Each product in the sum is a constructor, and the product itself is the parameters that the constructor accepts. Finally, we can think of constructors as functions which take some arbitrary input and produce an element of data. Our end point is a sum of functions from arbitrary input to data.

More abstractly, if we are constructing an element of some data type A we call one of the constructors

Now we’ll turn to codata. Codata is defined as a product of functions, these functions being the destructors. The input to a destructor is always an element of the codata type and possibly some other parameters. The output is usually something that is not of the codata type. Thus constructing an element of some codata type A means defining

This hopefully makes the duality between the two clearer.

Now we understand what codata is, we will turn to representing codata in Scala.

3.2 Codata in Scala

We have already seen an example of codata, which I have repeated below.

trait Set[A] {
  
  def contains(elt: A): Boolean
  
  def insert(elt: A): Set[A]
  
  def union(that: Set[A]): Set[A]
}

The abstract definition of this, which is a product of functions, defines a Set with elements of type A as:

Notice that the first parameter of each function is the type we are defining, Set[A].

The translation to Scala is:

This gives us the Scala representation we started with.

This is only half the story for codata. We also need to actually implement the interface we’ve just defined. There are three approaches we can use:

  1. a final subclass, in the case where we want to name the implementation;
  2. an anonymous subclass; or
  3. more rarely, an object.

Neither final nor anonymous subclasses can be further extended, meaning we cannot create deep inheritance hierarchies. This in turn avoids the difficulties that come from reasoning about deep hierarchies. Using a class rather than a case class means we don’t expose implementation details like constructor arguments.

Some examples are in order. Here’s a simple example of Set, which uses a List to hold the elements in the set.

final class ListSet[A](elements: List[A]) extends Set[A] {

  def contains(elt: A): Boolean =
    elements.contains(elt)

  def insert(elt: A): Set[A] =
    ListSet(elt :: elements)

  def union(that: Set[A]): Set[A] =
    elements.foldLeft(that) { (set, elt) => set.insert(elt) }
}
object ListSet {
  def empty[A]: Set[A] = ListSet(List.empty)
}

This uses the first implementation approach, a final subclass. Where would we use an anonymous subclass? They are most useful when implementing methods that return our codata type. Let’s take union as an example. It returns our codata type, Set, and we could implement it as shown below.

trait Set[A] {
  
  def contains(elt: A): Boolean
  
  def insert(elt: A): Set[A]
  
  def union(that: Set[A]): Set[A] = {
    val self = this
    new Set[A] {
      def contains(elt: A): Boolean =
        self.contains(elt) || that.contains(elt)
        
      def insert(elt: A): Set[A] =
        // Arbitrary choice to insert into self
        self.insert(elt).union(that)
    }
  }
}

This uses an anonymous subclass to implement union on the Set trait, and hence defines the method for all subclasses. I haven’t made the method final so that subclasses can override it with a more efficient implementation. This does open up the danger of implementation inheritance. This is an example of where theory and craft diverge. In theory we never want implementation inheritance, but in practice it can be useful as an optimization.

It can also be useful to implement utility methods defined purely in terms of the destructors. Let’s say we wanted to implement a method containsAll that checks if a Set contains all the elements in an Iterable collection.

def containsAll(elements: Iterable[A]): Boolean

We can implement this purely in terms of contains on Set and forall on Iterable.

trait Set[A] {
  
  def contains(elt: A): Boolean
  
  def insert(elt: A): Set[A]
  
  def union(that: Set[A]): Set[A]
  
  def containsAll(elements: Iterable[A]): Boolean =
    elements.forall(elt => this.contains(elt))
}

Once again we could make this a final method. In this case it’s probably more justified as it’s difficult to imagine a more efficient implementation.

Data and codata are both realized in Scala as variations of the same language features of classes and objects. This means we can define types that have properties of both data and codata. We have actually already done this. When we define data we must define names for the fields within the data, thus defining destructors. This is the same in most languages, which don’t make a hard distinction between data and codata.

Part of the appeal, I think, of classes and objects is that they can express so many conceptually different abstractions with the same language constructs. This gives them a surface appearance of simplicity; it seems we need to learn only one abstraction to solve a huge of number of coding problems. However this apparent simplicity hides real complexity, as this variety of uses forces us to reverse engineer the conceptual intention from the code.

3.3 Structural Recursion and Corecursion for Codata

In this section we’ll build a library for streams, also known as lazy lists. These are the codata equivalent of lists. Whereas a list must have a finite length, streams have an infinite length. We’ll use this example to explore structural recursion and structural corecursion as applied to codata.

Let’s start by reviewing structural recursion and corecursion. The key idea is to use the input or output type, respectively, to drive the process of writing the method. We’ve already seen how this works with data, where we emphasized structural recursion. With codata it’s more often the case that structural corecursion is used. The steps for using structural corecursion are:

  1. recognize the output of the method or function is codata;
  2. write down the skeleton to construct an instance of the codata type, usually using an anonymous subclass; and
  3. fill in the methods, using strategies such as structural recursion or following the types to help.

It’s important that any computation takes places within the methods, and so only runs when the methods are called. Once we start creating streams the importance of this will become clear.

For structural recursion the steps are:

  1. recognize the input of the method or function is codata;
  2. note the codata’s destructors as possible sources of values in writing the method; and
  3. complete the method, using strategies such as following the types or structural corecursion and the methods identified above.

Our first step is to define our stream type. As this is codata, it is defined in terms of its destructors. The destructors that define a Stream of elements of type A are:

Note these are almost the destructors of List. We haven’t defined isEmpty as a destructor because our streams never end and thus this method would always return false. (A lot of real implementations, such as the LazyList in the Scala standard library, do define such a method which allows them to represent finite and infinite lists in the same structure. We’re not doing this for simplicity and because we want to work with codata in its purest form.)

We can translate this to Scala, as we’ve previously seen, giving us

trait Stream[A] {
  def head: A
  def tail: Stream[A]
}

Now we can create an instance of Stream. Let’s create a never-ending stream of ones. We will start with the skeleton below and apply strategies to complete the code.

val ones: Stream[Int] = ???

The first strategy is structural corecursion. We’re returning an instance of codata, so we can insert the skeleton to construct a Stream.

val ones: Stream[Int] =
  new Stream[Int] {
    def head: Int = ???
    def tail: Stream[Int] = ???
  }

Here I’ve used the anonymous subclass approach, so I can just write all the code in one place.

The next step is to fill in the method bodies. The first method, head, is trivial. The answer is 1 by definition.

val ones: Stream[Int] =
  new Stream[Int] {
    def head: Int = 1
    def tail: Stream[Int] = ???
  }

It’s not so obvious what to do with tail. We want to return a Stream[Int] so we could apply structural corecursion again.

val ones: Stream[Int] =
  new Stream[Int] {
    def head: Int = 1
    def tail: Stream[Int] =
      new Stream[Int] {
        def head: Int = 1
        def tail: Stream[Int] = ???
      }
  }

This approach doesn’t seem like it’s going to work. We’ll have to write this out an infinite number of times to correctly implement the method, which might be a problem.

Instead we can follow the types. We need to return a Stream[Int]. We have one in scope: ones. This is exactly the Stream we need to return: the infinite stream of ones!

val ones: Stream[Int] =
  new Stream[Int] {
    def head: Int = 1
    def tail: Stream[Int] = ones
  }

You might be alarmed to see the circular reference to ones in tail. This works because it is within a method, and so is only evaluated when that method is called. This delaying of evaluation is what allows us to represent an infinite number of elements, as we only ever evaluate a finite portion of them. This is a core difference from data, which is fully evaluated when it is constructed.

Let’s check that our definition of ones does indeed work. We can’t extract all the elements from an infinite Stream (at least, not in finite time) so in general we’ll have to resort to checking a finite sequence of elements.

ones.head
// res0: Int = 1
ones.tail.head
// res1: Int = 1
ones.tail.tail.head
// res2: Int = 1

This all looks correct. We’ll often want to check our implementation in this way, so let’s implement a method, take, to make this easier.

trait Stream[A] {
  def head: A
  def tail: Stream[A]
  
  def take(count: Int): List[A] =
    count match {
      case 0 => Nil
      case n => head :: tail.take(n - 1)
    }
}

We can use either the structural recursion or structural corecursion strategies for data to implement take. Since we’ve already covered these in detail I won’t go through them here. The important point is that take only uses the destructors when interacting with the Stream.

Now we can more easily check our implementations are correct.

ones.take(5)
// res4: List[Int] = List(1, 1, 1, 1, 1)

For our next task we’ll implement map. Implementing a method on Stream allows us to see both structural recursion and corecursion for codata in action. As usual we begin by writing out the method skeleton.

trait Stream[A] {
  def head: A
  def tail: Stream[A]
  
  def map[B](f: A => B): Stream[B] = 
    ???
}

Now we have a choice of strategy to use. Since we haven’t used structural recursion yet, let’s start with that. The input is codata, a Stream, and the structural recursion strategy tells us we should consider using the destructors. Let’s write them down to remind us of them.

trait Stream[A] {
  def head: A
  def tail: Stream[A]
  
  def map[B](f: A => B): Stream[B] = {
    this.head ???
    this.tail ???
  }
}

To make progress we can follow the types or use structural corecursion. Let’s choose corecursion to see another example of it in use.

trait Stream[A] {
  def head: A
  def tail: Stream[A]
  
  def map[B](f: A => B): Stream[B] = {
    this.head ???
    this.tail ???
    
    new Stream[B] {
      def head: B = ???
      def tail: Stream[B] = ???
    }
  }
}

Now we’ve used structural recursion and structural corecursion, a bit of following the types is in order. This quickly arrives at the correct solution.

trait Stream[A] {
  def head: A
  def tail: Stream[A]
  
  def map[B](f: A => B): Stream[B] = {
    val self = this 
    new Stream[B] {
      def head: B = f(self.head)
      def tail: Stream[B] = self.tail.map(f)
    }
  }
}

There are two important points. Firstly, notice how I gave the name self to this. This is so I can access the value inside the new Stream we are creating, where this would be bound to this new Stream. Next, notice that we access self.head and self.tail inside the methods on the new Stream. This maintains the correct semantics of only performing computation when it has been asked for. If we performed the computation outside of the methods that we would do it too early, which is some cases can lead to an infinite loop.

As our final example, let’s return to constructing Stream, and implement the universal constructor unfold. We start with the skeleton for unfold, remembering the seed parameter.

trait Stream[A] {
  def head: A
  def tail: Stream[A]
}
object Stream {
  def unfold[A, B](seed: A): Stream[B] =
    ???
}

It’s natural to apply structural corecursion to make progress.

trait Stream[A] {
  def head: A
  def tail: Stream[A]
}
object Stream {
  def unfold[A, B](seed: A): Stream[B] =
    new Stream[B]{
      def head: B = ???
      def tail: Stream[B] = ???
    }
}

Now we can follow the types, adding parameters as we need them. This gives us the complete method shown below.

trait Stream[A] {
  def head: A
  def tail: Stream[A]
}
object Stream {
  def unfold[A, B](seed: A, f: A => B, next: A => A): Stream[B] =
    new Stream[B]{
      def head: B = 
        f(seed)
      def tail: Stream[B] = 
        unfold(next(seed), f, next)
    }
}

We can use this to implement some interesting streams. Here’s a stream that alternates between 1 and -1.

val alternating = Stream.unfold(
  true, 
  x => if x then 1 else -1, 
  x => !x
)

We can check it works.

alternating.take(5)
// res11: List[Int] = List(1, -1, 1, -1, 1)

Exercise: Stream Combinators

It’s time for you to get some practice with structural recursion and structural corecursion using codata. Implement filter, zip, and scanLeft on Stream. They have the same semantics as the same methods on List, and the signatures shown below.

trait Stream[A] {
  def head: A
  def tail: Stream[A]

  def filter(pred: A => Boolean): Stream[A]
  def zip[B](that: Stream[B]): Stream[(A, B)]
  def scanLeft[B](zero: B)(f: (B, A) => B): Stream[B]
}

For all of these methods I found that structural corecursion was the most natural way to tackle them. You could start with structural recursion, though.

You might be worried about the inefficiency of filter. That’s something we’ll discuss a bit later.

trait Stream[A] {
  def head: A
  def tail: Stream[A]

  def filter(pred: A => Boolean): Stream[A] = {
    val self = this
    new Stream[A] {
      def head: A = {
        def loop(stream: Stream[A]): A =
          if pred(stream.head) then stream.head
          else loop(stream.tail)
          
        loop(self)
      }
      
      def tail: Stream[A] = {
        def loop(stream: Stream[A]): Stream[A] =
          if pred(stream.head) then stream.tail.filter(pred)
          else loop(stream.tail)
          
        loop(self)
      }
    }
  }

  def zip[B](that: Stream[B]): Stream[(A, B)] = {
    val self = this 
    new Stream[(A, B)] {
      def head: (A, B) = (self.head, that.head)
      
      def tail: Stream[(A, B)] =
        self.tail.zip(that.tail)
    }
  }

  def scanLeft[B](zero: B)(f: (B, A) => B): Stream[B] = {
    val self = this
    new Stream[B] {
      def head: B = zero
      
      def tail: Stream[B] =
        self.tail.scanLeft(f(zero, self.head))(f)
    }
  }
}

We can do some neat things with the methods defined above. For example, here is the stream of natural numbers.

val naturals = Stream.ones.scanLeft(0)((b, a) => b + a)

As usual, we should check it works.

naturals.take(5)
// res15: List[Int] = List(0, 1, 2, 3, 4)

We could also define naturals using unfold. More interesting is defining it in terms of itself.

val naturals: Stream[Int] =
  new Stream {
    def head = 1
    def tail = naturals.map(_ + 1)
  }

This might be confusing. If so, spend a bit of time thinking about it. It really does work!

naturals.take(5)
// res17: List[Int] = List(1, 2, 3, 4, 5)

3.3.1 Efficiency and Effects

You may have noticed that our implement recomputes values, possibly many times. A good example is the implementation of filter. This recalculates the head and tail on each call, which could be a very expensive operation.

def filter(pred: A => Boolean): Stream[A] = {
  val self = this
  new Stream[A] {
    def head: A = {
      def loop(stream: Stream[A]): A =
        if pred(stream.head) then stream.head
        else loop(stream.tail)
        
      loop(self)
    }
    
    def tail: Stream[A] = {
      def loop(stream: Stream[A]): Stream[A] =
        if pred(stream.head) then stream.tail.filter(pred)
        else loop(stream.tail)
        
      loop(self)
    }
  }
}

We know that delaying the computation until the method is called is important, because that is how we can handle infinite and self-referential data. However we don’t need to redo this computation on successive calls. We can instead cache the result from the first call and use that next time. Scala makes this easy with lazy val, which is a val that is not computed until its first call. Additionally, Scala’s use of the uniform access principle means we can implement a method with no parameters using a lazy val. Here’s a quick example demonstrating it in use.

def always[A](elt: => A): Stream[A] =
  new Stream[A] {
    lazy val head: A = elt
    lazy val tail: Stream[A] = always(head)
  }
  
val twos = always(2)

As usual we should check our work.

twos.take(5)
// res18: List[Int] = List(2, 2, 2, 2, 2)

We get the same result whether we use a method or a lazy val, because we are assuming that we are only dealing with pure computations that have no dependency on state that might change. In this case a lazy val simply consumes additional space to save on time.

Recomputing a result every time it is needed is known as call by name, while caching the result the first time it is computed is known as call by need. These two different evaluation strategies can be applied to individual values, as we’ve done here, or across an entire programming. Haskell, for example, uses call by need. All values in Haskell are only computed the first time they are need. This is approach is sometimes known as lazy evaluation. Another alternative, called call by value, computes results when they are defined instead of waiting until they are needed. This is the default in Scala.

We can illustrate the difference between call by name and call by need if we use an impure computation. For example, we can define a stream of random numbers. Random number generators depend on some internal state.

Here’s the call by name implementation, using the methods we have already defined.

import scala.util.Random

val randoms: Stream[Double] = 
  Stream.unfold(Random, r => r.nextDouble(), r => r)

Notice that we get different results each time we take a section of the Stream. We would expect these results to be the same.

randoms.take(5)
// res19: List[Double] = List(
//   0.9237544457421732,
//   0.8642186540625004,
//   0.8698989306256442,
//   0.22468019943285522,
//   0.3767615145643115
// )
randoms.take(5)
// res20: List[Double] = List(
//   0.7267811044418853,
//   0.7859444709051769,
//   0.8694361699640497,
//   0.05752032099131699,
//   0.7662521778815536
// )

Now let’s define the same stream in a call by need style, using lazy val.

val randomsByNeed: Stream[Double] =
  new Stream[Double] {
    lazy val head: Double = Random.nextDouble()
    lazy val tail: Stream[Double] = randomsByNeed
  }

This time we get the same result when we take a section, and each number is the same.

randomsByNeed.take(5)
// res21: List[Double] = List(
//   0.5985585836455464,
//   0.5985585836455464,
//   0.5985585836455464,
//   0.5985585836455464,
//   0.5985585836455464
// )
randomsByNeed.take(5)
// res22: List[Double] = List(
//   0.5985585836455464,
//   0.5985585836455464,
//   0.5985585836455464,
//   0.5985585836455464,
//   0.5985585836455464
// )

If we wanted a stream that had a different random number for each element but those numbers were constant, we could redefine unfold to use call by need.

def unfoldByNeed[A, B](seed: A, f: A => B, next: A => A): Stream[B] =
  new Stream[B]{
    lazy val head: B = 
      f(seed)
    lazy val tail: Stream[B] = 
      unfoldByNeed(next(seed), f, next)
  }

Now redefining randomsByNeed using unfoldByNeed gives us the result we are after. First, redefine it.

val randomsByNeed2 =
  unfoldByNeed(Random, r => r.nextDouble(), r => r)

Then check it works.

randomsByNeed2.take(5)
// res23: List[Double] = List(
//   0.21419192838220846,
//   0.9695332496155052,
//   0.5332322197450731,
//   0.25855021551871094,
//   0.864891048874998
// )
randomsByNeed2.take(5)
// res24: List[Double] = List(
//   0.21419192838220846,
//   0.9695332496155052,
//   0.5332322197450731,
//   0.25855021551871094,
//   0.864891048874998
// )

These subtleties are one of the reasons that functional programmers try to avoid using state as far as possible.

3.4 Relating Data and Codata

In this section we’ll explore the relationship between data and codata, and in paritcular converting one to the other. We’ll look at it in two ways: firstly a very surface-level relationship between the two, and then a deep connection via fold.

Remember that data is a sum of products, where the products are constructors and we can view constructors as functions. So we can view data as a sum of functions. Meanwhile, codata is a product of functions. We can easily make a direct correspondence between the functions-as-constructors and the functions in codata. What about the difference between the sum and the product that remains. Well, when we have a product of functions we only call one at any point in our code. So the logical or is in the choice of function to call.

Let’s see how this works with a familiar example of data, List. As an algebraic data type we can define

enum List[A] {
  case Pair(head: A, tail: List[A])
  case Empty()
}

The codata equivalent is

trait List[A] {
  def pair(head: A, tail: List[A]): List[A]
  def empty: List[A]
}

In the codata implementation we are explicitly representing the constructors as methods, and pushing the choice of constructor to the caller. In a few chapters we’ll see a use for this relationship, but for now we’ll leave it and move on.

The other way to view the relationship is a connection via fold. We’ve already learned how to derive the fold for any algebraic data type. For Bool, defined as

enum Bool {
  case True
  case False
}

the fold method is

enum Bool {
  case True
  case False
  
  def fold[A](t: A)(f: A): A =
    this match {
      case True => t
      case False => f
    }
}

We know that fold is universal: we can write any other method in terms of it. It therefore provides a universal destructor and is the key to treating data as codata. In this case the fold is something we use all the time, except we usually call it if.

Here’s the codata version of Bool, with fold renamed to if. (Note that Scala allows us to define methods with the same name as key words, in this case if, but we have to surround them in backticks to use them.)

trait Bool {
  def `if`[A](t: A)(f: A): A
}

Now we can define the two instances of Bool purely as codata.

val True = new Bool {
  def `if`[A](t: A)(f: A): A = t
}

val False = new Bool {
  def `if`[A](t: A)(f: A): A = f
}

Let’s see this in use by defining and in terms of if, and then creating some examples. First the definition of and.

def and(l: Bool, r: Bool): Bool =
  new Bool {
    def `if`[A](t: A)(f: A): A =
      l.`if`(r)(False).`if`(t)(f)
  }

Now the examples. This is simple enough that we can try the entire truth table.

and(True, True).`if`("yes")("no")
// res1: String = "yes"
and(True, False).`if`("yes")("no")
// res2: String = "no"
and(False, True).`if`("yes")("no")
// res3: String = "no"
and(False, False).`if`("yes")("no")
// res4: String = "no"

Exercise: Or and Not

Test your understanding of Bool by implementing or and not in the same way we implemented and above.

We can follow the same structure as and.

def or(l: Bool, r: Bool): Bool =
  new Bool {
    def `if`[A](t: A)(f: A): A =
      l.`if`(True)(r).`if`(t)(f)
  }

def not(b: Bool): Bool =
  new Bool {
    def `if`[A](t: A)(f: A): A =
      b.`if`(False)(True).`if`(t)(f)
  }

Once again, we can test the entire truth table.

or(True, True).`if`("yes")("no")
// res5: String = "yes"
or(True, False).`if`("yes")("no")
// res6: String = "yes"
or(False, True).`if`("yes")("no")
// res7: String = "yes"
or(False, False).`if`("yes")("no")
// res8: String = "no"

not(True).`if`("yes")("no")
// res9: String = "no"
not(False).`if`("yes")("no")
// res10: String = "yes"

Notice that, once again, computation only happens on demand. In this case, nothing happens until if is actually called. Until that point we’re just building up a representation of what we want to happen. This again points to how codata can handle infinite data, by only computing the finite amount required by the actual computation.

The rules here for converting from data to codata are:

  1. On the interface (trait) defining the codata, define a method with the same signature as fold.
  2. Define an implementation of the interface for each product case in the data. The data’s constructor arguments become constructor arguments on the codata classes. If there are no constructor arguments, as in Bool, we can define values instead of classes.
  3. Each implementation implements the case of fold that it corresponds to.

Let’s apply this to a slightly more complex example: List. We’ll start by defining it as data and implementing fold. I’ve chosen to implement foldRight but foldLeft would be just as good.

enum List[A] {
  case Pair(head: A, tail: List[A])
  case Empty()
  
  def foldRight[B](empty: B)(f: (A, B) => B): B =
    this match { 
      case Pair(head, tail) => f(head, tail.foldRight(empty)(f))
      case Empty() => empty
    }
}

Now let’s implement it as codata. We start by defining the interface with the fold method. In this case I’m calling it foldRight as it’s going to exactly mirror the foldRight we just defined.

trait List[A] {
  def foldRight[B](empty: B)(f: (A, B) => B): B
}

Now we define the implementations. There is one for Pair and one for Empty, which are the two cases in data definition of List. Notice that in this case the classes have constructor arguments, which correspond to the constructor arguments on the correspnding product types.

final class Pair[A](head: A, tail: List[A]) extends List[A] {
  def foldRight[B](empty: B)(f: (A, B) => B): B =
    ???
}

final class Empty[A]() extends List[A] {
  def foldRight[B](empty: B)(f: (A, B) => B): B =
    ???
}

I didn’t implement the bodies offoldRight so I could show this as a separate step. The implementation here directly mirrors foldRight on the data implementation, and we can use the same strategies to implement the codata equivalents. That is to say, we can use the recursion rule, reasoning by case, and following the types. I’m going to skip these details as we’ve already gone through them in depth. The final code is shown below.

final class Pair[A](head: A, tail: List[A]) extends List[A] {
  def foldRight[B](empty: B)(f: (A, B) => B): B =
    f(head, tail.foldRight(empty)(f))
}

final class Empty[A]() extends List[A] {
  def foldRight[B](empty: B)(f: (A, B) => B): B =
    empty
}

This code is almost the same as the dynamic dispatch implementation, which again shows the relationship between codata and object-oriented code.

The transformation from data to codata goes under several names: refunctionalization, Church encoding, and Böhm-Berarducci encoding. The latter two terms specifically refer to transformations into the untyped and typed lambda calculus respectively. The lambda calculus is a simple model programming language that contains only functions. We’re going to take a quick detour to show that we can, indeed, encode lists using just functions. This demonstrates that objects and functions have equivalent power.

The starting point is creating a type alias List, which defines a list as a fold. This uses a polymorphic function type, which is new in Scala 3. Inspect the type signature and you’ll see it is the same as foldRight above.

type List[A, B] = (B, (A, B) => B) => B

Now we can define Pair and Empty as functions. The first parameter list is the constructor arguments, and the second parameter list is the parameters for foldRight.

val Empty: [A, B] => () => List[A, B] = 
  [A, B] => () => (empty, f) => empty

val Pair: [A, B] => (A, List[A, B]) => List[A, B] =
  [A, B] => (head: A, tail: List[A, B]) => (empty, f) => 
    f(head, tail(empty, f))

Finally, let’s see an example to show it working. We will first define the list containing 1, 2, 3. Due to a restriction in polymorphic function types, I have to add the useless empty parameter.

val list: [B] => () => List[Int, B] = 
  [B] => () => Pair(1, Pair(2, Pair(3, Empty())))

Now we can compute the sum and product of the elements in this list.

val sum = list()(0, (a, b) => a + b)
// sum: Int = 6
val product = list()(1, (a, b) => a * b)
// product: Int = 6

It works!

The purpose of this little demonstration is to show that functions are just objects (in the codata sense) with a single method. Scala this makes apparent, as functions are objects with an apply method.

We’ve seen that data can be translated to codata. The reverse is also possible: we simply tabulate the results of each possible method call. In other words, the data representation is memoisation, a lookup table, or a cache.

Although we can convert data to codata and vice versa, there are good reasons to choose one over the other. We’ve already seen one reason: with codata we can represent infinite structures. In this next section we’ll see another difference: the extensibility that data and codata permit.

3.5 Data and Codata Extensibility

We have seen that codata can represent types with an infinite number of elements, such as Stream. This is one expressive difference from data, which must always be finite. We’ll now look at another, which is the type of extensibility we get from data and from codata. Together these gives use guidelines to choose between the two.

Firstly, let’s define extensibility. It means the ability to add new features without modifying existing code. (If we allow modification of existing code then any extension becomes trivial.) In particular there are two dimensions along which we can extend code: adding new functions or adding new elements. We will see that data and codata have orthogonal extensibility: it’s easy to add new functions to data but adding new elements is impossible without modifying existing code, while adding new elements to codata is straight-forward but adding new functions is not.

Let’s start with a concrete example of both data and codata. For data we’ll use the familiar List type.

enum List[A] {
  case Empty()
  case Pair(head: A, tail: List[A])
}

For codata, we’ll use Set as our exemplar.

trait Set[A] {
  def contains(elt: A): Boolean
  def insert(elt: A): Set[A]
  def union(that: Set[A]): Set[A]
}

We know there are lots of methods we can define on List. The standard library is full of them! We also know that any method we care to write can be written using structural recursion. Finally, we can write these methods without modifying existing code.

Imagine filter was not defined on List. We can easily implement it as

import List.*

def filter[A](list: List[A], pred: A => Boolean): List[A] = 
  list match {
    case Empty() => Empty()
    case Pair(head, tail) => 
      if pred(head) then Pair(head, filter(tail, pred))
      else filter(tail, pred)
  }

We could even use an extension method to make it appear as a normal method.

extension [A](list: List[A]) {
  def filter(pred: A => Boolean): List[A] = 
    list match {
      case Empty() => Empty()
      case Pair(head, tail) => 
        if pred(head) then Pair(head, tail.filter(pred))
        else tail.filter(pred)
    }
}

This shows we can add new functions to data without issue.

What about adding new elements to data? Perhaps we want to add a special case to optimize single-element lists. This is impossible without changing existing code. By definition, we cannot add a new element to an enum without changing the enum. Adding such a new element would break all existing pattern matches, and so require they all change. So in summary we can add new functions to data, but not new elements.

Now let’s look at codata. This has the opposite extensibility; duality strikes again! In the codata case we can easily add new elements. We simply implement the trait that defines the codata interface. We saw this when we defined, for example, ListSet.

final class ListSet[A](elements: List[A]) extends Set[A] {

  def contains(elt: A): Boolean =
    elements.contains(elt)

  def insert(elt: A): Set[A] =
    ListSet(elt :: elements)

  def union(that: Set[A]): Set[A] =
    elements.foldLeft(that) { (set, elt) => set.insert(elt) }
}
object ListSet {
  def empty[A]: Set[A] = ListSet(List.empty)
}

What about adding new functionality? If the functionality can be defined in terms of existing functionality then we’re ok. We can easily define this functionality, and we can use the extension method trick to make it appear like a built-in. However, if we want to define a function that cannot be expressed in terms of existing functions we are out of luck. Let’s saw we want to define some kind of iterator over the elements of a Set. We might use a LazyList, the standard library’s equivalent of Stream we defined earlier, because we know some sets have an infinite number of elements. Well, we can’t do this without changing the definition of Set, which in turn breaks all existing implementations. We cannot define it in a different way because we don’t know all the possible implementations of Set.

So in summary we can add new elements to codata, but not new functions.

If we tabulate this we clearly see that data and codata have orthogonal extensibility.

Extension Data Codata
Add elements No Yes
Add functions Yes No

This difference in extensibility gives us another rule for choosing between data and codata as an implementation strategy, in addition to the finite vs infinite distinction we saw earlier. If we want extensibilty of functions but not elements we should use data. If we have a fixed interface but an unknown number of possible implementations we should use codata.

You might wonder if we can have both forms of extensibility. Achieving this is called the expression problem. There are various ways to solve the expression problem, and we’ll see one that works particularly well in Scala in a later chapter.

3.6 Exercise: Sets

In this extended exercise we’ll explore the Set interface we have already used in several examples, reproduced below.

trait Set[A] {
  
  /** True if this set contains the given element */
  def contains(elt: A): Boolean
  
  /** Construct a new set containing the given element */
  def insert(elt: A): Set[A]
  
  /** Construct the union of this and that set */
  def union(that: Set[A]): Set[A]
}

We also saw a simple implementation, storing the elements in the set in a List.

final class ListSet[A](elements: List[A]) extends Set[A] {

  def contains(elt: A): Boolean =
    elements.contains(elt)

  def insert(elt: A): Set[A] =
    ListSet(elt :: elements)

  def union(that: Set[A]): Set[A] =
    elements.foldLeft(that) { (set, elt) => set.insert(elt) }
}
object ListSet {
  def empty[A]: Set[A] = ListSet(List.empty)
}

The implementation for union is a bit unsatisfactory; it’s doesn’t use any of our strategies for writing code. We can implement both union and insert in a generic way that works for all sets (in other words, is implemented on the Set trait) and uses the strategies we’ve seen in this chapter. Go ahead and do this.

I used structural corecursion to implement these methods. I decided to name the subclasses, as I think it’s a little bit clearer what’s going on in this case.

trait Set[A] {
  
  def contains(elt: A): Boolean
  
  def insert(elt: A): Set[A] =
    InsertOneSet(elt, this)
  
  def union(that: Set[A]): Set[A] =
    UnionSet(this, that)
}

final class InsertOneSet[A](element: A, source: Set[A]) 
    extends Set[A] {

  def contains(elt: A): Boolean =
    elt == element || source.contains(elt)
}

final class UnionSet[A](first: Set[A], second: Set[A])
    extends Set[A] {

  def contains(elt: A): Boolean =
    first.contains(elt) || second.contains(elt)
}

Your next challenge is to implement Evens, the set of all even integers, which we’ll represent as a Set[Int]. This is an infinite set; we cannot directly enumerate all the elements in this set. (We actually could enumerate all the even elements that are 32-bit Ints, but we don’t want to as this would use excessive amounts of space.)

I implemented Evens using an object. This is possible because all possible instances of this set are the same, so we only need one instance.

object Evens extends Set[Int] {

  def contains(elt: Int): Boolean =
    (elt % 2 == 0)
}

It turns out, perhaps surprisingly, that this works. Let’s define a few sets using Evens and ListSet.

val evensAndOne = Evens.insert(1)
val evensAndOthers = 
  Evens.union(ListSet.empty.insert(1).insert(3))

Now show that they work as expected.

evensAndOne.contains(1)
// res1: Boolean = true
evensAndOthers.contains(1)
// res2: Boolean = true
evensAndOne.contains(2)
// res3: Boolean = true
evensAndOthers.contains(2)
// res4: Boolean = true
evensAndOne.contains(3)
// res5: Boolean = false
evensAndOthers.contains(3)
// res6: Boolean = true

We can generalize this idea to defining sets in terms of indicator functions, which is a function of type A => Boolean, returning returns true if the input belows to the set. Implement IndicatorSet, which is constructed with a single indicator function parameter.

final class IndicatorSet[A](indicator: A => Boolean)
    extends Set[A] {

  def contains(elt: A): Boolean =
    indicator(elt)
}

To test this, let’s define the infinite set of odd integers.

val odds = IndicatorSet[Int](_ % 2 == 1)

Now we’ll show it works as expected.

odds.contains(1)
// res7: Boolean = true
odds.contains(2)
// res8: Boolean = false
odds.contains(3)
// res9: Boolean = true

Taking the union of even and odd integers gives us a set that contains all integers.

val integers = Evens.union(odds)

It has the expected behaviour.

integers.contains(1)
// res10: Boolean = true
integers.contains(2)
// res11: Boolean = true
integers.contains(3)
// res12: Boolean = true

3.7 Conclusions

In this chapter we’ve explored codata, the dual of data. Codata is defined by its interface—what we can do with it—as opposed to data, which is defined by what it is. More formally, codata is a product of destructors, where destructors are functions from the codata type (and, optionally, some other inputs) to some type. By avoiding the elements of object-oriented programming that make it hard to reason about—state and implementation inheritance—codata brings elements of object-oriented programming that accord with the other functional programming strategies. In Scala we define codata as a trait, and implement it as a final class, anonymous subclass, or an object.

We have two strategies for implementing methods using codata: structural corecursion, which we can use when the result is codata, and structural recursion, which we can use when an input is codata. Structural corecursion is usually the more useful of the two, as it gives more structure (pun intended) to the method we are implementing. The reverse is true for data.

We saw that data is connected to codata via fold: any data can instead be implemented as codata with a single destructor that is the fold for that data. The reverse is also: we can enumerate all potential pairs of inputs and outputs of destructors to represent codata as data. However this does not mean that data and codata are equivalent. We have seen many examples of codata representing infinite structures, such as sets of all even numbers and streams of all natural numbers. We have also seen that data and codata offer different forms of extensibility: data makes it easy to add new functions, but adding new elements requires changing existing code, while it is easy to add new elements to codata but we change existing code if we add new functions.

The earliest reference I could find to codata in programming languages is Hagino [1989]. This is much more recent than algebraic data, which I think explains why codata is relatively unknown. There are some excellent recent papers that deal with codata. I highly recommend Codata in Action [Downen et al. 2019], which inspired large portions of this chapter. Exploring Codata: The Relation to Object-Orientation [Sullivan 2019] is also worthwhile. How to Add Laziness to a Strict Language Without Even Being Odd [Wadler et al. 1998] is an older paper that discusses the implementation of streams, and in particular the difference between a not-quite-lazy-enough implementation they label odd and the version we saw, which they call even. These correspond to Stream and LazyList in the Scala standard library respectively. Classical (Co)Recursion: Programming [Downen and Ariola 2021] is an interesting survey of corecursion in different languages, and covers many of the same examples that I used here. Finally, if you really want to get into the weeds of the relationship between data and codata, Beyond Church encoding: Boehm-Berarducci isomorphism of algebraic data types and polymorphic lambda-terms [Kiselyov 2005] is for you.

4 Contextual Abstraction

All but the simplest programs depend on the context in which they run. The number of available CPU cores is an example of context provided by the computer, which a program might adapt to by changing how work is distributed. Other forms of context include configuration read from files and environment variables, and (and we’ll see at lot of this later) values created at compile-time, such as serialization formats, in response to the type of some method parameters.

Scala is one of the few languages that provides features for contextual abstraction, known as implicits in Scala 2 or given instances in Scala 3. In Scala these features are intimately related to types; types are used to select between different available given instances and drive construction of given instances at compile-time.

Most Scala programmers are less confident with the features for contextual abstraction than with other parts of the language, and they are often entirely novel to programmers coming from other languages. Hence this chapter will start by reviewing the abstractions formely known as implicits: given instances and using clauses. We will then look at one of their major uses, type classes3. Type classes allow us to extend existing types with new functionality, without using traditional inheritance, and without altering the original source code. Type classes are the core of Cats, which we will be exploring in the next part of this book.

4.1 The Mechanics of Contextual Abstraction

In section we’ll go through the main Scala language features for contextual abstraction. Once we have a firm understanding of the mechanics of contextual abstraction we’ll move on to their use.

The language features for contextual abstraction have changed name from Scala 2 to Scala 3, but they work in largely the same way. In the table below I show the Scala 3 features, and their Scala 2 equivalents. If you use Scala 2 you’ll find that most of the code works simply by replacing given with implicit val and using with implicit.

Scala 3 Scala 2
given instance implicit value
using clause implicit parameter

Let’s now explain how these language features work.

4.1.1 Using Clauses

We’ll start with using clauses. A using clause is a method parameter list that starts with the using keyword. We use the term context parameters for the parameters in a using clause.

def double(using x: Int) = x + x

The using keyword applies to all parameters in the list, so in add below both x and y are context parameters.

def add(using x: Int, y: Int) = x + y

We can have normal parameter lists, and multiple using clauses, in the same method.

def addAll(x: Int)(using y: Int)(using z: Int): Int =
  x + y + z

We cannot pass parameters to a using clause in the normal way. We must proceed the parameters with the using keyword as shown below.

double(using 1)
// res0: Int = 2
add(using 1, 2)
// res1: Int = 3
addAll(1)(using 2)(using 3)
// res2: Int = 6

However this is not the typical way to pass parameters. In fact we don’t usually explicit pass parameters to using clause at all. We usually use given instances instead, so let’s turn to them.

4.1.2 Given Instances

A given instance is a value that is defined with the given keyword. Here’s a simple example.

given theMagicNumber: Int = 3

We can use a given instance like a normal value.

theMagicNumber * 2

However, it’s more common to use them with a using clause. When we call a method that has a using clause, and we do not explicitly supply values for the context parameters, the compiler will look for given instances of the required type. If it finds a given instance it will automatically use it to complete the method call.

For example, we defined double above with a single Int context parameter. The given instance we just defined, theMagicNumber, also has type Int. So if we call double without providing any value for the context parameter the compiler will provide the value theMagicNumber for us.

double
// res4: Int = 6

The same given instance will be used for multiple parameters in a using clause with the same type, as in add defined above.

add
// res5: Int = 6

The above are the most important points for using clauses and given instances. We’ll now turn to some of the details of their semantics.

4.1.3 Given Scope and Imports

Given instances are usually not explicitly passed to using clauses. Their whole reason for existence is to get the compiler to do this for us. This could make code hard to understand, so we need to be very clear about which given instances are candidates to be supplied to a using clause. In this section we’ll look at the given scope, which is all the places that the compiler will look for given instances, and the special syntax for importing given instances.

The first rule we should know about the given scope is that it starts at the call site, where the method with a using clause is called, not at the definition site where the method is defined. This means the following code does not compile, because the given instance is not in scope at the call site, even though it is in scope at the definition site.

object A {
  given a: Int = 1
  def whichInt(using int: Int): Int = int
}

A.whichInt
// error:
// No given instance of type Int was found for parameter int of method whichInt in object A
// A.whichInt
//   ^^^^^^^^

The second rule, which we have been relying on in all our examples so far, is that the given scope includes the lexical scope at the call site. The lexical scope is where we usually look up the values associated with names (like the names of method parameters or val declarations). This means the following code works, as a is defined in a scope that includes the call site.

object A {
  given a: Int = 1
  
  object B {
    C.whichInt 
  }
  
  object C {
    def whichInt(using int: Int): Int = int
  }
}

However, if there are multiple given instances in the same scope the compiler will not arbitrarily choose one. Instead it fails with an error telling us the choice is ambiguous.

object A {
  given a: Int = 1
  given b: Int = 2
    
  def whichInt(using int: Int): Int = int
    
  whichInt
}
// error:
// Ambiguous given instances: both given instance a in object A and
// given instance b in object A match type Int of parameter int of 
// method whichInt in object A

We can import given instances from other scopes, just like we can import normal declarations, but we must explicitly say we want to import given instances. The following code does not work because we have not explicitly imported the given instances.

object A {
  given a: Int = 1

  def whichInt(using int: Int): Int = int
}
object B {
  import A.*
    
  whichInt
}
// error:
// No given instance of type Int was found for parameter int of method whichInt in object A
// 
// Note: given instance a in object A was not considered because it was not imported with `import given`.
//   whichInt
//           ^

It works when we do explicitly import them using import A.given.

object A {
  given a: Int = 1

  def whichInt(using int: Int): Int = int
}
object B {
  import A.{given, *}
    
  whichInt
}

One final wrinkle: the given scope includes the companion objects of any type involved in the type of the using clause. This is best illustrated with an example. We’ll start by defining a type Sound that represents the sound made by its type variable A, and a method soundOf to access that sound.

trait Sound[A] {
  def sound: String
}

def soundOf[A](using s: Sound[A]): String =
  s.sound

Now we’ll define some given instances. Notice that they are defined on the relevant companion objects.

trait Cat
object Cat {
  given catSound: Sound[Cat] =
    new Sound[Cat]{
      def sound: String = "meow"
    }
}

trait Dog
object Dog {
  given dogSound: Sound[Dog] = 
    new Sound[Dog]{
      def sound: String = "woof"
    }
}

When we call soundOf we don’t have to explicitly bring the instances into scope. They are automatically in the given scope by virtue of being defined on the companion objects of the types we use (Cat and Dog). If we had defined these instances on the Sound companion object they would also be in the given scope; when looking for a Sound[A] both the companion objects of Sound and A are in scope.

soundOf[Cat]
// res12: String = "meow"
soundOf[Dog]
// res13: String = "woof"

We should almost always be defining given instances on companion objects. This simple organization scheme means that users do not have to explicitly import them but can easily find the implementations if they wish to inspect them.

4.1.3.1 Given Instance Priority

Notice that given instance selection is based entirely on types. We don’t even pass any values to soundOf! This means given instances are easiest to use when there is only one instance for each type. In this case we can just put the instances on a relevant companion object and everything works out.

However, this is not always possible (though it’s often an indication of a bad design if it is not). For cases where we need multiple instances for a type, we can use the instance priority rules to select between them. We’ll look at the three most important rules below.

The first rule is that explicitly passing an instance takes priority over everything else.

given a: Int = 1
def whichInt(using int: Int): Int = int
whichInt(using 2)
// res15: Int = 2

The second rule is that instances in the lexical scope take priority over instances in a companion object

trait Sound[A] {
  def sound: String
}
trait Cat
object Cat {
  given catSound: Sound[Cat] =
    new Sound[Cat]{
      def sound: String = "meow"
    }
}

def soundOf[A](using s: Sound[A]): String =
  s.sound
given purr: Sound[Cat]  =
  new Sound[Cat]{
    def sound: String = "purr"
  }

soundOf[Cat]
// res17: String = "purr"

The final rule is that instances in a closer lexical scope take preference over those further away.

{
  given growl: Sound[Cat] =
   new Sound[Cat]{
     def sound: String = "growl"
   }
   
  {
    given mew: Sound[Cat] =
     new Sound[Cat]{
       def sound: String = "mew"
     }
     
    soundOf[Cat]
  }
}
// res18: String = "mew"

We’re now seen most of the details of how given instances and using clauses work. This is a craft level explanation, and it naturally leads to the question: where would use these tools? This is what we’ll address next, where we look at type classes and their implementation in Scala.

4.2 Anatomy of a Type Class

Let’s now look at how type classes are implemented. There are three important components to a type class: the type class itself, which defines an interface, type class instances, which implement the type class for particular types, and the methods that use type classes. The table below shows the language features that correspond to each component.

Type Class Concept Language Feature
Type class trait
Type class instance given instance
Type class use using clause

Let’s see how this works in detail.

4.2.1 The Type Class

A type class is an interface or API that represents some functionality we want implemented. In Scala a type class is represented by a trait with at least one type parameter. For example, we can represent generic “serialize to JSON” behaviour as follows:

// Define a very simple JSON AST
sealed trait Json
final case class JsObject(get: Map[String, Json]) extends Json
final case class JsString(get: String) extends Json
final case class JsNumber(get: Double) extends Json
case object JsNull extends Json

// The "serialize to JSON" behaviour is encoded in this trait
trait JsonWriter[A] {
  def write(value: A): Json
}

JsonWriter is our type class in this example, with Json and its subtypes providing supporting code. When we come to implement instances of JsonWriter, the type parameter A will be the concrete type of data we are writing.

4.2.2 Type Class Instances

The instances of a type class provide implementations of the type class for specific types we care about, which can include types from the Scala standard library and types from our domain model.

In Scala we create type class instances by defining given instances implementing the type class.

object JsonWriterInstances {
  given stringWriter: JsonWriter[String] =
    new JsonWriter[String] {
      def write(value: String): Json =
        JsString(value)
    }
  
  final case class Person(name: String, email: String)
  
  given JsonWriter[Person] with
    def write(value: Person): Json =
      JsObject(Map(
        "name" -> JsString(value.name),
        "email" -> JsString(value.email)
      ))
  
  // etc...
}

In this example we define two type class instances of JsonWriter, one for String and one for Person. The definition for String uses the syntax we saw in the previous section. The definition for Person uses two bits of syntax that are new in Scala 3. Firstly, writing given JsonWriter[Person] creates an anonymous given instance. We declare just the type and don’t need to name the instance. This is fine because we don’t usually need to refer to given instances by name. The second bit of syntax is the use of with to implement a trait directly without having to write out new JsonWriter[Person] and so on.

In a real implementation we’d usually want to define the instances on a companion object: the instance for String on the JsonWriter companion object (because we cannot define it on the String companion object) and the instance for Person on the Person companion object. I haven’t done this here because I would need to redeclare JsonWriter, as a type and it’s companion object must be declared at the same time.

4.2.3 Type Class Use

A type class use is any functionality that requires a type class instance to work. In Scala this means any method that accepts instances of the type class as part of a using clause.

We’re going to look at two patterns of type class usage, which we call interface objects and interface syntax. You’ll find these in Cats and other libraries.

4.2.3.1 Interface Objects

The simplest way of creating an interface that uses a type class is to place methods in a singleton object:

object Json {
  def toJson[A](value: A)(using w: JsonWriter[A]): Json =
    w.write(value)
}

To use this object, we import any type class instances we care about and call the relevant method:

import JsonWriterInstances.{*, given}
Json.toJson(Person("Dave", "dave@example.com"))
// res1: Json = JsObject(
//   get = Map(
//     "name" -> JsString(get = "Dave"),
//     "email" -> JsString(get = "dave@example.com")
//   )
// )

The compiler spots that we’ve called the toJson method without providing the given instances. It tries to fix this by searching for given instances of the relevant types and inserting them at the call site.

4.2.3.2 Interface Syntax

We can alternatively use extension methods to extend existing types with interface methods4. This is sometimes referred to as as syntax for the type class, which is the term used by Cats. Scala 2 has an equivalent to extension methods known as implicit classes.

Here’s an example defining an extension method to add a toJson method to any type for which we have a JsonWriter instance.

object JsonSyntax {
  extension [A](value: A) {
    def toJson(using w: JsonWriter[A]): Json =
      w.write(value)
  }
}

We use interface syntax by importing it alongside the instances for the types we need:

import JsonWriterInstances.given
import JsonSyntax.*
Person("Dave", "dave@example.com").toJson
// res2: Json = JsObject(
//   get = Map(
//     "name" -> JsString(get = "Dave"),
//     "email" -> JsString(get = "dave@example.com")
//   )
// )

Extension Methods on Traits

In Scala 3 we can define extension methods directly on a type class trait. Since we’re defining toJson as just calling write on JsonWriter, we could instead define toJson directly on JsonWriter and avoid creating an separate extension method.

trait JsonWriter[A] {
  extension (value: A) def toJson: Json
}

object JsonWriter {
  given stringWriter: JsonWriter[String] =
    new JsonWriter[String] {
      extension (value: String) 
        def toJson: Json = JsString(value)
    }
  
  // etc...
}

We do not advocate this approach, because of a limitation in how Scala searches for extension methods. The following code fails because Scala only looks within the String companion object for extension methods, and consequently does not find the extension method on the instance in the JsonWriter companion object.

"A string".toJson
// error:
// value toJson is not a member of String
// "A string".toJson
// ^^^^^^^^^^^^^^^^^

This means that users will have to explicitly import at least the instances for the built-in types (for which we cannot modify the companion objects).

import JsonWriter.given

"A string".toJson
// res5: Json = JsString(get = "A string")

For consistency we recommend separating the syntax from the type class instances and always explicitly importing it, rather than requiring explicit imports for only some extension methods.

4.2.3.3 The summon Method

The Scala standard library provides a generic type class interface called summon. Its definition is very simple:

def summon[A](using value: A): A =
  value

We can use summon to summon any value in the given scope. We provide the type we want and summon does the rest:

summon[JsonWriter[String]]
// res6: JsonWriter[String] = repl.MdocSession$MdocApp3$JsonWriter$$anon$7@181fdee9

Most type classes in Cats provide other means to summon instances. However, summon is a good fallback for debugging purposes. We can insert a call to summon within the general flow of our code to ensure the compiler can find an instance of a type class and ensure that there are no ambiguity errors.

4.3 Type Class Composition

So far we’ve seen type classes as a way to get the compiler to pass values to methods. This is nice but it does seem like we’ve introduced a lot of new concepts for a small gain. The real power of type classes lies in the compiler’s ability to combine given instances to construct new given instances. This is known as type class composition.

Type class composition works by a feature of given instances we have not yet seen: given instances can themselves have context parameters. However, before we go into this let’s see a motivational example.

Consider defining a JsonWriter for Option. We would need a JsonWriter[Option[A]] for every A we care about in our application. We could try to brute force the problem by creating a library of given instances:

given optionIntWriter: JsonWriter[Option[Int]] =
  ???

given optionPersonWriter: JsonWriter[Option[Person]] =
  ???

// and so on...

However, this approach clearly doesn’t scale. We end up requiring two given instances for every type A in our application: one for A and one for Option[A].

Fortunately, we can abstract the code for handling Option[A] into a common constructor based on the instance for A:

Here is the same code written out using a parameterized given instance:

given optionWriter[A](using writer: JsonWriter[A]): JsonWriter[Option[A]] =
  new JsonWriter[Option[A]] {
    def write(option: Option[A]): Json =
      option match {
        case Some(aValue) => writer.write(aValue)
        case None         => JsNull
      }
  }

This method constructs a JsonWriter for Option[A] by relying on a context parameter to fill in the A-specific functionality. When the compiler sees an expression like this:

Json.toJson(Option("A string"))

it searches for an given instance JsonWriter[Option[String]]. It finds the given instance for JsonWriter[Option[A]]:

Json.toJson(Option("A string"))(using optionWriter[String])

and recursively searches for a JsonWriter[String] to use as the context parameter to optionWriter:

Json.toJson(Option("A string"))(using optionWriter(using stringWriter))

In this way, given instance resolution becomes a search through the space of possible combinations of given instance, to find a combination that creates a type class instance of the correct overall type.

4.3.1 Type Class Composition in Scala 2

In Scala 2 we can achieve the same effect with an implicit method with implicit parameters. Here’s the Scala 2 equivalent of optionWriter above.

implicit def scala2OptionWriter[A]
    (implicit writer: JsonWriter[A]): JsonWriter[Option[A]] =
  new JsonWriter[Option[A]] {
    def write(option: Option[A]): Json =
      option match {
        case Some(aValue) => writer.write(aValue)
        case None         => JsNull
      }
  }

Make sure you make the method’s parameter implicit! If you don’t, you’ll end up defining an implicit conversion. Implicit conversion is an older programming pattern that is frowned upon in modern Scala code. Fortunately, the compiler will warn you should you do this.

4.4 What Type Classes Are

We’ve have now seen the mechanics of type classes: they are a specific arrangement of trait, given instances, and using clauses. This is a very craft-level explanation. Let’s now raise the level of the explanation with three different views of type classes.

The first view goes back Chapter 3, where we looked at codata. The type class itself—the trait—is an example of codata with the usual advantages of codata (we can easily add implementations) and disadvantages (we cannot easily change the interface). Given instances and using clauses add the ability to chose the codata implementation based on the type of the context parameter and the instances in the given scope, and to compose instances from smaller components.

Raising the level of abstraction again, we can say that type classes allow us to implement functionality (the type class instance) separately from the type to which it applies, so that the implementation only needs to be defined at the point of the use—the call site—not at the point of declaration.

Raising the level again, we can say type classes allow us to implement ad-hoc polymorphism. I find it easiest to understand ad-hoc polymorphism in contrast to parametric polymorphism. Parametric polymorphism is what we get with type parameters, also known as generic types. It allows us to treat all types in a uniform way. For example, the following function calculates the length of any list of an arbitrary type A.

def length[A](list: List[A]): Int =
  list match {
    case Nil => 0
    case x :: xs => 1 + length(xs)
  }

We can implement length because we don’t require any particular functionality from the values of type A that make up the elements of the list. We don’t call any methods on them, and indeed we cannot call any methods on them because we don’t know what concrete type A will be at the point where length is defined5.

Ad-hoc polymorphism allows us to call methods on values with a generic type. The methods we can call are exactly those defined by the type class. For example, we can use the Numeric type class from the standard library to write a method that adds together elements of any type that implements that type class.

import scala.math.Numeric

def add[A](x: A, y: A)(using n: Numeric[A]): A = {
  n.plus(x, y)
}

So parametric polymorphism can be understood as meaning any type, while ad-hoc polymorphism means any type that also implements this functionality. In ad-hoc polymorphism there doesn’t have to be any particular type relationship between the concrete types that implement the functionality of interest. This is in contast to object-oriented style polymorphism (i.e. codata) where all concrete types must be subtypes of the type that defines the functionality of interest.

4.5 Exercise: Display Library

Scala provides a toString method to let us convert any value to a String. This method comes with a few disadvantages:

  1. It is implemented for every type in the language. There are situations where we don’t want to be able to view data. For example, we may want to ensure we don’t log sensitive information, such as passwords, in plain text.

  2. We can’t customize toString for types we don’t control.

Let’s define a Display type class to work around these problems:

  1. Define a type class Display[A] containing a single method display. display should accept a value of type A and return a String.

  2. Create instances of Display for String and Int on the Display companion object.

  3. On the Display companion object create two generic interface methods:

    • display accepts a value of type A and a Display of the corresponding type. It uses the relevant Display to convert the A to a String.

    • print accepts the same parameters as display and returns Unit. It prints the displayed A value to the console using println.

These steps define the three main components of our type class. First we define Display—the type class itself:

trait Display[A] {
  def display(value: A): String
}

Then we define some default instances of Display and package them in the Display companion object:

object Display {
  given stringDisplay: Display[String] with {
    def display(input: String) = input
  }

  given intDisplay: Display[Int] with {
    def display(input: Int) = input.toString
  }
}

Finally we extend the Display companion object to provide a basic interface:

object Display {
  given stringDisplay: Display[String] with {
    def display(input: String) = input
  }

  given intDisplay: Display[Int] with {
    def display(input: Int) = input.toString
  }

  def display[A](input: A)(using p: Display[A]): String =
    p.display(input)

  def print[A](input: A)(using Display[A]): Unit =
    println(display(input))
}

Notice that the Display instance on print is anonymous. This is allowed in Scala 3, and works because we only pass it to display.

4.5.1 Using the Library

The code above forms a general purpose printing library that we can use in multiple applications. Let’s define an “application” now that uses the library.

First we’ll define a data type to represent a well-known type of furry animal:

final case class Cat(name: String, age: Int, color: String)

Next we’ll create an implementation of Display for Cat that returns content in the following format:

NAME is a AGE year-old COLOR cat.

Finally, use the type class on the console or in a short demo app: create a Cat and print it to the console:

// Define a cat:
val cat = Cat(/* ... */)

// Print the cat!

This is a standard use of the type class pattern. First we define custom data type for our application:

final case class Cat(name: String, age: Int, color: String)

Then we define type class instances for the types we care about. These either go into the companion object of Cat or a separate object to act as a namespace:

given catDisplay: Display[Cat] = new Display[Cat] {
  def display(cat: Cat) = {
    val name  = Display.display(cat.name)
    val age   = Display.display(cat.age)
    val color = Display.display(cat.color)
    s"$name is a $age year-old $color cat."
  }
}

Finally, we use the type class by bringing the relevant instances into scope and using interface object/syntax. If we defined the instances in companion objects Scala brings them into scope for us automatically. Otherwise we use an import to access them:

val cat = Cat("Garfield", 41, "ginger and black")
Display.print(cat)
// Garfield is a 41 year-old ginger and black cat.

4.5.2 Better Syntax

Let’s make our printing library easier to use by adding extension methods for its functionality:

  1. Create an object DisplaySyntax.

  2. Define display and print as extension methods on DisplaySyntax.

  3. Use the extension methods to print the example Cat you created in the previous exercise.

First we define DisplaySyntax with the extension methods we want.

object DisplaySyntax {
  extension [A](value: A)(using p: Display[A]) {
    def display: String = p.display(value)
    def print: Unit = Display.print(value)
  }
}

Now we can show everything working by calling print on a Cat.

import DisplaySyntax.*

Cat("Garfield", 41, "ginger and black").print
// Garfield is a 41 year-old ginger and black cat.

We get a compile error if we haven’t defined an instance of Display for the relevant type:

import java.util.Date
new Date().print
// error:
// value print is not a member of java.util.Date.
// An extension method was tried, but could not be fully constructed:
// 
//     repl.MdocSession.MdocApp1.DisplaySyntax.print[java.util.Date](
//       new java.util.Date())(
//       /* missing */summon[repl.MdocSession.MdocApp1.Display[java.util.Date]])
// 
//     failed with:
// 
//         No given instance of type repl.MdocSession.MdocApp1.Display[java.util.Date] was found for parameter p of method print in object DisplaySyntax
// new Date().print
// ^^^^^^^^^^^^^^^^

4.6 Type Classes and Variance

In this section we’ll discuss how variance interacts with type class instance selection. Variance is one of the darker corners of Scala’s type system, so we start by reviewing it before moving on to its interaction with type classes.

4.6.1 Variance

Variance concerns the relationship between an instance defined on a type and its subtypes. For example, if we define a JsonWriter[Option[Int]], will the expression Json.toJson(Some(1)) select this instance? (Remember that Some is a subtype of Option).

We need two concepts to explain variance: type constructors, and subtyping.

Variance applies to any type constructor, which is the F in a type F[A]. So, for example, List, Option, and JsonWriter are all type constructors. A type constructor must have at least one type parameter, and may have more. So Either, with two type parameters, is also a type constructor.

Subtyping is a relationship between types. We say that B is a subtype of A if we can use a value of type B anywhere we expect a value of type A. We may sometimes use the shorthand B <: A to indicate that B is a subtype of A.

Variance concerns the subtyping relationship between types F[A] and F[B], given a subtyping relationship between A and B. If B is a subtype of A then

  1. if F[B] <: F[A] we say F is covariant in A; else
  2. if F[B] >: F[A] we say F is contravariant in A; else
  3. if there is no subtyping relationship between F[B] and F[A] we say F is invariant in A.

When we define a type constructor we can also add variance annotations to its type parameters. For example, we denote covariance with a + symbol:

trait F[+A] // the "+" means "covariant"

If we don’t add a variance annotation, the type parameter is invariant. Let’s now look at covariance, contravariance, and invariance in detail.

4.6.2 Covariance

Covariance means that the type F[B] is a subtype of the type F[A] if B is a subtype of A. This is useful for modelling many types, including collections like List and Option:

trait List[+A]
trait Option[+A]

The covariance of Scala collections allows us to substitute collections of one type with a collection of a subtype in our code. For example, we can use a List[Circle] anywhere we expect a List[Shape] because Circle is a subtype of Shape:

sealed trait Shape
final case class Circle(radius: Double) extends Shape
val circles: List[Circle] = ???
val shapes: List[Shape] = circles

Generally speaking, covariance is used for outputs: data that we can later get out of a container type such as List, or otherwise returned by some method.

4.6.3 Contravariance

What about contravariance? We write contravariant type constructors with a - symbol like this:

trait F[-A]

Perhaps confusingly, contravariance means that the type F[B] is a subtype of F[A] if A is a subtype of B. This is useful for modelling types that represent inputs, like our JsonWriter type class above:

trait JsonWriter[-A] {
  def write(value: A): Json
}

Let’s unpack this a bit further. Remember that variance is all about the ability to substitute one value for another. Consider a scenario where we have two values, one of type Shape and one of type Circle, and two JsonWriters, one for Shape and one for Circle:

val shape: Shape = ???
val circle: Circle = ???

val shapeWriter: JsonWriter[Shape] = ???
val circleWriter: JsonWriter[Circle] = ???
def format[A](value: A, writer: JsonWriter[A]): Json =
  writer.write(value)

Now ask yourself the question: “Which combinations of value and writer can I pass to format?” We can write a Circle with either writer because all Circles are Shapes. Conversely, we can’t write a Shape with circleWriter because not all Shapes are Circles.

This relationship is what we formally model using contravariance. JsonWriter[Shape] is a subtype of JsonWriter[Circle] because Circle is a subtype of Shape. This means we can use shapeWriter anywhere we expect to see a JsonWriter[Circle].

4.6.4 Invariance

Invariance is the easiest situation to describe. It’s what we get when we don’t write a + or - in a type constructor:

trait F[A]

This means the types F[A] and F[B] are never subtypes of one another, no matter what the relationship between A and B. This is the default semantics for Scala type constructors.

4.6.5 Variance and Instance Selection

When the compiler searches for a given instance it looks for one matching the type or subtype. Thus we can use variance annotations to control type class instance selection to some extent.

There are two issues that tend to arise. Let’s imagine we have an algebraic data type like:

enum A {
  case B
  case C
}

The issues are:

  1. Will an instance defined on a supertype be selected if one is available? For example, can we define an instance for A and have it work for values of type B and C?

  2. Will an instance for a subtype be selected in preference to that of a supertype. For instance, if we define an instance for A and B, and we have a value of type B, will the instance for B be selected in preference to A?

It turns out we can’t have both at once. The three choices give us behaviour as follows:

Type Class Variance Invariant Covariant Contravariant
Supertype instance used? No No Yes
More specific type preferred? No Yes No

Let’s see some examples, using the following types to show the subtyping relationship.

trait Animal
trait Cat extends Animal
trait DomesticShorthair extends Cat

Now we’ll define three different type classes for the three types of variance, and define an instance of each for the Cat type.

trait Inv[A] {
  def result: String
}
object Inv {
  given Inv[Cat] with
    def result = "Invariant"
    
  def apply[A](using instance: Inv[A]): String =
    instance.result
}

trait Co[+A] {
  def result: String
}
object Co {
  given Co[Cat] with
    def result = "Covariant"

  def apply[A](using instance: Co[A]): String =
    instance.result
}

trait Contra[-A] {
  def result: String
}
object Contra {
  given Contra[Cat] with
    def result = "Contravariant"

  def apply[A](using instance: Contra[A]): String =
    instance.result
}

Now the cases that work, all of which select the Cat instance. For the invariant case we must ask for exactly the Cat type. For the covariant case we can ask for a supertype of Cat. For contravariance we can ask for a subtype of Cat.

Inv[Cat]
// res1: String = "Invariant"
Co[Animal]
// res2: String = "Covariant"
Co[Cat]
// res3: String = "Covariant"
Contra[DomesticShorthair]
// res4: String = "Contravariant"
Contra[Cat]
// res5: String = "Contravariant"

Now cases that fail. With invariance any type that is not Cat will fail. So the supertype fails

Inv[Animal]
// error: 
// No given instance of type MdocApp0.this.Inv[MdocApp0.this.Animal] was found for parameter instance of method apply in object Inv

as does the subtype.

Inv[DomesticShorthair]
// error: 
// No given instance of type MdocApp0.this.Inv[MdocApp0.this.DomesticShorthair] was found for parameter instance of method apply in object Inv

Covariance fails for any subtype of the type for which the instance is declared.

Co[DomesticShorthair]
// error: 
// No given instance of type MdocApp0.this.Co[MdocApp0.this.DomesticShorthair] was found for parameter instance of method apply in object Co

Contravariance fails for any supertype of the type for which the instance is declared.

Contra[Animal]
// error: 
// No given instance of type MdocApp0.this.Contra[MdocApp0.this.Animal] was found for parameter instance of method apply in object Contra

It’s clear there is no perfect system. The most choice is to use invariant type classes. This allows us to specify more specific instances for subtypes if we want. It does mean that if we have, for example, a value of type Some[Int], our type class instance for Option will not be used. We can solve this problem with a type annotation like Some(1) : Option[Int] or by using “smart constructors” like the Option.apply, Option.empty, some, and none methods we saw in Section 6.3.3.

4.7 Conclusions

In this chapter we took a first look at type classes. We saw the components that make up a type class:

We saw that type classes can be composed from components using type class composition. This is one form of metaprogramming in Scala, where we can get the compiler to do work for us based on our program’s types.

We can view type classes as marrying codata with tools to select and compose implementations based on type. We can also view type classes as shifting implementation from the definition site to the call site. Finally, can see type classes as a mechanism for ad-hoc polymorphism, allowing us to define common functionality for otherwise unrelated types.

Type classes were first described in Kaes [1988] and Wadler and Blott [1989]. Oliveira et al. [2010] details the encoding of type classes in Scala 2, and compares Scala’s and Haskell’s approach to type classes. Note that type classes are not restricted to Haskell and Scala. For examples, Rust’s traits are essentially type classes.

As we have seen, Scala’s support for type classes is based on implicit parameters (known as using clauses in Scala 3). Implicit parameters [Lewis et al. 2000] were motivated by a desire to decompose type classes into smaller orthogonal language features, but they have been shown to be useful for other tasks. Křikava et al. [2019] surveys different uses of implicits in Scala. See Oliveira and Gibbons [2010] for a particularly mind-bending example. We’ll see some of these different uses in later chapters.

Scala 3 has a few language features related to contextual abstraction that we haven’t mentioned in this chapter. Context functions [Odersky et al. 2017] allow functions to have using clauses. They are something the community is still exploring, and well defined use cases have yet to emerge. Generic derivation allows us to write code that generates type classes instances. Although this is extremely useful I think it’s conceptually quite simple and doesn’t warrant space in this book.

5 Reified Interpreters

The interpreter strategy is perhaps the most important in all of functional programming. The central idea is to separate description from action. When we use the interpreter strategy our program consists of two parts: the description, instructions, or program that describes what we want to do, and the interpreter that carries the actions in the description. In this chapter we’ll start exploring the design and implementation of interpreters, focusing on implementations using algebraic data types.

Interpreters arise whenever there is this distinction between description and action. You may think an interpreter is a complex piece requiring a lot of development effort, but I hope to show you this is not the case. You probably already use lots of interpreters in your daily coding without realizing it. For example, consider the code below which is taken from a web framework called Krop

val route =
  Route(
    Request.get(Path.root / "user" / Param.int),
    Response.ok(Entity.text)
  ).handle(userId => s"You asked for the user ${userId.toString}")

This defines a route, which matches GET requests for the path "/user/<int>", and responds with an Ok containing text. This kind of routing library is ubiquitous in web frameworks, is simple to write, and yet contains everything we need for the interpreter strategy.

Interpreters are so important because they are the key to enabling compositionality and reasoning, particularly while allowing effects. For example, imagine implementing a graphics library using the interpreter strategy. A program simply describes what we want to draw on the screen, but critically it does not draw anything. The interpreter takes this description and creates the drawing described by it. We can freely compose descriptions only because they do not carry out any effects. For example, if we have a description that describes a circle, and one for a square, we can compose them by saying we should draw the circle next to the square thereby creating a new description. If we immediately drew pictures there would be nothing to compose with. Similarly, it’s easier to reason about pictures in this system because a program describes exactly what will appear on the screen, and there is no state from prior drawing that we need to worry about.

Throughout this chapter we will explore the interpreter strategy by building a series of interpreters for regular expressions. We’ve chosen to use regular expressions because they are already familiar to many and they are simple to work with. This means we can focus on the details of the interpreter strategy without getting caught up in problem specific details, but we still end up with a realistic and useful result.

We’ll start with a basic implementation strategy that uses algebraic data types and structural recursion. We’ll then look at transformations to turn our interpreter into a version that avoids using the stack and hence avoids the possibility of stack overflow.

5.1 Regular Expressions

We’ll start this case study by briefly describing the usual task for regular expressions—matching text—and then take a more theoretical view. We’ll then move on to implementation.

We most commonly use regular expressions to determine if a string matches a particular pattern. The simplest regular expression is one that matches only one string. In Scala we can create a regular expression by calling the r method on String. Here’s a regular expression that matches exactly the string "Scala".

val regexp = "Scala".r

We can see that it matches only "Scala" and fails if we give it a shorter or longer input.

regexp.matches("Scala")
// res0: Boolean = true
regexp.matches("Sca")
// res1: Boolean = false
regexp.matches("Scalaland")
// res2: Boolean = false

Notice we already have a separation between description and action. The description is the regular expression itself, created by calling the r method, and the action is calling the matches method on the regular expression.

There are some characters that have a special meaning within the String describing a regular expression. For example, the character * matches the preceding character zero or more times.

val regexp = "Scala*".r
regexp.matches("Scal")
// res4: Boolean = true
regexp.matches("Scala")
// res5: Boolean = true
regexp.matches("Scalaaaa")
// res6: Boolean = true

We can also use parentheses to group sequences of characters. For example, if we wanted to match all the strings like "Scala", "Scalala", "Scalalala" and so on, we could use the following regular expression.

val regexp = "Scala(la)*".r

Let’s check it matches what we’re looking for.

regexp.matches("Scala")
// res8: Boolean = true
regexp.matches("Scalalalala")
// res9: Boolean = true

We should also check it fails to match as expected.

regexp.matches("Sca")
// res10: Boolean = false
regexp.matches("Scalal")
// res11: Boolean = false
regexp.matches("Scalaland")
// res12: Boolean = false

That’s all I’m going to say about Scala’s built-in regular expressions. If you’d like to learn more there are many resources online. The JDK documentation is one example, which describes all the features available in the JVM implementation of regular expressions.

Let’s turn to the theoretical description, such as we might find in a textbook. A regular expression is:

  1. the empty regular expression that matches nothing;
  2. a string, which matches exactly that string (including the empty string);
  3. the concatenation of two regular expressions, which matches the first regular expression and then the second;
  4. the union of two regular expressions, which matches if either expression matches; and
  5. the repetition of a regular expression (often known as the Kleene star), which matches zero or more repetitions of the underlying expression.

This kind of description may seem very abstract if you’re not used to it. It is very useful for our purposes because it defines a minimal API that we can easily implement. Let’s walk through the description and see how each part relates to code.

The empty regular expression is defining a constructor with type () => Regexp, which we can simplify to a value of type Regexp. In Scala we put constructors on the companion object, so this tells us we need

object Regexp {
  val empty: Regexp =
    ???
}

The second part tells us we need another constructor, this one with type String => Regexp.

object Regexp {
  val empty: Regexp =
    ???

  def apply(string: String): Regexp =
    ???
}

The other three components all take a regular expression and produce a regular expression. In Scala these will become methods on the Regexp type. Let’s model this as a trait for now, and define these methods.

The first method, the concatenation of two regular expressions, is conventionally called ++ in Scala.

trait Regexp {
  def ++(that: Regexp): Regexp
}

Union is conventionally called orElse.

trait Regexp {
  def ++(that: Regexp): Regexp
  def orElse(that: Regexp): Regexp
}

Repetition we’ll call repeat, and define an alias * that matches how this operation is written in conventional regular expressions.

trait Regexp {
  def ++(that: Regexp): Regexp
  def orElse(that: Regexp): Regexp
  def repeat: Regexp
  def `*`: Regexp = this.repeat
}

We’re missing one thing: a method to actually match our regular expression against some input. Let’s call this method matches.

trait Regexp {
  def ++(that: Regexp): Regexp
  def orElse(that: Regexp): Regexp
  def repeat: Regexp
  def `*`: Regexp = this.repeat
  
  def matches(input: String): Boolean
}

This completes our API. Now we can turn to implementation. We’re going to represent Regexp as an algebraic data type, and each method that returns a Regexp will return an instance of this algebraic data type. What should be the elements that make up the algebraic data type? There will be one element for each method, and the constructor arguments will be exactly the parameters passed to the method including the hidden this parameter for methods on the trait.

Here’s the resulting code.

enum Regexp {
  def ++(that: Regexp): Regexp =
    Append(this, that)

  def orElse(that: Regexp): Regexp =
    OrElse(this, that)

  def repeat: Regexp =
    Repeat(this)

  def `*`: Regexp = this.repeat
  
  def matches(input: String): Boolean =
    ???
  
  case Append(left: Regexp, right: Regexp)
  case OrElse(first: Regexp, second: Regexp)
  case Repeat(source: Regexp)
  case Apply(string: String)
  case Empty
}
object Regexp {
  val empty: Regexp = Empty
  
  def apply(string: String): Regexp =
    Apply(string)
}

A quick note about this. We can think of every method on an object as accepting a hidden parameter that is the object itself. This is this. (If you have used Python, it makes this explicit as the self parameter.) As we consider this to be a parameter to a method call, and our implementation strategy is to capture all the method parameters in a data structure, we must make sure we capture this when it is available. The only case where we don’t capture this is when we are defining a constructor on a companion object.

Notice that we haven’t implemented matches. It doesn’t return a Regexp so we cannot return an element of our algebraic data type. What should we do here? Regexp is an algebraic data type and matches transforms an algebraic data type into a Boolean. Therefore we can use structural recursion! Let’s write out the skeleton, including the recursion rule.

def matches(input: String): Boolean =
  this match {
    case Append(left, right)   => left.matches(???) ??? right.matches(???)
    case OrElse(first, second) => first.matches(???) ??? second.matches(???)
    case Repeat(source)        => source.matches(???) ???
    case Apply(string)         => ???
    case Empty                 => ???
  }

Now we can apply the usual strategies to complete the implementation. Let’s reason independently by case, starting with the case for Empty. This case is trivial as it always fails to match, so we just return false.

def matches(input: String): Boolean =
  this match {
    case Append(left, right)   => left.matches(???) ??? right.matches(???)
    case OrElse(first, second) => first.matches(???) ??? second.matches(???)
    case Repeat(source)        => source.matches(???) ???
    case Apply(string)         => ???
    case Empty                 => false
  }

Let’s move on to the Append case. This should match if the left regular expression matches the start of the input, and the right regular expression matches starting where the left regular expression stopped. This has uncovered a hidden requirement: we need to keep an index into the input that tells us where we should start matching from. Using a nested method is the easiest way to keep around additional information that we need. Here I’ve created a nested method that returns an Option[Int]. The Int is the new index to use, and we return an Option to indicate if the regular expression matched or not.

def matches(input: String): Boolean = {
  def loop(regexp: Regexp, idx: Int): Option[Int] =
    regexp match {
      case Append(left, right) =>
        loop(left, idx).flatMap(idx => loop(right, idx))
      case OrElse(first, second) => 
        loop(first, idx) ??? loop(second, ???)
      case Repeat(source) => 
        loop(source, idx) ???
      case Apply(string) => 
        ???
      case Empty =>
        None
    }

  // Check we matched the entire input
  loop(this, 0).map(idx => idx == input.size).getOrElse(false)
}

Now we can go ahead and complete the implementation.

def matches(input: String): Boolean = {
  def loop(regexp: Regexp, idx: Int): Option[Int] =
    regexp match {
      case Append(left, right) =>
        loop(left, idx).flatMap(i => loop(right, i))
      case OrElse(first, second) => 
        loop(first, idx).orElse(loop(second, idx))
      case Repeat(source) =>
        loop(source, idx)
          .flatMap(i => loop(regexp, i))
          .orElse(Some(idx))
      case Apply(string) =>
        Option.when(input.startsWith(string, idx))(idx + string.size)
    }

  // Check we matched the entire input
  loop(this, 0).map(idx => idx == input.size).getOrElse(false)
}

The implementation for Repeat is a little tricky, so I’ll walk through the code.

case Repeat(source) =>
  loop(source, idx)
    .flatMap(i => loop(regexp, i))
    .orElse(Some(idx))

The first line (loop(source, index)) is seeing if the source regular expression matches. If it does we loop again, but on regexp (which is Repeat(source)), not source. This is because we want to repeat an indefinite number of times. If we looped on source we would only try twice. Remember that failing to match is still a success; repeat matches zero or more times. This condition is handled by the orElse clause.

We should test that our implementation works.

Here’s the example regular expression we started the chapter with.

val regexp = Regexp("Sca") ++ Regexp("la") ++ Regexp("la").repeat

Here are cases that should succeed.

regexp.matches("Scala")
// res14: Boolean = true
regexp.matches("Scalalalala")
// res15: Boolean = true

Here are cases that should fail.

regexp.matches("Sca")
// res16: Boolean = false
regexp.matches("Scalal")
// res17: Boolean = false
regexp.matches("Scalaland")
// res18: Boolean = false

Success! At this point we could add many extensions to our library. For example, regular expressions usually have a method (by convention denoted +) that matches one or more times, and one that matches zero or once (usually denoted ?). These are both conveniences we can build on our existing API. However, our goal at the moment is to fully understand interpreters and the implementation technique we’ve used here. So in the next section we’ll discuss these in detail.

Regular Expression Semantics

Our regular expression implementation handles union differently to Scala’s built-in regular expressions. Look at the following example comparing the two.

val r1 = "(z|zxy)ab".r
val r2 = Regexp("z").orElse(Regexp("zxy")) ++ Regexp("ab")
r1.matches("zxyab")
// res19: Boolean = true
r2.matches("zxyab")
// res20: Boolean = false

The reason for this difference is that our implementation commits to the first branch in a union that successfully matches some of the input, regardless of how that affects later matching. We should instead try both branches, but doing so makes the implementation more complex. The semantics of regular expressions are not essential to what we’re trying to do here; we’re just using them as an example to motivate the programming strategies we’re learning. I decided the extra complexity of implementing union in the usual way outweighed the benefits, and so kept the simpler implementation. Don’t worry, we’ll see how to do it properly in the next chapter!

5.2 Interpreters and Reification

There are two different programming strategies at play in the regular expression code we’ve just written:

  1. the interpreter strategy; and
  2. the interpreter’s implementation strategy of reification.

Remember the essence of the interpreter strategy is to separate description and action. Therefore, whenever we use the interpreter strategy we need at least two things: a description and an interpreter. Descriptions are programs; things that we want to happen. The interpreter runs the programs, carrying out the actions described within them.

In the regular expression example, a Regexp value is a program. It is a description of a pattern we are looking for within a String. The matches method is an interpreter. It carries out the instructions in the description, checking the pattern matches the entire input. We could have other interpreters, such as one that matches if at least some part of the input matches the pattern.

5.2.1 The Structure of Interpreters

All uses of the interpreter strategy have a particular structure to their methods. There are three different kinds of methods:

  1. constructors, or introduction forms, with type A => Program. Here A is any type that isn’t a program, and Program is the type of programs. Constructors conventionally live on the Program companion object in Scala. We see that apply is a constructor of Regexp. It has type String => Regexp, which matches the pattern A => Program for a constructor. The other constructor, empty, is just a value of type Regexp. This is equivalent to a method with type () => Regexp and so it also matches the pattern for a constructor.

  2. combinators have at least one program input and a program output. The type is similar to Program => Program but there are often additional parameters. All of ++, orElse, and repeat are combinators in our regular expression example. They all have a Regexp input (the this parameter) and produce a Regexp. Some of them have additional parameters, such as ++ or orElse. For both these methods the single additional parameter is a Regexp, but it is not the case that additional parameters to a combinator must be of the program type. Conventionally these methods live on the Program type.

  3. destructors, interpreters, or elimination forms, have type Program => A. In our regular expression example we have a single interpreter, matches, but we could easily add more. For example, we often want to extract elements from the input or find a match at any location in the input.

This structure is often called an algebra or combinator library in the functional programming world. When we talk about constructors and destructors in an algebra we’re talking at a more abstract level then when we talk about constructors and destructors on algebraic data types. A constructor of an algebra is an abstract concept, at the theory level in my taxonomy, that we can choose to concretely implement at the craft level with the constructor of an algebraic data type. There are other possible implementations. We’ll see one later.

5.2.2 Implementing Interpreters with Reification

Now that we understand the components of an interpreter we can talk more clearly about the implementation strategy we used. We used a strategy called reification, defunctionalization, deep embedding, or an initial algebra.

Reification, in an abstract sense, means to make concrete what is abstract. Concretely, reification in the programming sense means to turn methods or functions into data. When using reification in the interpreter strategy we reify all the components that produce the Program type. This means reifying constructors and combinators.

Here are the rules for reification:

  1. We define some type, which we’ll call Program, to represent programs.
  2. We implement Program as an algebraic data type.
  3. All constructors and combinators become product types within the Program algebraic data type.
  4. Each product type holds exactly the parameters to the constructor or combinator, including the this parameter for combinators.

Once we’ve defined the Program algebraic data type, the interpreter becomes a structural recursion on Program.

Exercise: Arithmetic

Now it’s your turn to practice using reification. Your task is to implement an interpreter for arithmetic expressions. An expression is:

Reify this description as a type Expression.

The trick here is to recognize how the textual description relates to code, and to apply reification correctly.

enum Expression {
  case Literal(value: Double)
  case Addition(left: Expression, right: Expression)
  case Subtraction(left: Expression, right: Expression)
  case Multiplication(left: Expression, right: Expression)
  case Division(left: Expression, right: Expression)
}
object Expression {
  def apply(value: Double): Expression =
    Literal(value)
}

Now implement an interpreter eval that produces a Double. This interpreter should interpret the expression using the usual rules of arithmetic.

Our interpreter is a structural recursion.

enum Expression {
  case Literal(value: Double)
  case Addition(left: Expression, right: Expression)
  case Subtraction(left: Expression, right: Expression)
  case Multiplication(left: Expression, right: Expression)
  case Division(left: Expression, right: Expression)
  
  def eval: Double =
    this match {
      case Literal(value)              => value
      case Addition(left, right)       => left.eval + right.eval
      case Subtraction(left, right)    => left.eval - right.eval
      case Multiplication(left, right) => left.eval * right.eval
      case Division(left, right)       => left.eval / right.eval
    }
}
object Expression {
  def apply(value: Double): Expression =
    Literal(value)
}

Add methods +, - and so on that make your system a bit nicer to use. Then write some expressions and show that it works as expected.

Here’s the complete code.

enum Expression {
  case Literal(value: Double)
  case Addition(left: Expression, right: Expression)
  case Subtraction(left: Expression, right: Expression)
  case Multiplication(left: Expression, right: Expression)
  case Division(left: Expression, right: Expression)

  def +(that: Expression): Expression =
    Addition(this, that)

  def -(that: Expression): Expression =
    Subtraction(this, that)

  def *(that: Expression): Expression =
    Multiplication(this, that)

  def /(that: Expression): Expression =
    Division(this, that)

  def eval: Double =
    this match {
      case Literal(value)              => value
      case Addition(left, right)       => left.eval + right.eval
      case Subtraction(left, right)    => left.eval - right.eval
      case Multiplication(left, right) => left.eval * right.eval
      case Division(left, right)       => left.eval / right.eval
    }
}
object Expression {
  def apply(value: Double): Expression =
    Literal(value)
}

Here’s an example showing use, and that the code is correct.

val fortyTwo = ((Expression(15.0) + Expression(5.0)) * Expression(2.0) + Expression(2.0)) / Expression(1.0)
fortyTwo.eval
// res2: Double = 42.0

5.3 Tail Recursive Interpreters

Structural recursion, as we have written it, uses the stack. This is not often a problem, but particularly deep recursions can lead to the stack running out of space. A solution is to write a tail recursive program. A tail recursive program does not need to use any stack space, and so is sometimes known as stack safe. Any program can be turned into a tail recursive version, which does not use the stack and therefore cannot run out of stack space.

The Call Stack

Method and function calls are usually implemented using an area of memory known as the call stack, or just the stack for short. Every method or function call uses a small amount of memory on the stack, called a stack frame. When the method or function returns, this memory is freed and becomes available for future calls to use.

A large number of method calls, without corresponding returns, can require more stack frames than the stack can accommodate. When there is no more memory available on the stack we say we have overflowed the stack. In Scala a StackOverflowError is raised when this happens.

In this section we will discuss tail recursion, converting programs to tail recursive form, and limitations and workarounds for the Scala’s runtimes.

5.3.1 The Problem of Stack Safety

Let’s start by seeing the problem. In Scala we can create a repeated String using the * method.

"a" * 4
// res0: String = "aaaa"

We can match such a String with a regular expression and repeat.

Regexp("a").repeat.matches("a" * 4)
// res1: Boolean = true

However, if we make the input very long the interpreter will fail with a stack overflow exception.

Regexp("a").repeat.matches("a" * 20000)
// java.lang.StackOverflowError

This is because the interpreter calls loop for each instance of a repeat, without returning. However, all is not lost. We can rewrite the interpreter in a way that consumes a fixed amount of stack space, and therefore match input that is as large as we like.

5.3.2 Tail Calls and Tail Position

Our starting point is tail calls. A tail call is a method call that does not take any additional stack space. Only method calls that are in tail position are candidates to be turned into tail calls. Even then, runtime limitations mean that not all calls in tail position will be converted to tail calls.

A method call in tail position is a call that immediately returns the value returned by the call. Let’s see an example. Below are two versions of a method to calculate the sum of the integers from 0 to count.

def isntTailRecursive(count: Int): Int =
  count match {
    case 0 => 0
    case n => n + isntTailRecursive(n - 1)
  }

def isTailRecursive(count: Int): Int = {
  def loop(count: Int, accum: Int): Int =
    count match {
      case 0 => accum
      case n => loop(n - 1, accum + n)
    }
    
  loop(count, 0)
}

The method call to isntTailRecursive in

case n => n + isntTailRecursive(n - 1)

is not in tail position, because the value returned by the call is then used in the addition. However, the call to loop in

case n => loop(n - 1, accum + n)

is in tail position because the value returned by the call to loop is itself immediately returned. Similarly, the call to loop in

loop(count, 0)

is also in tail position.

A method call in tail position is a candidate to be turned into a tail call. Limitations of Scala’s runtimes mean that not all calls in tail position can be made tail calls. Currently, only calls from a method to itself that are also in tail position will be converted to tail calls. This means

case n => loop(n - 1, accum + n)

is converted to a tail call, because loop is calling itself. However, the call

loop(count, 0)

is not converted to a tail call, because the call is from isTailRecursive to loop. This will not cause issues with stack consumption, however, because this call only happens once.

Runtimes and Tail Calls

Scala supports three different platforms: the JVM, Javascript via Scala.js, and native code via Scala Native. Each platform provides what is known as a runtime, which is code that supports our Scala code when it is running. The garbage collector, for example, is part of the runtime.

At the time of writing none of Scala’s runtimes support full tail calls. However, there is reason to think this may change in the future. Project Loom should eventually add support for tail calls to the JVM. Scala Native is likely to support tail calls soon, as part of other work to implement continuations. Tail calls have been part of the Javascript specification for a long time, but remain unimplemented by the majority of Javascript runtimes. However, WebAssembly does support tail calls and will probably replace compiling Scala to Javascript in the medium term.

We can ask the Scala compiler to check that all self calls are in tail position by adding the @tailrec annotation to a method. The code will fail to compile if any calls from the method to itself are not in tail position.

import scala.annotation.tailrec

@tailrec
def isntTailRecursive(count: Int): Int =
  count match {
    case 0 => 0
    case n => n + isntTailRecursive(n - 1)
  }
// error:
// Cannot rewrite recursive call: it is not in tail position
//     case n => n + isntTailRecursive(n - 1)
//                   ^^^^^^^^^^^^^^^^^^^^^^^^

We can check the tail recursive version is truly tail recursive by passing it a very large input. The non-tail recursive version crashes.

isntTailRecursive(100000)
// java.lang.StackOverflowError

The tail recursive version runs just fine.

isTailRecursive(100000)
// res4: Int = 705082704

5.3.3 Continuation-Passing Style

Now that we know about tail calls, how do we convert the regular expression interpreter to use them? Any program can be converted to an equivalent program with all calls in tail position. This conversion is known as continuation-passing style or CPS for short. Our first step to understanding CPS is to understand continuations.

A continuation is an encapsulation of “what happens next”. Let’s return to our Regexp example. Here’s the full code for reference.

enum Regexp {
  def ++(that: Regexp): Regexp =
    Append(this, that)

  def orElse(that: Regexp): Regexp =
    OrElse(this, that)

  def repeat: Regexp =
    Repeat(this)

  def `*` : Regexp = this.repeat

  def matches(input: String): Boolean = {
    def loop(regexp: Regexp, idx: Int): Option[Int] =
      regexp match {
        case Append(left, right) =>
          loop(left, idx).flatMap(i => loop(right, i))
        case OrElse(first, second) =>
          loop(first, idx).orElse(loop(second, idx))
        case Repeat(source) =>
          loop(source, idx)
            .flatMap(i => loop(regexp, i))
            .orElse(Some(idx))
        case Apply(string) =>
          Option.when(input.startsWith(string, idx))(idx + string.size)
        case Empty =>
          None
      }

    // Check we matched the entire input
    loop(this, 0).map(idx => idx == input.size).getOrElse(false)
  }

  case Append(left: Regexp, right: Regexp)
  case OrElse(first: Regexp, second: Regexp)
  case Repeat(source: Regexp)
  case Apply(string: String)
  case Empty
}
object Regexp {
  val empty: Regexp = Empty

  def apply(string: String): Regexp =
    Apply(string)
}

Let’s consider the case for Append in matches.

case Append(left, right) =>
  loop(left, idx).flatMap(i => loop(right, i))

What happens next when we call loop(left, idx)? Let’s give the name result to the value returned by the call to loop. The answer is we run result.flatMap(i => loop(right, i)). We can represent this as a function, to which we pass result:

(result: Option[Int]) => result.flatMap(i => loop(right, i))

This is exactly the continuation, reified as a value.

As is often the case, there is a distinction between the concept and the representation. The concept of continuations always exists in code. A continuation means “what happens next”. In other words, it is the program’s control flow. There is always some concept of control flow, even if it is just “the program halts”. We can represent continuations as functions in code. This transforms the abstract concept of continuations into concrete values in our program, and hence reifies them.

Now that we know about continuations, and their reification as functions, we can move on to continuation-passing style. In CPS we, as the name suggests, pass around continuations. Specifically, each function or method takes an extra parameter that is a continuation. Instead of returning a value it calls that continuation with the value. This is another example of duality, in this case between returning a value and calling a continuation.

Let’s see how this works. We’ll start with a simple example written in the normal style, also known as direct style.

(1 + 2) * 3
// res5: Int = 9

To rewrite this in CPS style we need to create replacements for + and * with the extra continuation parameter.

type Continuation = Int => Int

def add(x: Int, y: Int, k: Continuation) = k(x + y)
def mul(x: Int, y: Int, k: Continuation) = k(x * y)

Now we can rewrite our example in CPS. (1 + 2) becomes add(1, 2, k), but what is k, the continuation? What we do next is multiply the result by 3. Thus the continuation is a => mul(a, 3, k2). What is the next continuation, k2? Here the program finishes, so we just return the value with the identity continuation b => b. Put it all together and we get

add(1, 2, a => mul(a, 3, b => b))
// res6: Int = 9

Notice that every continuation call is in tail position in the CPS code. This means that code written in CPS can potentially consume no stack space.

Now we can return to the interpreter loop for Regexp. We are going to CPS it, so we need to add an extra parameter for the continuation. In this case the contination accepts and returns the result type of loop: Option[Int].

def matches(input: String): Boolean = {
  // Define a type alias so we can easily write continuations
  type Continuation = Option[Int] => Option[Int]

  def loop(regexp: Regexp, idx: Int, cont: Continuation): Option[Int] =
  // etc...
}

Now we go through each case and convert it to CPS. Each continuation we construct must call cont as its final step. This is tedious and a bit error-prone, so good tests are helpful.

def matches(input: String): Boolean = {
  // Define a type alias so we can easily write continuations
  type Continuation = Option[Int] => Option[Int]

  def loop(
      regexp: Regexp,
      idx: Int,
      cont: Continuation
  ): Option[Int] =
    regexp match {
      case Append(left, right) =>
        val k: Continuation = _ match {
          case None    => cont(None)
          case Some(i) => loop(right, i, cont)
        }
        loop(left, idx, k)

      case OrElse(first, second) =>
        val k: Continuation = _ match {
          case None => loop(second, idx, cont)
          case some => cont(some)
        }
        loop(first, idx, k)

      case Repeat(source) =>
        val k: Continuation =
          _ match {
            case None    => cont(Some(idx))
            case Some(i) => loop(regexp, i, cont)
          }
        loop(source, idx, k)

      case Apply(string) =>
        cont(Option.when(input.startsWith(string, idx))(idx + string.size))
        
      case Empty =>
        cont(None)
    }

  // Check we matched the entire input
  loop(this, 0, identity).map(idx => idx == input.size).getOrElse(false)
}

Every call in this interpreter loop is in tail position. However Scala cannot convert these to tail calls because the calls go from loop to a continuation and vice versa. To make the interpreter fully stack safe we need to add trampolining.

Exercise: CPS Arithmetic

In a previous exercise we wrote an interpreter for arithmetic expressions. Your task now is to CPS this interpreter. For reference, the definition of an arithmetic expression is:

The continuations have a slightly different structure to the regular expression example. In the regular expression example, all the information needs by a continuation is either found in the parameter to the continuation (the index) or values extracted via pattern matching. In the arithmetic code we need values from previous continuations that are not passed as parameters. This is to compute binary operations like additions. The solution is to capture these values within the environment of the closure that represents the continuation.

type Continuation = Double => Double

enum Expression {
  case Literal(value: Double)
  case Addition(left: Expression, right: Expression)
  case Subtraction(left: Expression, right: Expression)
  case Multiplication(left: Expression, right: Expression)
  case Division(left: Expression, right: Expression)

  def eval: Double = {
    def loop(expr: Expression, cont: Continuation): Double =
      expr match {
        case Literal(value) => cont(value)
        case Addition(left, right) =>
          loop(left, l => loop(right, r => cont(l + r)))
        case Subtraction(left, right) =>
          loop(left, l => loop(right, r => cont(l - r)))
        case Multiplication(left, right) =>
          loop(left, l => loop(right, r => cont(l * r)))
        case Division(left, right) =>
          loop(left, l => loop(right, r => cont(l / r)))
      }

    loop(this, identity)
  }
  
  def +(that: Expression): Expression =
    Addition(this, that)

  def -(that: Expression): Expression =
    Subtraction(this, that)

  def *(that: Expression): Expression =
    Multiplication(this, that)

  def /(that: Expression): Expression =
    Division(this, that)
}
object Expression {
  def apply(value: Double): Expression =
    Literal(value)
}

5.3.4 Trampolining

Earlier we said that CPS utilizes the duality between function calls and returns: instead of returning a value we call a function with a value. This allows us to transform our code so it only has calls in tail positions. However, we still have a problem with stack safety. Scala’s runtimes don’t support full tail calls, so calls from a continuation to loop or from loop to a continuation will use a stack frame. We can use this same duality to avoid using the stack by, instead of making a call, returning a value that reifies the call we want to make. This idea is the core of trampolining. Let’s see it in action, which will help clear up what exactly this all means.

Our first step is to reify all the method calls made by the interpreter loop and the continuations. There are three cases: calls to loop, calls to a continuation, and, to avoid an infinite loop, the case when we’re done.

type Continuation = Option[Int] => Call

enum Call {
  case Loop(regexp: Regexp, index: Int, continuation: Continuation)
  case Continue(index: Option[Int], continuation: Continuation)
  case Done(index: Option[Int])
}

Now we update loop to return instances of Call instead of making the calls directly.

def loop(regexp: Regexp, idx: Int, cont: Continuation): Call =
  regexp match {
    case Append(left, right) =>
      val k: Continuation = _ match {
        case None    => Call.Continue(None, cont)
        case Some(i) => Call.Loop(right, i, cont)
      }
      Call.Loop(left, idx, k)

    case OrElse(first, second) =>
      val k: Continuation = _ match {
        case None => Call.Loop(second, idx, cont)
        case some => Call.Continue(some, cont)
      }
      Call.Loop(first, idx, k)

    case Repeat(source) =>
      val k: Continuation =
        _ match {
          case None    => Call.Continue(Some(idx), cont)
          case Some(i) => Call.Loop(regexp, i, cont)
        }
      Call.Loop(source, idx, k)

    case Apply(string) =>
      Call.Continue(
        Option.when(input.startsWith(string, idx))(idx + string.size),
        cont
      )

    case Empty =>
      Call.Continue(None, cont)
  }

This gives us an interpreter loop that returns values instead of making calls, and so does not consume stack space. However, we need to actually make these calls at some point, and doing this is the job of the trampoline. The trampoline is simply a tail recursive loop that makes calls until it reaches Done.

def trampoline(next: Call): Option[Int] =
  next match {
    case Call.Loop(regexp, index, continuation) =>
      trampoline(loop(regexp, index, continuation))
    case Call.Continue(index, continuation) =>
      trampoline(continuation(index))
    case Call.Done(index) => index
  }

Now every call has a corresponding return, so the stack usage is limited. Our interpreter can handle input of any size, up to the limits of available memory.

Here’s the complete code for reference.

// Define a type alias so we can easily write continuations
type Continuation = Option[Int] => Call

enum Call {
  case Loop(regexp: Regexp, index: Int, continuation: Continuation)
  case Continue(index: Option[Int], continuation: Continuation)
  case Done(index: Option[Int])
}

enum Regexp {
  def ++(that: Regexp): Regexp =
    Append(this, that)

  def orElse(that: Regexp): Regexp =
    OrElse(this, that)

  def repeat: Regexp =
    Repeat(this)

  def `*` : Regexp = this.repeat

  def matches(input: String): Boolean = {
    def loop(regexp: Regexp, idx: Int, cont: Continuation): Call =
      regexp match {
        case Append(left, right) =>
          val k: Continuation = _ match {
            case None    => Call.Continue(None, cont)
            case Some(i) => Call.Loop(right, i, cont)
          }
          Call.Loop(left, idx, k)

        case OrElse(first, second) =>
          val k: Continuation = _ match {
            case None => Call.Loop(second, idx, cont)
            case some => Call.Continue(some, cont)
          }
          Call.Loop(first, idx, k)

        case Repeat(source) =>
          val k: Continuation =
            _ match {
              case None    => Call.Continue(Some(idx), cont)
              case Some(i) => Call.Loop(regexp, i, cont)
            }
          Call.Loop(source, idx, k)

        case Apply(string) =>
          Call.Continue(
            Option.when(input.startsWith(string, idx))(idx + string.size),
            cont
          )

        case Empty =>
          Call.Continue(None, cont)
      }

    def trampoline(next: Call): Option[Int] =
      next match {
        case Call.Loop(regexp, index, continuation) =>
          trampoline(loop(regexp, index, continuation))
        case Call.Continue(index, continuation) =>
          trampoline(continuation(index))
        case Call.Done(index) => index
      }

    // Check we matched the entire input
    trampoline(loop(this, 0, opt => Call.Done(opt)))
      .map(idx => idx == input.size)
      .getOrElse(false)
  }

  case Append(left: Regexp, right: Regexp)
  case OrElse(first: Regexp, second: Regexp)
  case Repeat(source: Regexp)
  case Apply(string: String)
  case Empty
}
object Regexp {
  val empty: Regexp = Empty

  def apply(string: String): Regexp =
    Apply(string)
}

Exericse: Trampolined Arithmetic

Convert the CPSed arithmetic interpreter we wrote earlier to a trampolined version.

The process to produce this code is very similar to the regular expression example. We just identify all the different types of calls (which are the same as the regular expression example) and reify them.

type Continuation = Double => Call

enum Call {
  case Continue(value: Double, k: Continuation)
  case Loop(expr: Expression, k: Continuation)
  case Done(result: Double)
}

enum Expression {
  case Literal(value: Double)
  case Addition(left: Expression, right: Expression)
  case Subtraction(left: Expression, right: Expression)
  case Multiplication(left: Expression, right: Expression)
  case Division(left: Expression, right: Expression)

  def eval: Double = {
    def loop(expr: Expression, cont: Continuation): Call =
      expr match {
        case Literal(value) => Call.Continue(value, cont)
        case Addition(left, right) =>
          Call.Loop(
            left,
            l => Call.Loop(right, r => Call.Continue(l + r, cont))
          )
        case Subtraction(left, right) =>
          Call.Loop(
            left,
            l => Call.Loop(right, r => Call.Continue(l - r, cont))
          )
        case Multiplication(left, right) =>
          Call.Loop(
            left,
            l => Call.Loop(right, r => Call.Continue(l * r, cont))
          )
        case Division(left, right) =>
          Call.Loop(
            left,
            l => Call.Loop(right, r => Call.Continue(l / r, cont))
          )
      }

    def trampoline(call: Call): Double =
      call match {
        case Call.Continue(value, k) => trampoline(k(value))
        case Call.Loop(expr, k)      => trampoline(loop(expr, k))
        case Call.Done(result)       => result
      }

    trampoline(loop(this, x => Call.Done(x)))
  }

  def +(that: Expression): Expression =
    Addition(this, that)

  def -(that: Expression): Expression =
    Subtraction(this, that)

  def *(that: Expression): Expression =
    Multiplication(this, that)

  def /(that: Expression): Expression =
    Division(this, that)
}
object Expression {
  def apply(value: Double): Expression =
    Literal(value)
}

5.3.5 When Tail Recursion is Easy

Doing a full CPS conversion and trampoline can be quite involved. Some methods can made tail recursive without so large a change. Remember these examples we looked at earlier?

def isntTailRecursive(count: Int): Int =
  count match {
    case 0 => 0
    case n => n + isntTailRecursive(n - 1)
  }

def isTailRecursive(count: Int): Int = {
  def loop(count: Int, accum: Int): Int =
    count match {
      case 0 => accum
      case n => loop(n - 1, accum + n)
    }
    
  loop(count, 0)
}

The tail recursive version doesn’t seem to involve the complexity of CPS. How can we relate this to what we’ve just learned, and when can we avoid the work of CPS and trampolining?

Let’s use substitution to show how the stack is used by each method, for a small value of count.

isntTailRecursive(2)
// expands to
(2 match {
  case 0 => 0
  case n => n + isntTailRecursive(n - 1)
})
// expands to
(2 + isntTailRecursive(1))
// expands to
(2 + (1 match {
        case 0 => 0
        case n => n + isntTailRecursive(n - 1)
      }))
// expands to
(2 + (1 + isntTailRecursive(n - 1)))
// expands to
(2 + (1 + (0 match {
             case 0 => 0
             case n => n + isntTailRecursive(n - 1)
           })))
// expands to
(2 + (1 + (0)))
// expands to
3

Here each set of brackets indicates a new method call and hence a stack frame allocation.

Now let’s do the same for isTailRecursive.

isTailRecursive(2)
// expands to
(loop(2, 0))
// expands to
(2 match {
   case 0 => 0
   case n => loop(n - 1, 0 + n)
 })
// expands to
(loop(1, 2))
// call to loop is a tail call, so no stack frame is allocated 
// expands to
(1 match {
   case 0 => 2
   case n => loop(n - 1, 2 + n)
 })
// expands to
(loop(0, 3))
// call to loop is a tail call, so no stack frame is allocated 
// expands to
(0 match {
   case 0 => 3
   case n => loop(n - 1, 3 + n)
 })
// expands to
(3)
// expands to
3

The non-tail recursive function computes the result (2 + (1 + (0))) If we look closely, we’ll see that the tail recursive version computes (((2) + 1) + 0), which simply accumulates the result in the reverse order. This works because addition is associative, meaning (a + b) + c == a + (b + c). This is our first criteria for using the “easy” method for converting to a tail recursive form: the operation that accumulates results must be associative.

This doesn’t explain, though, how we come to realize that addition is the correct operation to use. The second criteria is that we don’t need any memory beyond the partial result calculated from the data we’ve already seen. Some implications of this are that we can stop at any time and have a usable result, and that we are only applying a single operation to the data. This is not the case in the regular expression example. For example, we have the following code in the Append case:

case Append(left, right) =>
  loop(left, idx).flatMap(i => loop(right, i))

To compute the result for the Append we need to compute and combine results from both left and right. So when we have computed the result for right we need to remember both the result from left and that we’re combining the two results using the rule for Append rather than, say, OrElse. It’s remembering this that is exactly what the continuation does, and what stops us from using the easy method we saw when summing the elements of a list.

So, in summary, if we are applying only a single associative operation to data we can use the simple method for writing a tail recursive method:

  1. define an structurally recursive loop with an additional parameter that is the partial result or accumulator;
  2. in the base cases return the accumulator; and
  3. in the recursive cases update the accumulator and call the loop in tail position.

You might be wondering how we handle tree-shaped data with this technique. One consequence of an associative operation is that we can transform any sequence of operations into a list-shaped sequence. If, for example, we have an expression tree that suggests we should call operations in the order (1 + 2) + (3 + 4) (where I’m using + to indicate the operation) we can rewrite that to (((1 + 2) + 3) + 4) via associativity. So we can transform our tree into a list and then apply the recipe above.

5.4 Conclusions

In this chapter we’ve discussed why we might want to build interpreters, and seen techniques for building them. To recap, the core of the interpreter strategy is a separation between description and action. The description is the program, and the interpreter is the action that carries out the program. This separation is allows for composition of programs, and managing effects by delaying them till the time the program is run. We sometimes call this structure an algebra, with constructs and combinators defining programs and destructors defining interpreters. Although the name of the strategy focuses on the interpreter, the design of the program is just as important as it is the user interface through which the programmer interacts with the system.

Our starting implementation strategy is reification of the algebra’s constructors and compositional methods as an algebraic data type. The interpreter is then a structural recursion over this ADT. We saw that the straightforward implementation is not stack-safe, and which caused us to introduction the idea of tail recursion and continuations. We reified continuations as functions, and saw that we can convert any program into continuation-passing style which has every method call in tail position. Due to Scala runtime limitations not all calls in tail position can be converted to tail calls, so we reified calls and returns into data structures used by a recursive loop called a trampoline. Underlying all these strategies in the concept of duality. We have seen a duality between functions and data, which we utilize in reification, and a duality between calling functions and returning data, which we use in continuations and trampolines.

Stack-safe interpreters are important in many situations, but the code is harder to read than the basic structural recursion. In some contexts a basic interpreter may be just fine. It’s unlikely to run out of stack space when evaluating a straightforward expression tree, as in the arithmetic example. The depth of such a tree grows logarithmically with the number of elements, so only extremely large trees will have sufficient depth that stack safety becomes relevant. However, in the regular expression example the stack consumption is determined not by the depth of the regular expression tree, but by the length of the input being matched. In this situation stack safety is more important. There may still be other constraints that allow a simpler implementation. For example, if we know the library will only used in situations where inputs were guaranteed to be small. As always, only use coding techniques where they make sense.

These ideas are classics in programming language theory. Definitional Interpreters for Higher-Order Programming Languages [Reynolds 1972] details defunctionalization, a limited form of reification and continuation passing style. (If you want to read this paper, I suggest the re-typeset version from 1998, which is much more readable than the original typewriter version.) These ideas are expanded on in Defunctionalization at Work [Danvy and Nielsen 2001]. Continuation-Passing Style, Defunctionalization, Accumulations, and Associativity [Gibbons 2022] is a very readable and elegant paper that highlights the importance of associativity in these transformations.

{#sec:part:two}

In this part of the book we move on to type classes. We looked at the implementation of type classes in Chapter 4. Our focus here is on a handful of specific type classes, that are both very useful for day-to-day programming tasks and as conceptual models that can drive program design. In this part we’ll be looking more at their use for day-to-day programming, while the case studies will focus on their role in design.

In Chapter 6 we introduce the Cats library. Cats provides implementation of the type classes we’re interested in, and so it saves a lot of time and typing to use it.

TODO: complete description

6 Using Cats

In this Chapter we’ll learn how to use the Cats library. Cats provides two main things: type classes and their instances, and some useful data structures. Our focus will mostly be on the type classes, though we will touch on the data structures where appropriate.

6.1 Quick Start

The easiest, and recommended, way to use Cats is to add the following imports:

import cats.*
import cats.syntax.all.*

The first import adds all the type classes (and makes their instances available, as they are found in the companion objects.) The second import adds the syntax helpers, which makes the type classes easier to work with. Note we don’t need to import cats.{*, given} as, at the time of writing, Cats is written in Scala 2 style (using implicits) and these are imported by the wildcard import.

If we want use some of Cats’ datastructures, we also need to add

import cats.data.*

6.2 Using Cats

Let’s now see how we work with Cats, using cats.Show as an example.

Show is Cats’ equivalent of the Display type class we defined in Section 4.5. It provides a mechanism for producing developer-friendly console output without using toString. Here’s an abbreviated definition:

package cats

trait Show[A] {
  def show(value: A): String
}

The easiest way to use Show is with the wildcard import above. However, we can also import Show directly from the cats package:

import cats.Show

The companion object of every Cats type class has an apply method that locates an instance for any type we specify:

val showInt = Show.apply[Int]

Once we have an instance we can call methods on it.

showInt.show(42)
// res0: String = "42"

More common, however, is to use the syntax or extension methods, which we imported with import cats.syntax.all.*. In the case of Show, an extension method show is defined.

42.show
// res1: String = "42"

If, for some reason, we wanted just the syntax for show, we could import cats.syntax.show.

import cats.syntax.show.* // for show

6.2.1 Defining Custom Instances

We can define an instance of Show simply by implementing the trait for a given type:

import java.util.Date

given dateShow: Show[Date] with 
  def show(date: Date): String =
    s"${date.getTime}ms since the epoch."
new Date().show
// res2: String = "1747131114870ms since the epoch."

However, Cats also provides a couple of convenient methods to simplify the process. There are two construction methods on the companion object of Show that we can use to define instances for our own types:

object Show {
  // Convert a function to a `Show` instance:
  def show[A](f: A => String): Show[A] =
    ???

  // Create a `Show` instance from a `toString` method:
  def fromToString[A]: Show[A] =
    ???
}

These allow us to quickly construct instances with less ceremony than defining them from scratch:

given dateShow: Show[Date] =
  Show.show(date => s"${date.getTime}ms since the epoch.")

As you can see, the code using construction methods is much terser than the code without. Many type classes in Cats provide helper methods like these for constructing instances, either from scratch or by transforming existing instances for other types.

6.2.1.1 Exercise: Cat Show

Re-implement the Cat application from Section 4.5.1 using Show instead of Display.

Using this data type to represent a well-known type of furry animal:

final case class Cat(name: String, age: Int, color: String)

create an implementation of Display for Cat that returns content in the following format:

NAME is a AGE year-old COLOR cat.

Then use the type class on the console or in a short demo app: create a Cat and print it to the console:

// Define a cat:
val cat = Cat(/* ... */)

// Print the cat!

First let’s import everything we need from Cats.

import cats.*
import cats.syntax.all.*

Our definition of Cat remains the same:

final case class Cat(name: String, age: Int, color: String)

In the companion object we replace our Display instance with an instance of Show using one of the definition helpers discussed above:

given catShow: Show[Cat] = Show.show[Cat] { cat =>
  val name  = cat.name.show
  val age   = cat.age.show
  val color = cat.color.show
  s"$name is a $age year-old $color cat."
}

Finally, we use the Show interface syntax to print our instance of Cat:

println(Cat("Garfield", 38, "ginger and black").show)
// Garfield is a 38 year-old ginger and black cat.

6.3 Example: Eq

We will finish off this chapter by looking at another useful type class: cats.Eq. Eq is designed to support type-safe equality and address annoyances using Scala’s built-in == operator.

Almost every Scala developer has written code like this before:

List(1, 2, 3).map(Option(_)).filter(item => item == 1)
// warning: Option[Int] and Int are unrelated: they will most likely never compare equal
// res: List[Option[Int]] = List()

Ok, many of you won’t have made such a simple mistake as this, but the principle is sound. The predicate in the filter clause always returns false because it is comparing an Int to an Option[Int].

This is programmer error—we should have compared item to Some(1) instead of 1. However, it’s not technically a type error because == works for any pair of objects, no matter what types we compare. Eq is designed to add some type safety to equality checks and work around this problem.

6.3.1 Equality, Liberty, and Fraternity

We can use Eq to define type-safe equality between instances of any given type:

package cats

trait Eq[A] {
  def eqv(a: A, b: A): Boolean
  // other concrete methods based on eqv...
}

The interface syntax, defined in cats.syntax.eq, provides two methods for performing equality checks provided there is an instance Eq[A] in scope:

6.3.2 Comparing Ints

Let’s look at a few examples. First we import the type class:

import cats.*

Now let’s grab an instance for Int:

val eqInt = Eq[Int]

We can use eqInt directly to test for equality:

eqInt.eqv(123, 123)
// res1: Boolean = true
eqInt.eqv(123, 234)
// res2: Boolean = false

Unlike Scala’s == method, if we try to compare objects of different types using eqv we get a compile error:

eqInt.eqv(123, "234")
// error:
// Found:    ("234" : String)
// Required: Int
// eqInt.eqv(123, "234")
//                ^^^^^

We can also import the interface syntax in cats.syntax.eq to use the === and =!= methods:

import cats.syntax.all.* // for === and =!=
123 === 123
// res4: Boolean = true
123 =!= 234
// res5: Boolean = true

Again, comparing values of different types causes a compiler error:

123 === "123"
// error:
// Found:    ("123" : String)
// Required: Int
// 123 === "123"
//         ^^^^^

6.3.3 Comparing Options

Now for a more interesting example—Option[Int].

Some(1) === None
// error:
// value === is not a member of Some[Int] - did you mean Some[Int].==?
// Some(1) === None
// ^^^^^^^^^^^

We have received an error here because the types don’t quite match up. We have Eq instances in scope for Int and Option[Int] but the values we are comparing are of type Some[Int]. To fix the issue we have to re-type the arguments as Option[Int]:

(Some(1) : Option[Int]) === (None : Option[Int])
// res8: Boolean = false

We can do this in a friendlier fashion using the Option.apply and Option.empty methods from the standard library:

Option(1) === Option.empty[Int]
// res9: Boolean = false

or using special syntax from cats.syntax.option:

1.some === none[Int]
// res10: Boolean = false
1.some =!= none[Int]
// res11: Boolean = true

6.3.4 Comparing Custom Types

We can define our own instances of Eq using the Eq.instance method, which accepts a function of type (A, A) => Boolean and returns an Eq[A]:

import java.util.Date

given dateEq: Eq[Date] =
  Eq.instance[Date] { (date1, date2) =>
    date1.getTime === date2.getTime
  }
val x = new Date() // now
val y = new Date() // a bit later than now
x === x
// res12: Boolean = true
x === y
// res13: Boolean = true

6.3.4.1 Exercise: Equality, Liberty, and Felinity

Implement an instance of Eq for our running Cat example:

final case class Cat(name: String, age: Int, color: String)

Use this to compare the following pairs of objects for equality and inequality:

val cat1 = Cat("Garfield",   38, "orange and black")
val cat2 = Cat("Heathcliff", 33, "orange and black")

val optionCat1 = Option(cat1)
val optionCat2 = Option.empty[Cat]

First we need our Cats imports. In this exercise we’ll be using the Eq type class and the Eq interface syntax, so we start by importing that.

import cats.*
import cats.syntax.all.* 

Our Cat class is the same as ever:

final case class Cat(name: String, age: Int, color: String)

We bring the Eq instances for Int and String into scope for the implementation of Eq[Cat]:

given catEqual: Eq[Cat] =
  Eq.instance[Cat] { (cat1, cat2) =>
    (cat1.name  === cat2.name ) &&
    (cat1.age   === cat2.age  ) &&
    (cat1.color === cat2.color)
  }

Finally, we test things out in a sample application:

val cat1 = Cat("Garfield",   38, "orange and black")
// cat1: Cat = Cat(name = "Garfield", age = 38, color = "orange and black")
val cat2 = Cat("Heathcliff", 32, "orange and black")
// cat2: Cat = Cat(name = "Heathcliff", age = 32, color = "orange and black")

cat1 === cat2
// res15: Boolean = false
cat1 =!= cat2
// res16: Boolean = true

val optionCat1 = Option(cat1)
// optionCat1: Option[Cat] = Some(
//   value = Cat(name = "Garfield", age = 38, color = "orange and black")
// )
val optionCat2 = Option.empty[Cat]
// optionCat2: Option[Cat] = None

optionCat1 === optionCat2
// res17: Boolean = false
optionCat1 =!= optionCat2
// res18: Boolean = true

7 Monoids and Semigroups

In this section we explore our first type classes, monoid and semigroup. These allow us to add or combine values. There are instances for Ints, Strings, Lists, Options, and many more. Let’s start by looking at a few simple types and operations to see what common principles we can extract.

7.0.0.1 Integer addition

Addition of Ints is a binary operation that is closed, meaning that adding two Ints always produces another Int:

2 + 1
// res0: Int = 3

There is also the identity element 0 with the property that a + 0 == 0 + a == a for any Int a:

2 + 0
// res1: Int = 2

0 + 2
// res2: Int = 2

There are also other properties of addition. For instance, it doesn’t matter in what order we add elements because we always get the same result. This is a property known as associativity:

(1 + 2) + 3
// res3: Int = 6

1 + (2 + 3)
// res4: Int = 6

7.0.0.2 Integer multiplication

The same properties for addition also apply for multiplication, provided we use 1 as the identity instead of 0:

1 * 3
// res5: Int = 3

3 * 1
// res6: Int = 3

Multiplication, like addition, is associative:

(1 * 2) * 3
// res7: Int = 6

1 * (2 * 3)
// res8: Int = 6

7.0.0.3 String and sequence concatenation

We can also add Strings, using string concatenation as our binary operator:

"One" ++ "two"
// res9: String = "Onetwo"

and the empty string as the identity:

"" ++ "Hello"
// res10: String = "Hello"

"Hello" ++ ""
// res11: String = "Hello"

Once again, concatenation is associative:

("One" ++ "Two") ++ "Three"
// res12: String = "OneTwoThree"

"One" ++ ("Two" ++ "Three")
// res13: String = "OneTwoThree"

Note that we used ++ above instead of the more usual + to suggest a parallel with sequences. We can do the same with other types of sequence, using concatenation as the binary operator and the empty sequence as our identity.

7.1 Definition of a Monoid

We’ve seen a number of “addition” scenarios above each with an associative binary addition and an identity element. It will be no surprise to learn that this is a monoid. Formally, a monoid for a type A is:

This definition translates nicely into Scala code. Here is a simplified version of the definition from Cats:

trait Monoid[A] {
  def combine(x: A, y: A): A
  def empty: A
}

In addition to providing the combine and empty operations, monoids must formally obey several laws. For all values x, y, and z, in A, combine must be associative and empty must be an identity element:

def associativeLaw[A](x: A, y: A, z: A)
      (using m: Monoid[A]): Boolean = {
  m.combine(x, m.combine(y, z)) ==
    m.combine(m.combine(x, y), z)
}

def identityLaw[A](x: A)
      (using m: Monoid[A]): Boolean = {
  (m.combine(x, m.empty) == x) &&
    (m.combine(m.empty, x) == x)
}

Integer subtraction, for example, is not a monoid because subtraction is not associative:

(1 - 2) - 3
// res14: Int = -4

1 - (2 - 3)
// res15: Int = 2

In practice we only need to think about laws when we are writing our own Monoid instances. Unlawful instances are dangerous because they can yield unpredictable results when used with the rest of Cats’ machinery. Most of the time we can rely on the instances provided by Cats and assume the library authors know what they’re doing.

7.2 Definition of a Semigroup

A semigroup is just the combine part of a monoid, without the empty part. While many semigroups are also monoids, there are some data types for which we cannot define an empty element. For example, we have just seen that sequence concatenation and integer addition are monoids. However, if we restrict ourselves to non-empty sequences and positive integers, we are no longer able to define a sensible empty element. Cats has a NonEmptyList data type that has an implementation of Semigroup but no implementation of Monoid.

A more accurate (though still simplified) definition of Cats’ Monoid is:

trait Semigroup[A] {
  def combine(x: A, y: A): A
}

trait Monoid[A] extends Semigroup[A] {
  def empty: A
}

We’ll see this kind of inheritance often when discussing type classes. It provides modularity and allows us to re-use behaviour. If we define a Monoid for a type A, we get a Semigroup for free. Similarly, if a method requires a parameter of type Semigroup[B], we can pass a Monoid[B] instead.

7.2.0.1 Exercise: The Truth About Monoids

We’ve seen a few examples of monoids but there are plenty more to be found. Consider Boolean. How many monoids can you define for this type? For each monoid, define the combine and empty operations and convince yourself that the monoid laws hold. Use the following definitions as a starting point:

trait Semigroup[A] {
  def combine(x: A, y: A): A
}

trait Monoid[A] extends Semigroup[A] {
  def empty: A
}

object Monoid {
  def apply[A](implicit monoid: Monoid[A]) =
    monoid
}

There are at least four monoids for Boolean! First, we have and with operator && and identity true:

given booleanAndMonoid: Monoid[Boolean] with {
  def combine(a: Boolean, b: Boolean) = a && b
  def empty = true
}

Second, we have or with operator || and identity false:

given booleanOrMonoid: Monoid[Boolean] with {
  def combine(a: Boolean, b: Boolean) = a || b
  def empty = false
}

Third, we have exclusive or with identity false:

given booleanEitherMonoid: Monoid[Boolean] with {
  def combine(a: Boolean, b: Boolean) =
    (a && !b) || (!a && b)

  def empty = false
}

Finally, we have exclusive nor (the negation of exclusive or) with identity true:

given booleanXnorMonoid: Monoid[Boolean] with {
  def combine(a: Boolean, b: Boolean) =
    (!a || b) && (a || !b)

  def empty = true
}

Showing that the identity law holds in each case is straightforward. Similarly associativity of the combine operation can be shown by enumerating the cases.

7.2.0.2 Exercise: All Set for Monoids

What monoids and semigroups are there for sets?

Set union forms a monoid along with the empty set:

given setUnionMonoid[A]: Monoid[Set[A]] with {
  def combine(a: Set[A], b: Set[A]) = a.union(b)
  def empty = Set.empty[A]
}

We need to define setUnionMonoid as a method rather than a value so we can accept the type parameter A. The type parameter allows us to use the same definition to summon Monoids for Sets of any type of data:

val intSetMonoid = Monoid[Set[Int]]
val strSetMonoid = Monoid[Set[String]]
intSetMonoid.combine(Set(1, 2), Set(2, 3))
// res18: Set[Int] = Set(1, 2, 3)
strSetMonoid.combine(Set("A", "B"), Set("B", "C"))
// res19: Set[String] = Set("A", "B", "C")

Set intersection forms a semigroup, but doesn’t form a monoid because it has no identity element:

given setIntersectionSemigroup[A]: Semigroup[Set[A]] with {
  def combine(a: Set[A], b: Set[A]) =
    a.intersect(b)
}

Set complement and set difference are not associative, so they cannot be considered for either monoids or semigroups. However, symmetric difference (the union less the intersection) does form a monoid with the empty set:

given symDiffMonoid[A]: Monoid[Set[A]] with {
  def combine(a: Set[A], b: Set[A]): Set[A] =
    (a.diff(b)).union(b.diff(a))

  def empty: Set[A] = Set.empty
}

7.3 Monoids in Cats

Now we’ve seen what monoids are, let’s look at their implementation in Cats. Once again we’ll look at the three main aspects of the implementation: the type class, the instances, and the interface.

7.3.1 The Monoid Type Class

The monoid type class is cats.kernel.Monoid, which is aliased as cats.Monoid. Monoid extends cats.kernel.Semigroup, which is aliased as cats.Semigroup. When using Cats we normally import type classes from the cats package:

import cats.Monoid
import cats.Semigroup

or just

import cats.*

Cats Kernel?

Cats Kernel is a subproject of Cats providing a small set of typeclasses for libraries that don’t require the full Cats toolbox. While these core type classes are technically defined in the cats.kernel package, they are all aliased to the cats package so we rarely need to be aware of the distinction.

The Cats Kernel type classes covered in this book are Eq, Semigroup, and Monoid. All the other type classes we cover are part of the main Cats project and are defined directly in the cats package.

7.3.2 Monoid Instances

Monoid follows the standard Cats pattern for the user interface: the companion object has an apply method that returns the type class instance for a particular type. For example, if we want the monoid instance for String, and we have the correct given instances in scope, we can write the following:

import cats.Monoid
Monoid[String].combine("Hi ", "there")
// res1: String = "Hi there"
Monoid[String].empty
// res2: String = ""

which is equivalent to:

Monoid.apply[String].combine("Hi ", "there")
// res3: String = "Hi there"
Monoid.apply[String].empty
// res4: String = ""

As we know, Monoid extends Semigroup. If we don’t need empty we can equivalently write:

import cats.Semigroup
Semigroup[String].combine("Hi ", "there")
// res5: String = "Hi there"

The standard type class instances for Monoid are all found on the appropriate companion objects, and so are automatically in the given scope with no further imports required.

7.3.3 Monoid Syntax

Cats provides syntax for the combine method in the form of the |+| operator. Because combine technically comes from Semigroup, we access the syntax by importing from cats.syntax.semigroup:

import cats.syntax.semigroup.* // for |+|
val stringResult = "Hi " |+| "there" |+| Monoid[String].empty
// stringResult: String = "Hi there"

val intResult = 1 |+| 2 |+| Monoid[Int].empty
// intResult: Int = 3

As always, unless there is compelling reason not, we recommend importing all the syntax with

import cats.syntax.all.*

7.3.3.1 Exercise: Adding All The Things

The cutting edge SuperAdder v3.5a-32 is the world’s first choice for adding together numbers. The main function in the program has signature def add(items: List[Int]): Int. In a tragic accident this code is deleted! Rewrite the method and save the day!

We can write the addition as a foldLeft using 0 and the + operator:

def add(items: List[Int]): Int =
  items.foldLeft(0)(_ + _)

We can alternatively write the fold using Monoids, although there’s not a compelling use case for this yet:

import cats.Monoid
import cats.syntax.all.*

def add(items: List[Int]): Int =
  items.foldLeft(Monoid[Int].empty)(_ |+| _)

Well done! SuperAdder’s market share continues to grow, and now there is demand for additional functionality. People now want to add List[Option[Int]]. Change add so this is possible. The SuperAdder code base is of the highest quality, so make sure there is no code duplication!

Now there is a use case for Monoids. We need a single method that adds Ints and instances of Option[Int]. We can write this as a generic method that accepts an implicit Monoid as a parameter:

import cats.Monoid
import cats.syntax.all.*

def add[A](items: List[A])(using monoid: Monoid[A]): A =
  items.foldLeft(monoid.empty)(_ |+| _)

We can optionally use Scala’s context bound syntax to write the same code in a shorter way:

def add[A: Monoid](items: List[A]): A =
  items.foldLeft(Monoid[A].empty)(_ |+| _)

We can use this code to add values of type Int and Option[Int] as requested:

add(List(1, 2, 3))
// res9: Int = 6
add(List(Some(1), None, Some(2), None, Some(3)))
// res10: Option[Int] = Some(value = 6)

Note that if we try to add a list consisting entirely of Some values, we get a compile error:

add(List(Some(1), Some(2), Some(3)))
// error: 
// No given instance of type cats.kernel.Monoid[Some[Int]] was found for a context parameter of method add in object MdocApp3

This happens because the inferred type of the list is List[Some[Int]], while Cats will only generate a Monoid for Option[Int]. We’ll see how to get around this in a moment.

SuperAdder is entering the POS (point-of-sale, not the other POS) market. Now we want to add up Orders:

case class Order(totalCost: Double, quantity: Double)

We need to release this code really soon so we can’t make any modifications to add. Make it so!

Easy—we simply define a monoid instance for Order!

given monoid: Monoid[Order] with {
  def combine(o1: Order, o2: Order) =
    Order(
      o1.totalCost + o2.totalCost,
      o1.quantity + o2.quantity
    )

  def empty = Order(0, 0)
}

7.4 Applications of Monoids

We now know what a monoid is—an abstraction of the concept of adding or combining—but where is it useful? Here are a few big ideas where monoids play a major role. These are explored in more detail in case studies later in the book.

7.4.1 Big Data

In big data applications like Spark and Flink we distribute data analysis over many machines, giving fault tolerance and scalability. This means each machine will return results over a portion of the data, and we must then combine these results to get our final result. In the vast majority of cases this can be viewed as a monoid.

If we want to calculate how many total visitors a web site has received, that means calculating an Int on each portion of the data. We know the monoid instance of Int is addition, which is the right way to combine partial results.

If we want to find out how many unique visitors a website has received, that’s equivalent to building a Set[User] on each portion of the data. We know the monoid instance for Set is the set union, which is the right way to combine partial results.

If we want to calculate 99% and 95% response times from our server logs, we can use a data structure called a QTree for which there is a monoid.

Hopefully you get the idea. Almost every analysis that we might want to do over a large data set is a monoid, and therefore we can build an expressive and powerful analytics system around this idea. This is exactly what Twitter’s Algebird and Summingbird projects have done. We explore this idea further in the map-reduce case study in Section 18.

7.4.2 Distributed Systems

In a distributed system, different machines may end up with different views of data. For example, one machine may receive an update that other machines did not receive. We would like to reconcile these different views, so every machine has the same data if no more updates arrive. This is called eventual consistency.

A particular class of data types support this reconciliation. These data types are called conflict-free replicated data types (CRDTs). The key operation is the ability to merge two data instances, with a result that captures all the information in both instances. This operation relies on having a monoid instance. We explore this idea further in the CRDT case study.

7.4.3 Monoids in the Small

The two examples above are cases where monoids inform the entire system architecture. There are also many cases where having a monoid around makes it easier to write a small code fragment. We’ll see lots of examples in the remainder of this book.

7.5 Summary

We hit a big milestone in this chapter—we covered our first type classes with fancy functional programming names:

We can use Semigroups and Monoids by importing two things: the type classes themselves, and the semigroup syntax to give us the |+| operator:

import cats.Monoid
import cats.syntax.semigroup.* // for |+|
"Scala" |+| " with " |+| "Cats"
// res0: String = "Scala with Cats"

With the correct instances in scope, we can set about adding anything we want:

Option(1) |+| Option(2)
// res1: Option[Int] = Some(value = 3)
val map1 = Map("a" -> 1, "b" -> 2)
val map2 = Map("b" -> 3, "d" -> 4)
map1 |+| map2
// res2: Map[String, Int] = Map("b" -> 5, "d" -> 4, "a" -> 1)
val tuple1 = ("hello", 123)
val tuple2 = ("world", 321)
tuple1 |+| tuple2
// res3: Tuple2[String, Int] = ("helloworld", 444)

We can also write generic code that works with any type for which we have an instance of Monoid:

def addAll[A](values: List[A])
      (using monoid: Monoid[A]): A =
  values.foldRight(monoid.empty)(_ |+| _)
addAll(List(1, 2, 3))
// res4: Int = 6
addAll(List(None, Some(1), Some(2)))
// res5: Option[Int] = Some(value = 3)

Monoids are a great gateway to Cats. They’re easy to understand and simple to use. However, they’re just the tip of the iceberg in terms of the abstractions Cats enables us to make. In the next chapter we’ll look at functors, the type class personification of the beloved map method. That’s where the fun really begins!

8 Functors

In this chapter we will investigate functors, an abstraction that allows us to represent sequences of operations within a context such as a List, an Option, or any one of thousands of other possibilities. Functors on their own aren’t so useful, but special cases of functors, such as monads and applicative functors, are some of the most commonly used abstractions.

8.1 Examples of Functors

Informally, a functor is anything with a map method. You probably know lots of types that have this: Option, List, and Either, to name a few.

We typically first encounter map when iterating over Lists. However, to understand functors we need to think of the method in another way. Rather than traversing the list, we should think of it as transforming all of the values inside in one go. We specify the function to apply, and map ensures it is applied to every item. The values change but the structure of the list (the number of elements and their order) remains the same:

List(1, 2, 3).map(n => n + 1)
// res0: List[Int] = List(2, 3, 4)

Similarly, when we map over an Option, we transform the contents but leave the Some or None context unchanged. The same principle applies to Either with its Left and Right contexts. This general notion of transformation, along with the common pattern of type signatures shown in Figure 1, is what connects the behaviour of map across different data types.

list-option-either-map Created with Sketch. Either[E, A] map Either[E, B] A => B Option[A] map Option[B] A => B List[A] map List[B] A => B
Figure 1: Type chart: mapping over List, Option, and Either

Because map leaves the structure of the context unchanged, we can call it repeatedly to sequence multiple computations on the contents of an initial data structure:

List(1, 2, 3).
  map(n => n + 1).
  map(n => n * 2).
  map(n => s"${n}!")
// res1: List[String] = List("4!", "6!", "8!")

We should think of map not as an iteration pattern, but as a way of sequencing computations on values ignoring some complication dictated by the relevant data type:

8.2 More Examples of Functors

The map methods of List, Option, and Either apply functions eagerly. However, the idea of sequencing computations is more general than this. Let’s investigate the behaviour of some other functors that apply the pattern in different ways.

Futures

Future is a functor that sequences asynchronous computations by queueing them and applying them as their predecessors complete. The type signature of its map method, shown in Figure 2, has the same shape as the signatures above. However, the behaviour is very different.

future-map Created with Sketch. Future[A] Future[B] A => B map
Figure 2: Type chart: mapping over a Future

When we work with a Future we have no guarantees about its internal state. The wrapped computation may be ongoing, complete, or rejected. If the Future is complete, our mapping function can be called immediately. If not, some underlying thread pool queues the function call and comes back to it later. We don’t know when our functions will be called, but we do know what order they will be called in. In this way, Future provides the same sequencing behaviour seen in List, Option, and Either:

import scala.concurrent.{Future, Await}
import scala.concurrent.ExecutionContext.Implicits.global
import scala.concurrent.duration._

val future: Future[String] =
  Future(123).
    map(n => n + 1).
    map(n => n * 2).
    map(n => s"${n}!")
Await.result(future, 1.second)
// res2: String = "248!"

Futures and Referential Transparency

Note that Scala’s Futures aren’t a great example of pure functional programming because they aren’t referentially transparent. Future always computes and caches a result and there’s no way for us to tweak this behaviour. This means we can get unpredictable results when we use Future to wrap side-effecting computations. For example:

import scala.util.Random

val future1 = {
  // Initialize Random with a fixed seed:
  val r = new Random(0L)

  // nextInt has the side-effect of moving to
  // the next random number in the sequence:
  val x = Future(r.nextInt())

  for {
    a <- x
    b <- x
  } yield (a, b)
}

val future2 = {
  val r = new Random(0L)

  for {
    a <- Future(r.nextInt())
    b <- Future(r.nextInt())
  } yield (a, b)
}
val result1 = Await.result(future1, 1.second)
// result1: Tuple2[Int, Int] = (-1155484576, -1155484576)
val result2 = Await.result(future2, 1.second)
// result2: Tuple2[Int, Int] = (-1155484576, -723955400)

Ideally we would like result1 and result2 to contain the same value. However, the computation for future1 calls nextInt once and the computation for future2 calls it twice. Because nextInt returns a different result every time we get a different result in each case.

This kind of discrepancy makes it hard to reason about programs involving Futures and side-effects. There also are other problematic aspects of Future's behaviour, such as the way it always starts computations immediately rather than allowing the user to dictate when the program should run. For more information see this excellent Reddit answer by Rob Norris.

When we look at Cats Effect we’ll see that the IO type solves these problems.

If Future isn’t referentially transparent, perhaps we should look at another similar data-type that is. You should recognise this one…

Functions (?!)

It turns out that single argument functions are also functors. To see this we have to tweak the types a little. A function A => B has two type parameters: the parameter type A and the result type B. To coerce them to the correct shape we can fix the parameter type and let the result type vary:

If we alias X => A as MyFunc[A], we see the same pattern of types we saw with the other examples in this chapter. We also see this in Figure 3:

function-map Created with Sketch. X => A X => B A => B map
Figure 3: Type chart: mapping over a Function1

In other words, “mapping” over a Function1 is function composition:

import cats.syntax.all.*     // for map

val func1: Int => Double =
  (x: Int) => x.toDouble

val func2: Double => Double =
  (y: Double) => y * 2
(func1.map(func2))(1)     // composition using map
(func1.andThen(func2))(1) // composition using andThen
// res3: Double = 2.0
func2(func1(1))           // composition written out by hand
// res4: Double = 2.0

How does this relate to our general pattern of sequencing operations? If we think about it, function composition is sequencing. We start with a function that performs a single operation and every time we use map we append another operation to the chain. Calling map doesn’t actually run any of the operations, but if we can pass an argument to the final function all of the operations are run in sequence. We can think of this as lazily queueing up operations similar to Future:

val func =
  ((x: Int) => x.toDouble).
    map(x => x + 1).
    map(x => x * 2).
    map(x => s"${x}!")
func(123)
// res5: String = "248.0!"

Partial Unification

For the above examples to work, in versions of Scala before 2.13, we need to add the following compiler option to build.sbt:

scalacOptions += "-Ypartial-unification"

otherwise we’ll get a compiler error:

func1.map(func2)
// <console>: error: value map is not a member of Int => Double
//        func1.map(func2)
                ^

We’ll look at why this happens in detail in Section 8.8.

8.3 Definition of a Functor

Every example we’ve looked at so far is a functor: a class that encapsulates sequencing computations. Formally, a functor is a type F[A] with an operation map with type (A => B) => F[B]. The general type chart is shown in Figure 4.

Figure 4: Type chart: generalised functor map

Cats encodes Functor as a type class, cats.Functor, so the method looks a little different. It accepts the initial F[A] as a parameter alongside the transformation function. Here’s a simplified version of the definition:

package cats

trait Functor[F[_]] {
  def map[A, B](fa: F[A])(f: A => B): F[B]
}

If you haven’t seen syntax like F[_] before, it’s time to take a brief detour to discuss type constructors and higher kinded types.

Functor Laws

Functors guarantee the same semantics whether we sequence many small operations one by one, or combine them into a larger function before mapping. To ensure this is the case the following laws must hold:

Identity: calling map with the identity function is the same as doing nothing:

fa.map(a => a) == fa

Composition: mapping with two functions f and g is the same as mapping with f and then mapping with g:

fa.map(g(f(_))) == fa.map(f).map(g)

8.4 Aside: Higher Kinds and Type Constructors

Kinds are like types for types. They describe the number of “holes” in a type. We distinguish between regular types that have no holes and “type constructors” that have holes we can fill to produce types.

For example, List is a type constructor with one hole. We fill that hole by specifying a parameter to produce a regular type like List[Int] or List[A]. The trick is not to confuse type constructors with generic types. List is a type constructor, List[A] is a type:

List    // type constructor, takes one parameter
List[A] // type, produced by applying a type parameter

There’s a close analogy here with functions and values. Functions are “value constructors”—they produce values when we supply parameters:

math.abs    // function, takes one parameter
math.abs(x) // value, produced by applying a value parameter

In Scala we declare type constructors using underscores. This specifies how many “holes” the type constructor has. However, to use them we refer to just the name.

// Declare F using underscores:
def myMethod[F[_]] = {

  // Reference F without underscores:
  val functor = Functor.apply[F]

  // ...
}

This is analogous to specifying function parameter types. When we declare a parameter we also give its type. However, to use them we refer to just the name.

// Declare f specifying parameter types
def f(x: Int): Int = 
  // Reference x without type
  x * 2

Armed with this knowledge of type constructors, we can see that the Cats definition of Functor allows us to create instances for any single-parameter type constructor, such as List, Option, Future, or a type alias such as MyFunc.

Language Feature Imports

In versions of Scala before 2.13 we need to “enable” the higher kinded type language feature, to suppress warnings from the compiler, whenever we declare a type constructor with A[_] syntax. We can either do this with a “language import” as above:

import scala.language.higherKinds

or by adding the following to scalacOptions in build.sbt:

scalacOptions += "-language:higherKinds"

In practice we find the scalacOptions flag to be the simpler of the two options.

8.5 Functors in Cats

Let’s look at the implementation of functors in Cats. We’ll examine the same aspects we did for monoids: the type class, the instances, and the syntax.

8.5.1 The Functor Type Class and Instances

The functor type class is cats.Functor. We obtain instances using the standard Functor.apply method on the companion object. As usual, default instances are found on companion objects and do not have to be explicity imported:

import cats.*
import cats.syntax.all.*
val list1 = List(1, 2, 3)
// list1: List[Int] = List(1, 2, 3)
val list2 = Functor[List].map(list1)(_ * 2)
// list2: List[Int] = List(2, 4, 6)

val option1 = Option(123)
// option1: Option[Int] = Some(value = 123)
val option2 = Functor[Option].map(option1)(_.toString)
// option2: Option[String] = Some(value = "123")

Functor provides a method called lift, which converts a function of type A => B to one that operates over a functor and has type F[A] => F[B]:

val func = (x: Int) => x + 1
// func: Function1[Int, Int] = repl.MdocSession$MdocApp0$$$Lambda$20217/0x00000008050ef040@1d389b3d

val liftedFunc = Functor[Option].lift(func)
// liftedFunc: Function1[Option[Int], Option[Int]] = cats.Functor$$Lambda$20218/0x00000008050ea840@2fb812fb

liftedFunc(Option(1))
// res1: Option[Int] = Some(value = 2)

The as method is the other method you are likely to use. It replaces with value inside the Functor with the given value.

Functor[List].as(list1, "As")
// res2: List[String] = List("As", "As", "As")

8.5.2 Functor Syntax

The main method provided by the syntax for Functor is map. It’s difficult to demonstrate this with Options and Lists as they have their own built-in map methods and the Scala compiler will always prefer a built-in method over an extension method. We’ll work around this with two examples.

First let’s look at mapping over functions. Scala’s Function1 type doesn’t have a map method (it’s called andThen instead) so there are no naming conflicts:

val func1 = (a: Int) => a + 1
val func2 = (a: Int) => a * 2
val func3 = (a: Int) => s"${a}!"
val func4 = func1.map(func2).map(func3)
func4(123)
// res3: String = "248!"

Let’s look at another example. This time we’ll abstract over functors so we’re not working with any particular concrete type. We can write a method that applies an equation to a number no matter what functor context it’s in:

def doMath[F[_]](start: F[Int])
    (implicit functor: Functor[F]): F[Int] =
  start.map(n => n + 1 * 2)
doMath(Option(20))
// res4: Option[Int] = Some(value = 22)
doMath(List(1, 2, 3))
// res5: List[Int] = List(3, 4, 5)

To illustrate how this works, let’s take a look at the definition of the map method in cats.syntax.functor. Here’s a simplified version of the code:

implicit class FunctorOps[F[_], A](src: F[A]) {
  def map[B](func: A => B)
      (implicit functor: Functor[F]): F[B] =
    functor.map(src)(func)
}

The compiler can use this extension method to insert a map method wherever no built-in map is available:

foo.map(value => value + 1)

Assuming foo has no built-in map method, the compiler detects the potential error and wraps the expression in a FunctorOps to fix the code:

new FunctorOps(foo).map(value => value + 1)

The map method of FunctorOps requires an implicit Functor as a parameter. This means this code will only compile if we have a Functor for F in scope. If we don’t, we get a compiler error:

final case class Box[A](value: A)

val box = Box[Int](123)
box.map(value => value + 1)
// error:
// value map is not a member of repl.MdocSession.MdocApp0.Box[Int]
// box.map(value => value + 1)
//    ^

The as method is also available as syntax.

List(1, 2, 3).as("As")
// res7: List[String] = List("As", "As", "As")

8.5.3 Instances for Custom Types

We can define a functor simply by defining its map method. Here’s an example of a Functor for Option, even though such a thing already exists in cats.instances. The implementation is trivial—we simply call Option's map method:

implicit val optionFunctor: Functor[Option] =
  new Functor[Option] {
    def map[A, B](value: Option[A])(func: A => B): Option[B] =
      value.map(func)
  }

Sometimes we need to inject dependencies into our instances. For example, if we had to define a custom Functor for Future (another hypothetical example—Cats provides one in cats.instances.future) we would need to account for the implicit ExecutionContext parameter on future.map. We can’t add extra parameters to functor.map so we have to account for the dependency when we create the instance:

import scala.concurrent.{Future, ExecutionContext}

implicit def futureFunctor
    (implicit ec: ExecutionContext): Functor[Future] =
  new Functor[Future] {
    def map[A, B](value: Future[A])(func: A => B): Future[B] =
      value.map(func)
  }

Whenever we summon a Functor for Future, either directly using Functor.apply or indirectly via the map extension method, the compiler will locate futureFunctor by implicit resolution and recursively search for an ExecutionContext at the call site. This is what the expansion might look like:

// We write this:
Functor[Future]

// The compiler expands to this first:
Functor[Future](futureFunctor)

// And then to this:
Functor[Future](futureFunctor(executionContext))

8.5.4 Exercise: Branching out with Functors

Write a Functor for the following binary tree data type. Verify that the code works as expected on instances of Branch and Leaf:

sealed trait Tree[+A]

final case class Branch[A](left: Tree[A], right: Tree[A])
  extends Tree[A]

final case class Leaf[A](value: A) extends Tree[A]

The semantics are similar to writing a Functor for List. We recurse over the data structure, applying the function to every Leaf we find. The functor laws intuitively require us to retain the same structure with the same pattern of Branch and Leaf nodes:

implicit val treeFunctor: Functor[Tree] =
  new Functor[Tree] {
    def map[A, B](tree: Tree[A])(func: A => B): Tree[B] =
      tree match {
        case Branch(left, right) =>
          Branch(map(left)(func), map(right)(func))
        case Leaf(value) =>
          Leaf(func(value))
      }
  }

Let’s use our Functor to transform some Trees:

Branch(Leaf(10), Leaf(20)).map(_ * 2)
// error:
// value map is not a member of repl.MdocSession.MdocApp0.Branch[Int]
// Branch(Leaf(10), Leaf(20)).map(_ * 2)
//                           ^

Oops! This falls foul of the same invariance problem we discussed in Section 4.6.1. The compiler can find a Functor instance for Tree but not for Branch or Leaf. Let’s add some smart constructors to compensate:

object Tree {
  def branch[A](left: Tree[A], right: Tree[A]): Tree[A] =
    Branch(left, right)

  def leaf[A](value: A): Tree[A] =
    Leaf(value)
}

Now we can use our Functor properly:

Tree.leaf(100).map(_ * 2)
// res9: Tree[Int] = Leaf(value = 200)

Tree.branch(Tree.leaf(10), Tree.leaf(20)).map(_ * 2)
// res10: Tree[Int] = Branch(left = Leaf(value = 20), right = Leaf(value = 40))

8.6 Contravariant and Invariant Functors

As we have seen, we can think of Functor's map method as “appending” a transformation to a chain. We’re now going to look at two other type classes, one representing prepending operations to a chain, and one representing building a bidirectional chain of operations. These are called contravariant and invariant functors respectively.

This Section is Optional!

You don’t need to know about contravariant and invariant functors to understand monads, which are the most important type class in this book and the focus of the next chapter. However, contravariant and invariant do come in handy in our discussion of Semigroupal and Applicative in Chapter 11.

If you want to move on to monads now, feel free to skip straight to Chapter 9. Come back here before you read Chapter 11.

8.6.1 Contravariant Functors and the contramap Method

The first of our type classes, the contravariant functor, provides an operation called contramap that represents “prepending” an operation to a chain. The general type signature is shown in Figure 5.

generic-contramap Created with Sketch. F[B] F[A] A => B contramap
Figure 5: Type chart: the contramap method

The contramap method only makes sense for data types that represent transformations. For example, we can’t define contramap for an Option because there is no way of feeding a value in an Option[B] backwards through a function A => B. However, we can define contramap for the Display type class we discussed in Section 4.5:

trait Display[A] {
  def display(value: A): String
}

A Display[A] represents a transformation from A to String. Its contramap method accepts a function func of type B => A and creates a new Display[B]:

trait Display[A] {
  def display(value: A): String

  def contramap[B](func: B => A): Display[B] =
    ???
}

def display[A](value: A)(using p: Display[A]): String =
  p.display(value)

8.6.1.1 Exercise: Showing off with Contramap

Implement the contramap method for Display above. Start with the following code template and replace the ??? with a working method body:

trait Display[A] {
  def display(value: A): String

  def contramap[B](func: B => A): Display[B] =
    new Display[B] {
      def display(value: B): String =
        ???
    }
}

If you get stuck, think about the types. You need to turn value, which is of type B, into a String. What functions and methods do you have available and in what order do they need to be combined?

Here’s a working implementation. We call func to turn the B into an A and then use our original Display to turn the A into a String. In a small show of sleight of hand we use a self alias to distinguish the outer and inner Displays:

trait Display[A] { self =>

  def display(value: A): String

  def contramap[B](func: B => A): Display[B] =
    new Display[B] {
      def display(value: B): String =
        self.display(func(value))
    }
}

def display[A](value: A)(using p: Display[A]): String =
  p.display(value)

For testing purposes, let’s define some instances of Display for String and Boolean:

given stringDisplay: Display[String] with {
  def display(value: String): String =
    s"'${value}'"
}

given booleanDisplay: Display[Boolean] with {
  def display(value: Boolean): String =
    if value then "yes" else "no"
}
display("hello")
// res2: String = "'hello'"
display(true)
// res3: String = "yes"

Now define an instance of Display for the following Box case class. This is an example of type class composotion as described in Section 4.3:

final case class Box[A](value: A)

Rather than writing out the complete definition from scratch (new Display[Box] etc…), create your instance from an existing instance using contramap.

Your instance should work as follows:

display(Box("hello world"))
// res4: String = "'hello world'"
display(Box(true))
// res5: String = "yes"

If we don’t have a Display for the type inside the Box, calls to display should fail to compile:

display(Box(123))
// error:
// No given instance of type repl.MdocSession.MdocApp1.Display[repl.MdocSession.MdocApp1.Box[Int]] was found for parameter p of method display in object MdocApp1.
// I found:
// 
//     repl.MdocSession.MdocApp1.boxDisplay[A](
//       /* missing */summon[repl.MdocSession.MdocApp1.Display[A]])
// 
// But no implicit values were found that match type repl.MdocSession.MdocApp1.Display[A].
// display(Box(123))
//                 ^

To make the instance generic across all types of Box, we base it on the Display for the type inside the Box. We can either write out the complete definition by hand:

given boxDisplay[A](
    using p: Display[A]
): Display[Box[A]] with {
  def display(box: Box[A]): String =
    p.display(box.value)
}

or use contramap to base the new instance on the using clause:

given boxDisplay[A](using p: Display[A]): Display[Box[A]] =
  p.contramap[Box[A]](_.value)

Using contramap is much simpler, and conveys the functional programming approach of building solutions by combining simple building blocks using pure functional combinators.

8.6.2 Invariant functors and the imap method

Invariant functors implement a method called imap that is informally equivalent to a combination of map and contramap. If map generates new type class instances by appending a function to a chain, and contramap generates them by prepending an operation to a chain, imap generates them via a pair of bidirectional transformations.

The most intuitive examples of this are a type class that represents encoding and decoding as some data type, such as Circe’s Codec and Play JSON’s Format. We can build our own Codec by enhancing Display to support encoding and decoding to/from a String:

trait Codec[A] {
  def encode(value: A): String
  def decode(value: String): A
  def imap[B](dec: A => B, enc: B => A): Codec[B] = ???
}
def encode[A](value: A)(using c: Codec[A]): String =
  c.encode(value)

def decode[A](value: String)(using c: Codec[A]): A =
  c.decode(value)

The type chart for imap is shown in Figure 6. If we have a Codec[A] and a pair of functions A => B and B => A, the imap method creates a Codec[B]:

generic-imap Created with Sketch. F[A] F[B] A => B , B => A imap
Figure 6: Type chart: the imap method

As an example use case, imagine we have a basic Codec[String], whose encode and decode methods both simply return the value they are passed:

given stringCodec: Codec[String] with {
  def encode(value: String): String = value
  def decode(value: String): String = value
}

We can construct many useful Codecs for other types by building off of stringCodec using imap:

given intCodec: Codec[Int] =
  stringCodec.imap(_.toInt, _.toString)

given booleanCodec: Codec[Boolean] =
  stringCodec.imap(_.toBoolean, _.toString)

Coping with Failure

Note that the decode method of our Codec type class doesn’t account for failures. If we want to model more sophisticated relationships we can move beyond functors to look at lenses and optics.

Optics are beyond the scope of this book. However, Julien Truffaut’s library Monocle provides a great starting point for further investigation.

8.6.2.1 Transformative Thinking with imap

Implement the imap method for Codec above.

Here’s a working implementation:

trait Codec[A] { self =>
  def encode(value: A): String
  def decode(value: String): A

  def imap[B](dec: A => B, enc: B => A): Codec[B] = {
    new Codec[B] {
      def encode(value: B): String =
        self.encode(enc(value))

      def decode(value: String): B =
        dec(self.decode(value))
    }
  }
}

Demonstrate your imap method works by creating a Codec for Double.

We can implement this using the imap method of stringCodec:

given doubleCodec: Codec[Double] =
  stringCodec.imap[Double](_.toDouble, _.toString)

Finally, implement a Codec for the following Box type:

final case class Box[A](value: A)

We need a generic Codec for Box[A] for any given A. We create this by calling imap on a Codec[A], which we bring into scope using an implicit parameter:

given boxCodec[A](using c: Codec[A]): Codec[Box[A]] =
  c.imap[Box[A]](Box(_), _.value)

Your instances should work as follows:

encode(123.4)
// res11: String = "123.4"
decode[Double]("123.4")
// res12: Double = 123.4

encode(Box(123.4))
// res13: String = "123.4"
decode[Box[Double]]("123.4")
// res14: Box[Double] = Box(value = 123.4)

What’s With the Names?

What’s the relationship between the terms “contravariance”, “invariance”, and “covariance” and these different kinds of functor?

If you recall from Section 4.6.1, variance affects subtyping, which is essentially our ability to use a value of one type in place of a value of another type without breaking the code.

Subtyping can be viewed as a conversion. If B is a subtype of A, we can always convert a B to an A.

Equivalently we could say that B is a subtype of A if there exists a function B => A. A standard covariant functor captures exactly this. If F is a covariant functor, wherever we have an F[B] and a conversion B => A we can always convert to an F[A].

A contravariant functor captures the opposite case. If F is a contravariant functor, whenever we have a F[A] and a conversion B => A we can convert to an F[B].

Finally, invariant functors capture the case where we can convert from F[A] to F[B] via a function A => B and vice versa via a function B => A.

8.7 Contravariant and Invariant in Cats

Let’s look at the implementation of contravariant and invariant functors in Cats, provided by the cats.Contravariant and cats.Invariant type classes respectively. Here’s a simplified version of the code:

trait Contravariant[F[_]] {
  def contramap[A, B](fa: F[A])(f: B => A): F[B]
}

trait Invariant[F[_]] {
  def imap[A, B](fa: F[A])(f: A => B)(g: B => A): F[B]
}

8.7.1 Contravariant in Cats

We can summon instances of Contravariant using the Contravariant.apply method. Cats provides instances for data types that consume parameters, including Eq, Show, and Function1. Here’s an example:

import cats.*

val showString = Show[String]

val showSymbol = Contravariant[Show].
  contramap(showString)((sym: Symbol) => s"'${sym.name}")
showSymbol.show(Symbol("dave"))
// res1: String = "'dave"

More conveniently, we can use cats.syntax.contravariant, which provides a contramap extension method:

import cats.syntax.contravariant.* // for contramap
showString
  .contramap[Symbol](sym => s"'${sym.name}")
  .show(Symbol("dave"))
// res2: String = "'dave"

8.7.2 Invariant in Cats

Among other types, Cats provides an instance of Invariant for Monoid. This is a little different from the Codec example we introduced in Section 8.6.2. If you recall, this is what Monoid looks like:

package cats

trait Monoid[A] {
  def empty: A
  def combine(x: A, y: A): A
}

Imagine we want to produce a Monoid for Scala’s Symbol type. Cats doesn’t provide a Monoid for Symbol but it does provide a Monoid for a similar type: String. We can write our new semigroup with an empty method that relies on the empty String, and a combine method that works as follows:

  1. accept two Symbols as parameters;
  2. convert the Symbols to Strings;
  3. combine the Strings using Monoid[String];
  4. convert the result back to a Symbol.

We can implement combine using imap, passing functions of type String => Symbol and Symbol => String as parameters. Here’ the code, written out using the imap extension method provided by cats.syntax.invariant:

import cats.*
import cats.syntax.invariant.* // for imap
import cats.syntax.semigroup.* // for |+|

given symbolMonoid: Monoid[Symbol] =
  Monoid[String].imap(Symbol.apply)(_.name)
Monoid[Symbol].empty
// res3: Symbol = '

Symbol("a") |+| Symbol("few") |+| Symbol("words")
// res4: Symbol = 'afewwords

8.8 Aside: Partial Unification

In Section 8.2 we saw a functor instance for Function1.

import cats.*
import cats.syntax.functor.*     // for map

val func1 = (x: Int)    => x.toDouble
val func2 = (y: Double) => y * 2
val func3 = func1.map(func2)
// func3: Function1[Int, Double] = cats.instances.Function1Instances0$$anon$11$$Lambda$20223/0x00000008050c1040@3c5fdfde

Function1 has two type parameters (the function argument and the result type):

trait Function1[-A, +B] {
  def apply(arg: A): B
}

However, Functor accepts a type constructor with one parameter:

trait Functor[F[_]] {
  def map[A, B](fa: F[A])(func: A => B): F[B]
}

The compiler has to fix one of the two parameters of Function1 to create a type constructor of the correct kind to pass to Functor. It has two options to choose from:

type F[A] = Int => A
type F[A] = A => Double

We know that the former of these is the correct choice. However the compiler doesn’t understand what the code means. Instead it relies on a simple rule, implementing what is called “partial unification”.

The partial unification in the Scala compiler works by fixing type parameters from left to right. In the above example, the compiler fixes the Int in Int => Double and looks for a Functor for functions of type Int => ?:

type F[A] = Int => A

val functor = Functor[F]

This left-to-right elimination works for a wide variety of common scenarios, including Functors for types such as Function1 and Either:

val either: Either[String, Int] = Right(123)
// either: Either[String, Int] = Right(value = 123)

either.map(_ + 1)
// res0: Either[String, Int] = Right(value = 124)

Partial unification is the default behaviour in Scala 2.13. In earlier versions of Scala we need to add the -Ypartial-unification compiler flag. In sbt we would add the compiler flag in build.sbt:

scalacOptions += "-Ypartial-unification"

The rationale behind this change is discussed in SI-2712.

8.8.1 Limitations of Partial Unification

There are situations where left-to-right elimination is not the correct choice. One example is the Or type in Scalactic, which is a conventionally left-biased equivalent of Either:

type PossibleResult = ActualResult Or Error

Another example is the Contravariant functor for Function1.

While the covariant Functor for Function1 implements andThen-style left-to-right function composition, the Contravariant functor implements compose-style right-to-left composition. In other words, the following expressions are all equivalent:

val func3a: Int => Double =
  a => func2(func1(a))

val func3b: Int => Double =
  func2.compose(func1)
// Hypothetical example. This won't actually compile:
val func3c: Int => Double =
  func2.contramap(func1)

If we try this for real, however, our code won’t compile:

import cats.syntax.contravariant.* // for contramap
val func3c = func2.contramap(func1)
// error:
// value contramap is not a member of Double => Double.
// An extension method was tried, but could not be fully constructed:
// 
//     cats.syntax.contravariant.toContravariantOps[[R] =>> Double => R, A](
//       repl.MdocSession.MdocApp.func2)(
//       cats.Invariant.catsContravariantForFunction1[R])
// val func3c = func2.contramap(func1)
//              ^^^^^^^^^^^^^^^

The problem here is that the Contravariant for Function1 fixes the return type and leaves the parameter type varying, requiring the compiler to eliminate type parameters from right to left, as shown below and in Figure 7:

type F[A] = A => Double
function-contramap Created with Sketch. A => X B => X B => A contramap
Figure 7: Type chart: contramapping over a Function1

The compiler fails simply because of its left-to-right bias. We can prove this by creating a type alias that flips the parameters on Function1:

type <=[B, A] = A => B
type F[A] = Double <= A

If we re-type func2 as an instance of <=, we reset the required order of elimination and we can call contramap as desired:

val func2b: Double <= Double = func2
val func3c = func2b.contramap(func1)
// func3c: Function1[Int, Double] = scala.Function1$$Lambda$20329/0x000000080503a040@1666d48b

The difference between func2 and func2b is purely syntactic—both refer to the same value and the type aliases are otherwise completely compatible. Incredibly, however, this simple rephrasing is enough to give the compiler the hint it needs to solve the problem.

It is rare that we have to do this kind of right-to-left elimination. Most multi-parameter type constructors are designed to be right-biased, requiring the left-to-right elimination that is supported by the compiler out of the box. However, it is useful to know about this quirk of elimination order in case you ever come across an odd scenario like the one above.

8.9 Summary

Functors represent sequencing behaviours. We covered three types of functor in this chapter:

Regular Functors are by far the most common of these type classes, but even then it is rare to use them on their own. Functors form a foundational building block of several more interesting abstractions that we use all the time. In the following chapters we will look at two of these abstractions: monads and applicative functors.

Functors for collections are extremely important, as they transform each element independently of the rest. This allows us to parallelise or distribute transformations on large collections, a technique leveraged heavily in “map-reduce” frameworks like Hadoop. We will investigate this approach in more detail in the map-reduce case study later in Section 18.

The Contravariant and Invariant type classes are less widely applicable but are still useful for building data types that represent transformations. We will revisit them to discuss the Semigroupal type class later in Chapter 11.

9 Monads

Monads are one of the most common abstractions in Scala. Many Scala programmers quickly become intuitively familiar with monads, even if we don’t know them by name.

Informally, a monad is anything with a constructor and a flatMap method. All of the functors we saw in the last chapter are also monads, including Option, List, and Future. We even have special syntax to support monads: for comprehensions. However, despite the ubiquity of the concept, the Scala standard library lacks a concrete type to encompass “things that can be flatMapped”.

In this chapter we will take a deep dive into monads. We will start by motivating them with a few examples. We’ll proceed to their formal definition, and see how we can create a concrete type as a type class. We’ll then look at their implementation in Cats. Finally, we’ll tour some interesting monads that you may not have seen, providing introductions and examples of their use.

9.1 What is a Monad?

This is the question that has been posed in a thousand blog posts, with explanations and analogies involving concepts as diverse as cats, Mexican food, space suits full of toxic waste, and monoids in the category of endofunctors (whatever that means). We’re going to solve the problem of explaining monads once and for all by stating very simply:

A monad is a mechanism for sequencing computations.

That was easy! Problem solved, right? But then again, last chapter we said functors were a mechanism for exactly the same thing. Ok, maybe we need some more discussion…

In Section 8.1 we said that functors allow us to sequence computations ignoring some complication. However, functors are limited in that they only allow this complication to occur once at the beginning of the sequence. They don’t account for further complications at each step in the sequence.

This is where monads come in. A monad’s flatMap method allows us to specify what happens next, taking into account an intermediate complication. The flatMap method of Option takes intermediate Options into account. The flatMap method of List handles intermediate Lists. And so on. In each case, the function passed to flatMap specifies the application-specific part of the computation, and flatMap itself takes care of the complication allowing us to flatMap again. Let’s ground things by looking at some examples.

9.1.1 Options as Monads

Option allows us to sequence computations that may or may not return values. Here are some examples:

def parseInt(str: String): Option[Int] =
  scala.util.Try(str.toInt).toOption

def divide(a: Int, b: Int): Option[Int] =
  if(b == 0) None else Some(a / b)

Each of these methods may “fail” by returning None. The flatMap method allows us to ignore this when we sequence operations:

def stringDivideBy(aStr: String, bStr: String): Option[Int] =
  parseInt(aStr).flatMap { aNum =>
    parseInt(bStr).flatMap { bNum =>
      divide(aNum, bNum)
    }
  }

The semantics are:

At each step, flatMap chooses whether to call our function, and our function generates the next computation in the sequence. This is shown in Figure 8.

option-flatmap Created with Sketch. Option[A] flatMap Option[B] A => Option[B]
Figure 8: Type chart: flatMap for Option

The result of the computation is an Option, allowing us to call flatMap again and so the sequence continues. This results in the fail-fast error handling behaviour that we know and love, where a None at any step results in a None overall:

stringDivideBy("6", "2")
// res0: Option[Int] = Some(value = 3)
stringDivideBy("6", "0")
// res1: Option[Int] = None
stringDivideBy("6", "foo")
// res2: Option[Int] = None
stringDivideBy("bar", "2")
// res3: Option[Int] = None

Every monad is also a functor (see below for proof), so we can rely on both flatMap and map to sequence computations that do and don’t introduce a new monad. Plus, if we have both flatMap and map we can use for comprehensions to clarify the sequencing behaviour:

def stringDivideBy(aStr: String, bStr: String): Option[Int] =
  for {
    aNum <- parseInt(aStr)
    bNum <- parseInt(bStr)
    ans  <- divide(aNum, bNum)
  } yield ans

9.1.2 Lists as Monads

When we first encounter flatMap as budding Scala developers, we tend to think of it as a pattern for iterating over Lists. This is reinforced by the syntax of for comprehensions, which look very much like imperative for loops:

for {
  x <- (1 to 3).toList
  y <- (4 to 5).toList
} yield (x, y)
// res5: List[Tuple2[Int, Int]] = List(
//   (1, 4),
//   (1, 5),
//   (2, 4),
//   (2, 5),
//   (3, 4),
//   (3, 5)
// )

However, there is another mental model we can apply that highlights the monadic behaviour of List. If we think of Lists as sets of intermediate results, flatMap becomes a construct that calculates permutations and combinations.

For example, in the for comprehension above there are three possible values of x and two possible values of y. This means there are six possible values of (x, y). flatMap is generating these combinations from our code, which states the sequence of operations:

9.1.3 Futures as Monads

Future is a monad that sequences computations without worrying that they may be asynchronous:

import scala.concurrent.Future
import scala.concurrent.ExecutionContext.Implicits.global

def doSomethingLongRunning: Future[Int] = ???
def doSomethingElseLongRunning: Future[Int] = ???

def doSomethingVeryLongRunning: Future[Int] =
  for {
    result1 <- doSomethingLongRunning
    result2 <- doSomethingElseLongRunning
  } yield result1 + result2

Again, we specify the code to run at each step, and flatMap takes care of all the horrifying underlying complexities of thread pools and schedulers.

If you’ve made extensive use of Future, you’ll know that the code above is running each operation in sequence. This becomes clearer if we expand out the for comprehension to show the nested calls to flatMap:

def doSomethingVeryLongRunning: Future[Int] =
  doSomethingLongRunning.flatMap { result1 =>
    doSomethingElseLongRunning.map { result2 =>
      result1 + result2
    }
  }

Each Future in our sequence is created by a function that receives the result from a previous Future. In other words, each step in our computation can only start once the previous step is finished. This is born out by the type chart for flatMap in Figure 9, which shows the function parameter of type A => Future[B].

future-flatmap Created with Sketch. Future[A] Future[B] A => Future[B] flatMap
Figure 9: Type chart: flatMap for Future

We can run futures in parallel, of course, but that is another story and shall be told another time. Monads are all about sequencing.

9.1.4 Definition of a Monad

While we have only talked about flatMap above, monadic behaviour is formally captured in two operations:

pure abstracts over constructors, providing a way to create a new monadic context from a plain value. flatMap provides the sequencing step we have already discussed, extracting the value from a context and generating the next context in the sequence. Here is a simplified version of the Monad type class in Cats:


trait Monad[F[_]] {
  def pure[A](value: A): F[A]

  def flatMap[A, B](value: F[A])(func: A => F[B]): F[B]
}

Monad Laws

pure and flatMap must obey a set of laws that allow us to sequence operations freely without unintended glitches and side-effects:

Left identity: calling pure and transforming the result with func is the same as calling func:

pure(a).flatMap(func) == func(a)

Right identity: passing pure to flatMap is the same as doing nothing:

m.flatMap(pure) == m

Associativity: flatMapping over two functions f and g is the same as flatMapping over f and then flatMapping over g:

m.flatMap(f).flatMap(g) == m.flatMap(x => f(x).flatMap(g))

9.1.5 Exercise: Getting Func-y

Every monad is also a functor. We can define map in the same way for every monad using the existing methods, flatMap and pure:


trait Monad[F[_]] {
  def pure[A](a: A): F[A]

  def flatMap[A, B](value: F[A])(func: A => F[B]): F[B]

  def map[A, B](value: F[A])(func: A => B): F[B] =
    ???
}

Try defining map yourself now.

At first glance this seems tricky, but if we follow the types we’ll see there’s only one solution. We are passed a value of type F[A]. Given the tools available there’s only one thing we can do: call flatMap:

trait Monad[F[_]] {
  def pure[A](value: A): F[A]

  def flatMap[A, B](value: F[A])(func: A => F[B]): F[B]

  def map[A, B](value: F[A])(func: A => B): F[B] =
    flatMap(value)(a => ???)
}

We need a function of type A => F[B] as the second parameter. We have two function building blocks available: the func parameter of type A => B and the pure function of type A => F[A]. Combining these gives us our result:

trait Monad[F[_]] {
  def pure[A](value: A): F[A]

  def flatMap[A, B](value: F[A])(func: A => F[B]): F[B]

  def map[A, B](value: F[A])(func: A => B): F[B] =
    flatMap(value)(a => pure(func(a)))
}

9.2 Monads in Cats

It’s time to give monads our standard Cats treatment. As usual we’ll look at the type class, instances, and syntax.

9.2.1 The Monad Type Class

The monad type class is cats.Monad. Monad extends two other type classes: FlatMap, which provides the flatMap method, and Applicative, which provides pure. Applicative also extends Functor, which gives every Monad a map method as we saw in the exercise above. We’ll discuss Applicatives in Chapter 11.

Here are some examples using pure and flatMap, and map directly:

import cats.Monad
val opt1 = Monad[Option].pure(3)
// opt1: Option[Int] = Some(value = 3)
val opt2 = Monad[Option].flatMap(opt1)(a => Some(a + 2))
// opt2: Option[Int] = Some(value = 5)
val opt3 = Monad[Option].map(opt2)(a => 100 * a)
// opt3: Option[Int] = Some(value = 500)

val list1 = Monad[List].pure(3)
// list1: List[Int] = List(3)
val list2 = Monad[List].
  flatMap(List(1, 2, 3))(a => List(a, a*10))
// list2: List[Int] = List(1, 10, 2, 20, 3, 30)
val list3 = Monad[List].map(list2)(a => a + 123)
// list3: List[Int] = List(124, 133, 125, 143, 126, 153)

Monad provides many other methods, including all of the methods from Functor. See the scaladoc for more information.

9.2.2 Default Instances

Cats provides instances for all the monads in the standard library (Option, List, Vector and so on). Cats also provides a Monad for Future. Unlike the methods on the Future class itself, the pure and flatMap methods on the monad can’t accept implicit ExecutionContext parameters (because the parameters aren’t part of the definitions in the Monad trait). To work around this, Cats requires us to have an ExecutionContext in scope when we summon a Monad for Future:

import scala.concurrent.*
import scala.concurrent.duration.*
val fm = Monad[Future]
// error:
// No given instance of type cats.Monad[scala.concurrent.Future] was found for parameter instance of method apply in object Monad.
// I found:
// 
//     cats.Invariant.catsInstancesForFuture(
//       /* missing */summon[scala.concurrent.ExecutionContext])
// 
// But no implicit values were found that match type scala.concurrent.ExecutionContext.
// val fm = Monad[Future]
//                      ^

Bringing the ExecutionContext into scope fixes the implicit resolution required to summon the instance:

import scala.concurrent.ExecutionContext.Implicits.global
val fm = Monad[Future]
// fm: Monad[[T >: Nothing <: Any] => Future[T]] = cats.instances.FutureInstances$$anon$1@1752a8b3

The Monad instance uses the captured ExecutionContext for subsequent calls to pure and flatMap:

val future = fm.flatMap(fm.pure(1))(x => fm.pure(x + 2))
Await.result(future, 1.second)
// res1: Int = 3

In addition to the above, Cats provides a host of new monads that we don’t have in the standard library. We’ll familiarise ourselves with some of these in a moment.

9.2.3 Monad Syntax

The syntax for monads comes from three places:

In practice it’s often easier to import everything in one go from cats.syntax.all.**. However, we’ll use the individual imports here for clarity.

We can use pure to construct instances of a monad. We’ll often need to specify the type parameter to disambiguate the particular instance we want.

import cats.syntax.applicative.* // for pure
1.pure[Option]
// res2: Option[Int] = Some(value = 1)
1.pure[List]
// res3: List[Int] = List(1)

It’s difficult to demonstrate the flatMap and map methods directly on Scala monads like Option and List, because they define their own explicit versions of those methods. Instead we’ll write a generic function that performs a calculation on parameters that come wrapped in a monad of the user’s choice:

import cats.Monad
import cats.syntax.functor.* // for map
import cats.syntax.flatMap.* // for flatMap

def sumSquare[F[_]: Monad](a: F[Int], b: F[Int]): F[Int] =
  a.flatMap(x => b.map(y => x*x + y*y))
sumSquare(Option(3), Option(4))
// res4: Option[Int] = Some(value = 25)
sumSquare(List(1, 2, 3), List(4, 5))
// res5: List[Int] = List(17, 26, 20, 29, 25, 34)

We can rewrite this code using for comprehensions. The compiler will “do the right thing” by rewriting our comprehension in terms of flatMap and map and inserting the correct conversions to use our Monad:

def sumSquare[F[_]: Monad](a: F[Int], b: F[Int]): F[Int] =
  for {
    x <- a
    y <- b
  } yield x*x + y*y
sumSquare(Option(3), Option(4))
// res7: Option[Int] = Some(value = 25)
sumSquare(List(1, 2, 3), List(4, 5))
// res8: List[Int] = List(17, 26, 20, 29, 25, 34)

That’s more or less everything we need to know about the generalities of monads in Cats. Now let’s take a look at some useful monad instances that we haven’t seen in the Scala standard library.

9.3 The Identity Monad

In the previous section we demonstrated Cats’ flatMap and map syntax by writing a method that abstracted over different monads:

import cats.Monad
import cats.syntax.functor.* // for map
import cats.syntax.flatMap.* // for flatMap

def sumSquare[F[_]: Monad](a: F[Int], b: F[Int]): F[Int] =
  for {
    x <- a
    y <- b
  } yield x*x + y*y

This method works well on Options and Lists but we can’t call it passing in plain values:

sumSquare(3, 4)
// error:
// Found:    (3 : Int)
// Required: ([_] =>> Any)[Int]
// Note that implicit conversions were not tried because the result of an implicit conversion
// must be more specific than ([_] =>> Any)[Int]
// sumSquare(3, 4)
//           ^
// error:
// Found:    (4 : Int)
// Required: ([_] =>> Any)[Int]
// Note that implicit conversions were not tried because the result of an implicit conversion
// must be more specific than ([_] =>> Any)[Int]
// sumSquare(3, 4)
//              ^

It would be incredibly useful if we could use sumSquare with parameters that were either in a monad or not in a monad at all. This would allow us to abstract over monadic and non-monadic code. Fortunately, Cats provides the Id type to bridge the gap:

import cats.Id
sumSquare(3 : Id[Int], 4 : Id[Int])
// res1: Int = 25

Id allows us to call our monadic method using plain values. However, the exact semantics are difficult to understand. We cast the parameters to sumSquare as Id[Int] and received an Id[Int] back as a result!

What’s going on? Here is the definition of Id to explain:

package cats

type Id[A] = A

Id is actually a type alias that turns an atomic type into a single-parameter type constructor. We can cast any value of any type to a corresponding Id:

"Dave" : Id[String]
// res2: String = "Dave"
123 : Id[Int]
// res3: Int = 123
List(1, 2, 3) : Id[List[Int]]
// res4: List[Int] = List(1, 2, 3)

Cats provides instances of various type classes for Id, including Functor and Monad. These let us call map, flatMap, and pure on plain values:

val a = Monad[Id].pure(3)
// a: Int = 3
val b = Monad[Id].flatMap(a)(_ + 1)
// b: Int = 4
import cats.syntax.functor.* // for map
import cats.syntax.flatMap.* // for flatMap
for {
  x <- a
  y <- b
} yield x + y
// res5: Int = 7

The ability to abstract over monadic and non-monadic code is extremely powerful. For example, we can run code asynchronously in production using Future and synchronously in test using Id. We’ll see this in our first case study in Chapter 17.

9.3.1 Exercise: Monadic Secret Identities

Implement pure, map, and flatMap for Id! What interesting discoveries do you uncover about the implementation?

Let’s start by defining the method signatures:

import cats.Id

def pure[A](value: A): Id[A] =
  ???

def map[A, B](initial: Id[A])(func: A => B): Id[B] =
  ???

def flatMap[A, B](initial: Id[A])(func: A => Id[B]): Id[B] =
  ???

Now let’s look at each method in turn. The pure operation creates an Id[A] from an A. But A and Id[A] are the same type! All we have to do is return the initial value:

def pure[A](value: A): Id[A] =
  value
pure(123)
// res7: Int = 123

The map method takes a parameter of type Id[A], applies a function of type A => B, and returns an Id[B]. But Id[A] is simply A and Id[B] is simply B! All we have to do is call the function—no boxing or unboxing required:

def map[A, B](initial: Id[A])(func: A => B): Id[B] =
  func(initial)
map(123)(_ * 2)
// res8: Int = 246

The final punch line is that, once we strip away the Id type constructors, flatMap and map are actually identical:

def flatMap[A, B](initial: Id[A])(func: A => Id[B]): Id[B] =
  func(initial)
flatMap(123)(_ * 2)
// res9: Int = 246

This ties in with our understanding of functors and monads as sequencing type classes. Each type class allows us to sequence operations ignoring some kind of complication. In the case of Id there is no complication, making map and flatMap the same thing.

Notice that we haven’t had to write type annotations in the method bodies above. The compiler is able to interpret values of type A as Id[A] and vice versa by the context in which they are used.

The only restriction we’ve seen to this is that Scala cannot unify types and type constructors when searching for given instances. Hence our need to re-type Int as Id[Int] in the call to sumSquare at the opening of this section:

sumSquare(3 : Id[Int], 4 : Id[Int])

9.4 Either

Let’s look at another useful monad: the Either type from the Scala standard library. In Scala 2.11 and earlier, many people didn’t consider Either a monad because it didn’t have map and flatMap methods. In Scala 2.12, however, Either became right biased.

9.4.1 Left and Right Bias

In Scala 2.11, Either had no default map or flatMap method. This made the Scala 2.11 version of Either inconvenient to use in for comprehensions. We had to insert calls to .right in every generator clause:

val either1: Either[String, Int] = Right(10)
val either2: Either[String, Int] = Right(32)
for {
  a <- either1.right
  b <- either2.right
} yield a + b

In Scala 2.12, Either was redesigned. The modern Either makes the decision that the right side represents the success case and thus supports map and flatMap directly. This makes for comprehensions much more pleasant:

for {
  a <- either1
  b <- either2
} yield a + b
// res1: Either[String, Int] = Right(value = 42)

Cats back-ports this behaviour to Scala 2.11 via the cats.syntax.either import, allowing us to use right-biased Either in all supported versions of Scala. In Scala 2.12+ we can either omit this import or leave it in place without breaking anything:

import cats.syntax.either.* // for map and flatMap

for {
  a <- either1
  b <- either2
} yield a + b

9.4.2 Creating Instances

In addition to creating instances of Left and Right directly, we can also import the asLeft and asRight extension methods from cats.syntax.either:

import cats.syntax.either.* // for asRight
val a = 3.asRight[String]
// a: Either[String, Int] = Right(value = 3)
val b = 4.asRight[String]
// b: Either[String, Int] = Right(value = 4)

for {
  x <- a
  y <- b
} yield x*x + y*y
// res3: Either[String, Int] = Right(value = 25)

These “smart constructors” have advantages over Left.apply and Right.apply because they return results of type Either instead of Left and Right. This helps avoid type inference problems caused by over-narrowing, like the issue in the example below:

def countPositive(nums: List[Int]) =
  nums.foldLeft(Right(0)) { (accumulator, num) =>
    if(num > 0) {
      accumulator.map(_ + 1)
    } else {
      Left("Negative. Stopping!")
    }
  }
// error:
// Found:    Either[Nothing, Int]
// Required: Right[Nothing, Int]
//       accumulator.map(_ + 1)
//       ^^^^^^^^^^^^^^^^^^^^^^
// error:
// Found:    Left[String, Any]
// Required: Right[Nothing, Int]
//       Left("Negative. Stopping!")
//       ^^^^^^^^^^^^^^^^^^^^^^^^^^^

This code fails to compile for two reasons:

  1. the compiler infers the type of the accumulator as Right instead of Either;
  2. we didn’t specify type parameters for Right.apply so the compiler infers the left parameter as Nothing.

Switching to asRight avoids both of these problems. asRight has a return type of Either, and allows us to completely specify the type with only one type parameter:

def countPositive(nums: List[Int]) =
  nums.foldLeft(0.asRight[String]) { (accumulator, num) =>
    if(num > 0) {
      accumulator.map(_ + 1)
    } else {
      Left("Negative. Stopping!")
    }
  }
countPositive(List(1, 2, 3))
// res5: Either[String, Int] = Right(value = 3)
countPositive(List(1, -2, 3))
// res6: Either[String, Int] = Left(value = "Negative. Stopping!")

cats.syntax.either adds some useful extension methods to the Either companion object. The catchOnly and catchNonFatal methods are great for capturing Exceptions as instances of Either:

Either.catchOnly[NumberFormatException]("foo".toInt)
// res7: Either[NumberFormatException, Int] = Left(
//   value = java.lang.NumberFormatException: For input string: "foo"
// )
Either.catchNonFatal(sys.error("Badness"))
// res8: Either[Throwable, Nothing] = Left(
//   value = java.lang.RuntimeException: Badness
// )

There are also methods for creating an Either from other data types:

Either.fromTry(scala.util.Try("foo".toInt))
// res9: Either[Throwable, Int] = Left(
//   value = java.lang.NumberFormatException: For input string: "foo"
// )
Either.fromOption[String, Int](None, "Badness")
// res10: Either[String, Int] = Left(value = "Badness")

9.4.3 Transforming Eithers

cats.syntax.either also adds some useful methods for instances of Either.

Users of Scala 2.11 or 2.12 can use orElse and getOrElse to extract values from the right side or return a default:

import cats.syntax.either.*
"Error".asLeft[Int].getOrElse(0)
// res11: Int = 0
"Error".asLeft[Int].orElse(2.asRight[String])
// res12: Either[String, Int] = Right(value = 2)

The ensure method allows us to check whether the right-hand value satisfies a predicate:

-1.asRight[String].ensure("Must be non-negative!")(_ > 0)
// res13: Either[String, Int] = Left(value = "Must be non-negative!")

The recover and recoverWith methods provide similar error handling to their namesakes on Future:

"error".asLeft[Int].recover {
  case _: String => -1
}
// res14: Either[String, Int] = Right(value = -1)

"error".asLeft[Int].recoverWith {
  case _: String => Right(-1)
}
// res15: Either[String, Int] = Right(value = -1)

There are leftMap and bimap methods to complement map:

"foo".asLeft[Int].leftMap(_.reverse)
// res16: Either[String, Int] = Left(value = "oof")
6.asRight[String].bimap(_.reverse, _ * 7)
// res17: Either[String, Int] = Right(value = 42)
"bar".asLeft[Int].bimap(_.reverse, _ * 7)
// res18: Either[String, Int] = Left(value = "rab")

The swap method lets us exchange left for right:

123.asRight[String]
// res19: Either[String, Int] = Right(value = 123)
123.asRight[String].swap
// res20: Either[Int, String] = Left(value = 123)

Finally, Cats adds a host of conversion methods: toOption, toList, toTry, toValidated, and so on.

9.4.4 Error Handling

Either is typically used to implement fail-fast error handling. We sequence computations using flatMap as usual. If one computation fails, the remaining computations are not run:

for {
  a <- 1.asRight[String]
  b <- 0.asRight[String]
  c <- if(b == 0) "DIV0".asLeft[Int]
       else (a / b).asRight[String]
} yield c * 100
// res21: Either[String, Int] = Left(value = "DIV0")

When using Either for error handling, we need to determine what type we want to use to represent errors. We could use Throwable for this:

type Result[A] = Either[Throwable, A]

This gives us similar semantics to scala.util.Try. The problem, however, is that Throwable is an extremely broad type. We have (almost) no idea about what type of error occurred.

Another approach is to define an algebraic data type to represent errors that may occur in our program:

enum LoginError {
  case UserNotFound(username: String)

  case PasswordIncorrect(username: String)

  case UnexpectedError 
}
case class User(username: String, password: String)

type LoginResult = Either[LoginError, User]

This approach solves the problems we saw with Throwable. It gives us a fixed set of expected error types and a catch-all for anything else that we didn’t expect. We also get the safety of exhaustivity checking on any pattern matching we do:

import LoginError.*

// Choose error-handling behaviour based on type:
def handleError(error: LoginError): Unit =
  error match {
    case UserNotFound(u) =>
      println(s"User not found: $u")

    case PasswordIncorrect(u) =>
      println(s"Password incorrect: $u")

    case UnexpectedError =>
      println(s"Unexpected error")
  }
val result1: LoginResult = User("dave", "passw0rd").asRight
// result1: Either[LoginError, User] = Right(
//   value = User(username = "dave", password = "passw0rd")
// )
val result2: LoginResult = UserNotFound("dave").asLeft
// result2: Either[LoginError, User] = Left(
//   value = UserNotFound(username = "dave")
// )

result1.fold(handleError, println)
// User(dave,passw0rd)
result2.fold(handleError, println)
// User not found: dave

9.4.5 Exercise: What is Best?

Is the error handling strategy in the previous examples well suited for all purposes? What other features might we want from error handling?

This is an open question. It’s also kind of a trick question—the answer depends on the semantics we’re looking for. Some points to ponder:

  • Error recovery is important when processing large jobs. We don’t want to run a job for a day and then find it failed on the last element.

  • Error reporting is equally important. We need to know what went wrong, not just that something went wrong.

  • In a number of cases, we want to collect all the errors, not just the first one we encountered. A typical example is validating a web form. It’s a far better experience to report all errors to the user when they submit a form than to report them one at a time.

9.5 Aside: Error Handling and MonadError

Cats provides an additional type class called MonadError that abstracts over Either-like data types that are used for error handling. MonadError provides extra operations for raising and handling errors.

This Section is Optional!

You won’t need to use MonadError unless you need to abstract over error handling monads. For example, you can use MonadError to abstract over Future and Try, or over Either and EitherT (which we will meet in Chapter 10).

If you don’t need this kind of abstraction right now, feel free to skip onwards to Section 9.6.

9.5.1 The MonadError Type Class

Here is a simplified version of the definition of MonadError:

package cats

trait MonadError[F[_], E] extends Monad[F] {
  // Lift an error into the `F` context:
  def raiseError[A](e: E): F[A]

  // Handle an error, potentially recovering from it:
  def handleErrorWith[A](fa: F[A])(f: E => F[A]): F[A]
  
  // Handle all errors, recovering from them:
  def handleError[A](fa: F[A])(f: E => A): F[A]

  // Test an instance of `F`,
  // failing if the predicate is not satisfied:
  def ensure[A](fa: F[A])(e: E)(f: A => Boolean): F[A]
}

MonadError is defined in terms of two type parameters:

To demonstrate how these parameters fit together, here’s an example where we instantiate the type class for Either:

import cats.MonadError

type ErrorOr[A] = Either[String, A]

val monadError = MonadError[ErrorOr, String]

ApplicativeError

In reality, MonadError extends another type class called ApplicativeError. However, we won’t encounter Applicatives until Chapter 11. The semantics are the same for each type class so we can ignore this detail for now.

9.5.2 Raising and Handling Errors

The two most important methods of MonadError are raiseError and handleErrorWith. raiseError is like the pure method for Monad except that it creates an instance representing a failure:

val success = monadError.pure(42)
// success: Either[String, Int] = Right(value = 42)
val failure = monadError.raiseError("Badness")
// failure: Either[String, Nothing] = Left(value = "Badness")

handleErrorWith is the complement of raiseError. It allows us to consume an error and (possibly) turn it into a success, similar to the recover method of Future:

monadError.handleErrorWith(failure) {
  case "Badness" =>
    monadError.pure("It's ok")

  case _ =>
    monadError.raiseError("It's not ok")
}
// res0: Either[String, String] = Right(value = "It's ok")

If we know we can handle all possible errors we can use handleWith.

monadError.handleError(failure) {
  case "Badness" => 42

  case _ => -1
}
// res1: Either[String, Int] = Right(value = 42)

There is another useful method called ensure that implements filter-like behaviour. We test the value of a successful monad with a predicate and specify an error to raise if the predicate returns false:

monadError.ensure(success)("Number too low!")(_ > 1000)
// res2: Either[String, Int] = Left(value = "Number too low!")

Cats provides syntax for raiseError and handleErrorWith via cats.syntax.applicativeError and ensure via cats.syntax.monadError:

import cats.syntax.applicative.*      // for pure
import cats.syntax.applicativeError.* // for raiseError etc
import cats.syntax.monadError.*       // for ensure
val success = 42.pure[ErrorOr]
// success: Either[String, Int] = Right(value = 42)
val failure = "Badness".raiseError[ErrorOr, Int]
// failure: Either[String, Int] = Left(value = "Badness")
failure.handleErrorWith{
  case "Badness" =>
    256.pure

  case _ =>
    ("It's not ok").raiseError
}
// res4: Either[String, Int] = Right(value = 256)
success.ensure("Number to low!")(_ > 1000)
// res5: Either[String, Int] = Left(value = "Number to low!")

There are other useful variants of these methods. See the source of cats.MonadError and cats.ApplicativeError for more information.

9.5.3 Instances of MonadError

Cats provides instances of MonadError for numerous data types including Either, Future, and Try. The instance for Either is customisable to any error type, whereas the instances for Future and Try always represent errors as Throwables:

import scala.util.Try

val exn: Throwable =
  new RuntimeException("It's all gone wrong")
exn.raiseError[Try, Int]
// res6: Try[Int] = Failure(
//   exception = java.lang.RuntimeException: It's all gone wrong
// )

9.5.4 Exercise: Abstracting

Implement a method validateAdult with the following signature

def validateAdult[F[_]](age: Int)(implicit me: MonadError[F, Throwable]): F[Int] =
  ???

When passed an age greater than or equal to 18 it should return that value as a success. Otherwise it should return a error represented as an IllegalArgumentException.

Here are some examples of use.

validateAdult[Try](18)
// res7: Try[Int] = Success(value = 18)
validateAdult[Try](8)
// res8: Try[Int] = Failure(
//   exception = java.lang.IllegalArgumentException: Age must be greater than or equal to 18
// )
type ExceptionOr[A] = Either[Throwable, A]
validateAdult[ExceptionOr](-1)
// res9: Either[Throwable, Int] = Left(
//   value = java.lang.IllegalArgumentException: Age must be greater than or equal to 18
// )

We can solve this using pure and raiseError. Note the use of type parameters to these methods, to aid type inference.

def validateAdult[F[_]](age: Int)(implicit me: MonadError[F, Throwable]): F[Int] =
  if(age >= 18) age.pure[F]
  else new IllegalArgumentException("Age must be greater than or equal to 18").raiseError[F, Int]

9.6 The Eval Monad

cats.Eval is a monad that allows us to abstract over different models of evaluation. We typically talk of two such models: eager and lazy, also called call-by-value and call-by-name respectively. Eval also allows for a result to be memoized, which gives us call-by-need evaluation.

Eval is also stack-safe, which means we can use it in very deep recursions without blowing up the stack.

9.6.1 Eager, Lazy, Memoized, Oh My!

What do these terms for models of evaluation mean? Let’s see some examples.

Let’s first look at Scala vals. We can see the evaluation model using a computation with a visible side-effect. In the following example, the code to compute the value of x happens at place where it is defined rather than on access. Accessing x recalls the stored value without re-running the code.

val x = {
  println("Computing X")
  math.random()
}
// Computing X
// x: Double = 0.5792767091094089

x // first access
// res0: Double = 0.5792767091094089
x // second access
// res1: Double = 0.5792767091094089

This is an example of call-by-value evaluation:

Let’s look at an example using a def. The code to compute y below is not run until we use it, and is re-run on every access:

def y = {
  println("Computing Y")
  math.random()
}

y // first access
// Computing Y
// res2: Double = 0.5795917038882374
y // second access
// Computing Y
// res3: Double = 0.5358221540864636

These are the properties of call-by-name evaluation:

Last but not least, lazy vals are an example of call-by-need evaluation. The code to compute z below is not run until we use it for the first time (lazy). The result is then cached and re-used on subsequent accesses (memoized):

lazy val z = {
  println("Computing Z")
  math.random()
}

z // first access
// Computing Z
// res4: Double = 0.4908591565434778
z // second access
// res5: Double = 0.4908591565434778

Let’s summarize. There are two properties of interest:

There are three possible combinations of these properties:

The final combination, eager and not memoized, is not possible.

9.6.2 Eval’s Models of Evaluation

Eval has three subtypes: Now, Always, and Later. They correspond to call-by-value, call-by-name, and call-by-need respectively. We construct these with three constructor methods, which create instances of the three classes and return them typed as Eval:

import cats.Eval
val now = Eval.now(math.random() + 1000)
// now: Eval[Double] = Now(value = 1000.6640380862943)
val always = Eval.always(math.random() + 3000)
// always: Eval[Double] = cats.Always@443b71
val later = Eval.later(math.random() + 2000)
// later: Eval[Double] = cats.Later@f9a4ace

We can extract the result of an Eval using its value method:

now.value
// res6: Double = 1000.6640380862943
always.value
// res7: Double = 3000.961091994659
later.value
// res8: Double = 2000.73816245745

Each type of Eval calculates its result using one of the evaluation models defined above. Eval.now captures a value right now. Its semantics are similar to a val—eager and memoized:

val x = Eval.now{
  println("Computing X")
  math.random()
}
// Computing X
// x: Eval[Double] = Now(value = 0.24752348384946177)

x.value // first access
// res10: Double = 0.24752348384946177
x.value // second access
// res11: Double = 0.24752348384946177

Eval.always captures a lazy computation, similar to a def:

val y = Eval.always{
  println("Computing Y")
  math.random()
}
// y: Eval[Double] = cats.Always@5b8a8b7f

y.value // first access
// Computing Y
// res12: Double = 0.5572796866993928
y.value // second access
// Computing Y
// res13: Double = 0.7542338094572812

Finally, Eval.later captures a lazy, memoized computation, similar to a lazy val:

val z = Eval.later{
  println("Computing Z")
  math.random()
}
// z: Eval[Double] = cats.Later@4447adec

z.value // first access
// Computing Z
// res14: Double = 0.46307146265616594
z.value // second access
// res15: Double = 0.46307146265616594

The three behaviours are summarized below:

Scala Cats Properties
val Now eager, memoized
def Always lazy, not memoized
lazy val Later lazy, memoized

9.6.3 Eval as a Monad

Like all monads, Eval's map and flatMap methods add computations to a chain. In this case, however, the chain is stored explicitly as a list of functions. The functions aren’t run until we call Eval's value method to request a result:

val greeting = Eval
  .always{ println("Step 1"); "Hello" }
  .map{ str => println("Step 2"); s"$str world" }
// greeting: Eval[String] = cats.Eval$$anon$4@2c84a9f8

greeting.value
// Step 1
// Step 2
// res16: String = "Hello world"

Note that, while the semantics of the originating Eval instances are maintained, mapping functions are always called lazily on demand (def semantics):

val ans = for {
  a <- Eval.now{ println("Calculating A"); 40 }
  b <- Eval.always{ println("Calculating B"); 2 }
} yield {
  println("Adding A and B")
  a + b
}
// Calculating A
// ans: Eval[Int] = cats.Eval$$anon$4@25852eae

ans.value // first access
// Calculating B
// Adding A and B
// res17: Int = 42
ans.value // second access
// Calculating B
// Adding A and B
// res18: Int = 42

Eval has a memoize method that allows us to memoize a chain of computations. The result of the chain up to the call to memoize is cached, whereas calculations after the call retain their original semantics:

val saying = Eval
  .always{ println("Step 1"); "The cat" }
  .map{ str => println("Step 2"); s"$str sat on" }
  .memoize
  .map{ str => println("Step 3"); s"$str the mat" }
// saying: Eval[String] = cats.Eval$$anon$4@29cebc14

saying.value // first access
// Step 1
// Step 2
// Step 3
// res19: String = "The cat sat on the mat"
saying.value // second access
// Step 3
// res20: String = "The cat sat on the mat"

9.6.4 Trampolining and Eval.defer

One useful property of Eval is that its map and flatMap methods are trampolined. This means we can nest calls to map and flatMap arbitrarily without consuming stack frames. We call this property “stack safety”.

For example, consider this function for calculating factorials:

def factorial(n: BigInt): BigInt =
  if(n == 1) n else n * factorial(n - 1)

It is relatively easy to make this method stack overflow:

factorial(50000)
// java.lang.StackOverflowError
//   ...

We can rewrite the method using Eval to make it stack safe:

def factorial(n: BigInt): Eval[BigInt] =
  if(n == 1) {
    Eval.now(n)
  } else {
    factorial(n - 1).map(_ * n)
  }
factorial(50000).value
// java.lang.StackOverflowError
//   ...

Oops! That didn’t work—our stack still blew up! This is because we’re still making all the recursive calls to factorial before we start working with Eval's map method. We can work around this using Eval.defer, which takes an existing instance of Eval and defers its evaluation. The defer method is trampolined like map and flatMap, so we can use it as a quick way to make an existing operation stack safe:

def factorial(n: BigInt): Eval[BigInt] =
  if(n == 1) {
    Eval.now(n)
  } else {
    Eval.defer(factorial(n - 1).map(_ * n))
  }
factorial(50000).value
// res: A very big value

Eval is a useful tool to enforce stack safety when working on very large computations and data structures. However, we must bear in mind that trampolining is not free. It avoids consuming stack by creating a chain of function objects on the heap. There are still limits on how deeply we can nest computations, but they are bounded by the size of the heap rather than the stack.

9.6.5 Exercise: Safer Folding using Eval

The naive implementation of foldRight below is not stack safe. Make it so using Eval:

def foldRight[A, B](as: List[A], acc: B)(fn: (A, B) => B): B =
  as match {
    case head :: tail =>
      fn(head, foldRight(tail, acc)(fn))
    case Nil =>
      acc
  }

The easiest way to fix this is to introduce a helper method called foldRightEval. This is essentially our original method with every occurrence of B replaced with Eval[B], and a call to Eval.defer to protect the recursive call:

import cats.Eval

def foldRightEval[A, B](as: List[A], acc: Eval[B])
    (fn: (A, Eval[B]) => Eval[B]): Eval[B] =
  as match {
    case head :: tail =>
      Eval.defer(fn(head, foldRightEval(tail, acc)(fn)))
    case Nil =>
      acc
  }

We can redefine foldRight simply in terms of foldRightEval and the resulting method is stack safe:

def foldRight[A, B](as: List[A], acc: B)(fn: (A, B) => B): B =
  foldRightEval(as, Eval.now(acc)) { (a, b) =>
    b.map(fn(a, _))
  }.value
foldRight((1 to 100000).toList, 0L)(_ + _)
// res24: Long = 5000050000L

9.7 The Writer Monad

cats.data.Writer is a monad that lets us carry a log along with a computation. We can use it to record messages, errors, or additional data about a computation, and extract the log alongside the final result.

One common use for Writers is recording sequences of steps in multi-threaded computations where standard imperative logging techniques can result in interleaved messages from different contexts. With Writer the log for the computation is tied to the result, so we can run concurrent computations without mixing logs.

Cats Data Types

Writer is the first data type we’ve seen from the cats.data package. This package provides instances of various type classes that produce useful semantics. Other examples from cats.data include the monad transformers that we will see in the next chapter, and the Validated type we will encounter in Chapter 11.

9.7.1 Creating and Unpacking Writers

A Writer[W, A] carries two values: a log of type W and a result of type A. We can create a Writer from values of each type as follows:

import cats.data.Writer
import cats.instances.vector._ // for Monoid
Writer(Vector(
  "It was the best of times",
  "it was the worst of times"
), 1859)
// res0: WriterT[Id, Vector[String], Int] = WriterT(
//   run = (Vector("It was the best of times", "it was the worst of times"), 1859)
// )

Notice that the type reported on the console is actually WriterT[Id, Vector[String], Int] instead of Writer[Vector[String], Int] as we might expect. In the spirit of code reuse, Cats implements Writer in terms of another type, WriterT. WriterT is an example of a new concept called a monad transformer, which we will cover in the next chapter.

Let’s try to ignore this detail for now. Writer is a type alias for WriterT, so we can read types like WriterT[Id, W, A] as Writer[W, A]:

type Writer[W, A] = WriterT[Id, W, A]

For convenience, Cats provides a way of creating Writers specifying only the log or the result. If we only have a result we can use the standard pure syntax. To do this we must have a Monoid[W] in scope so Cats knows how to produce an empty log:

import cats.instances.vector._   // for Monoid
import cats.syntax.applicative._ // for pure

type Logged[A] = Writer[Vector[String], A]
123.pure[Logged]
// res1: WriterT[Id, Vector[String], Int] = WriterT(run = (Vector(), 123))

If we have a log and no result we can create a Writer[Unit] using the tell syntax from cats.syntax.writer:

import cats.syntax.writer._ // for tell
Vector("msg1", "msg2", "msg3").tell
// res2: WriterT[Id, Vector[String], Unit] = WriterT(
//   run = (Vector("msg1", "msg2", "msg3"), ())
// )

If we have both a result and a log, we can either use Writer.apply or we can use the writer syntax from cats.syntax.writer:

import cats.syntax.writer._ // for writer
val a = Writer(Vector("msg1", "msg2", "msg3"), 123)
// a: WriterT[Id, Vector[String], Int] = WriterT(
//   run = (Vector("msg1", "msg2", "msg3"), 123)
// )
val b = 123.writer(Vector("msg1", "msg2", "msg3"))
// b: WriterT[Id, Vector[String], Int] = WriterT(
//   run = (Vector("msg1", "msg2", "msg3"), 123)
// )

We can extract the result and log from a Writer using the value and written methods respectively:

val aResult: Int =
  a.value
// aResult: Int = 123
val aLog: Vector[String] =
  a.written
// aLog: Vector[String] = Vector("msg1", "msg2", "msg3")

We can extract both values at the same time using the run method:

val (log, result) = b.run
// log: Vector[String] = Vector("msg1", "msg2", "msg3")
// result: Int = 123

9.7.2 Composing and Transforming Writers

The log in a Writer is preserved when we map or flatMap over it. flatMap appends the logs from the source Writer and the result of the user’s sequencing function. For this reason it’s good practice to use a log type that has an efficient append and concatenate operations, such as a Vector:

val writer1 = for {
  a <- 10.pure[Logged]
  _ <- Vector("a", "b", "c").tell
  b <- 32.writer(Vector("x", "y", "z"))
} yield a + b
// writer1: WriterT[Id, Vector[String], Int] = WriterT(
//   run = (Vector("a", "b", "c", "x", "y", "z"), 42)
// )

writer1.run
// res3: Tuple2[Vector[String], Int] = (
//   Vector("a", "b", "c", "x", "y", "z"),
//   42
// )

In addition to transforming the result with map and flatMap, we can transform the log in a Writer with the mapWritten method:

val writer2 = writer1.mapWritten(_.map(_.toUpperCase))
// writer2: WriterT[Id, Vector[String], Int] = WriterT(
//   run = (Vector("A", "B", "C", "X", "Y", "Z"), 42)
// )

writer2.run
// res4: Tuple2[Vector[String], Int] = (
//   Vector("A", "B", "C", "X", "Y", "Z"),
//   42
// )

We can transform both log and result simultaneously using bimap or mapBoth. bimap takes two function parameters, one for the log and one for the result. mapBoth takes a single function that accepts two parameters:

val writer3 = writer1.bimap(
  log => log.map(_.toUpperCase),
  res => res * 100
)
// writer3: WriterT[Id, Vector[String], Int] = WriterT(
//   run = (Vector("A", "B", "C", "X", "Y", "Z"), 4200)
// )

writer3.run
// res5: Tuple2[Vector[String], Int] = (
//   Vector("A", "B", "C", "X", "Y", "Z"),
//   4200
// )

val writer4 = writer1.mapBoth { (log, res) =>
  val log2 = log.map(_ + "!")
  val res2 = res * 1000
  (log2, res2)
}
// writer4: WriterT[Id, Vector[String], Int] = WriterT(
//   run = (Vector("a!", "b!", "c!", "x!", "y!", "z!"), 42000)
// )

writer4.run
// res6: Tuple2[Vector[String], Int] = (
//   Vector("a!", "b!", "c!", "x!", "y!", "z!"),
//   42000
// )

Finally, we can clear the log with the reset method and swap log and result with the swap method:

val writer5 = writer1.reset
// writer5: WriterT[Id, Vector[String], Int] = WriterT(run = (Vector(), 42))

writer5.run
// res7: Tuple2[Vector[String], Int] = (Vector(), 42)

val writer6 = writer1.swap
// writer6: WriterT[Id, Int, Vector[String]] = WriterT(
//   run = (42, Vector("a", "b", "c", "x", "y", "z"))
// )

writer6.run
// res8: Tuple2[Int, Vector[String]] = (
//   42,
//   Vector("a", "b", "c", "x", "y", "z")
// )

9.7.3 Exercise: Show Your Working

Writers are useful for logging operations in multi-threaded environments. Let’s confirm this by computing (and logging) some factorials.

The factorial function below computes a factorial and prints out the intermediate steps as it runs. The slowly helper function ensures this takes a while to run, even on the very small examples below:

def slowly[A](body: => A) =
  try body finally Thread.sleep(100)

def factorial(n: Int): Int = {
  val ans = slowly(if(n == 0) 1 else n * factorial(n - 1))
  println(s"fact $n $ans")
  ans
}

Here’s the output—a sequence of monotonically increasing values:

factorial(5)
// fact 0 1
// fact 1 1
// fact 2 2
// fact 3 6
// fact 4 24
// fact 5 120
// res9: Int = 120

If we start several factorials in parallel, the log messages can become interleaved on standard out. This makes it difficult to see which messages come from which computation:

import scala.concurrent._
import scala.concurrent.ExecutionContext.Implicits._
import scala.concurrent.duration._

Await.result(Future.sequence(Vector(
  Future(factorial(5)),
  Future(factorial(5))
)), 5.seconds)
// fact 0 1
// fact 0 1
// fact 1 1
// fact 1 1
// fact 2 2
// fact 2 2
// fact 3 6
// fact 3 6
// fact 4 24
// fact 4 24
// fact 5 120
// fact 5 120
// res: scala.collection.immutable.Vector[Int] =
//   Vector(120, 120)

Rewrite factorial so it captures the log messages in a Writer. Demonstrate that this allows us to reliably separate the logs for concurrent computations.

We’ll start by defining a type alias for Writer so we can use it with pure syntax:

import cats.data.Writer
import cats.instances.vector._
import cats.syntax.applicative._ // for pure

type Logged[A] = Writer[Vector[String], A]
42.pure[Logged]
// res11: WriterT[Id, Vector[String], Int] = WriterT(run = (Vector(), 42))

We’ll import the tell syntax as well:

import cats.syntax.writer._ // for tell
Vector("Message").tell
// res12: WriterT[Id, Vector[String], Unit] = WriterT(
//   run = (Vector("Message"), ())
// )

Finally, we’ll import the Semigroup instance for Vector. We need this to map and flatMap over Logged:

import cats.instances.vector._ // for Monoid
41.pure[Logged].map(_ + 1)
// res13: WriterT[Id, Vector[String], Int] = WriterT(run = (Vector(), 42))

With these in scope, the definition of factorial becomes:

def factorial(n: Int): Logged[Int] =
  for {
    ans <- if(n == 0) {
             1.pure[Logged]
           } else {
             slowly(factorial(n - 1).map(_ * n))
           }
    _   <- Vector(s"fact $n $ans").tell
  } yield ans

When we call factorial, we now have to run the return value to extract the log and our factorial:

val (log, res) = factorial(5).run
// log: Vector[String] = Vector(
//   "fact 0 1",
//   "fact 1 1",
//   "fact 2 2",
//   "fact 3 6",
//   "fact 4 24",
//   "fact 5 120"
// )
// res: Int = 120

We can run several factorials in parallel as follows, capturing their logs independently without fear of interleaving:

Await.result(Future.sequence(Vector(
  Future(factorial(5)),
  Future(factorial(5))
)).map(_.map(_.written)), 5.seconds)
// res: scala.collection.immutable.Vector[cats.Id[Vector[String]]] = 
//   Vector(
//     Vector(fact 0 1, fact 1 1, fact 2 2, fact 3 6, fact 4 24, fact 5 120), 
//     Vector(fact 0 1, fact 1 1, fact 2 2, fact 3 6, fact 4 24, fact 5 120)
//   )

9.8 The Reader Monad

cats.data.Reader is a monad that allows us to sequence operations that depend on some input. Instances of Reader wrap up functions of one argument, providing us with useful methods for composing them.

One common use for Readers is dependency injection. If we have a number of operations that all depend on some external configuration, we can chain them together using a Reader to produce one large operation that accepts the configuration as a parameter and runs our program in the order specified.

9.8.1 Creating and Unpacking Readers

We can create a Reader[A, B] from a function A => B using the Reader.apply constructor:

import cats.data.Reader
final case class Cat(name: String, favoriteFood: String)

val catName: Reader[Cat, String] =
  Reader(cat => cat.name)
// catName: Kleisli[Id, Cat, String] = Kleisli(
//   run = repl.MdocSession$MdocApp0$$$Lambda$21158/0x00000008052e1040@18d3ebcb
// )

We can extract the function again using the Reader's run method and call it using apply as usual:

catName.run(Cat("Garfield", "lasagne"))
// res1: String = "Garfield"

So far so simple, but what advantage do Readers give us over the raw functions?

9.8.2 Composing Readers

The power of Readers comes from their map and flatMap methods, which represent different kinds of function composition. We typically create a set of Readers that accept the same type of configuration, combine them with map and flatMap, and then call run to inject the config at the end.

The map method simply extends the computation in the Reader by passing its result through a function:

val greetKitty: Reader[Cat, String] =
  catName.map(name => s"Hello ${name}")
greetKitty.run(Cat("Heathcliff", "junk food"))
// res2: String = "Hello Heathcliff"

The flatMap method is more interesting. It allows us to combine readers that depend on the same input type. To illustrate this, let’s extend our greeting example to also feed the cat:

val feedKitty: Reader[Cat, String] =
  Reader(cat => s"Have a nice bowl of ${cat.favoriteFood}")

val greetAndFeed: Reader[Cat, String] =
  for {
    greet <- greetKitty
    feed  <- feedKitty
  } yield s"$greet. $feed."
greetAndFeed(Cat("Garfield", "lasagne"))
// res3: String = "Hello Garfield. Have a nice bowl of lasagne."
greetAndFeed(Cat("Heathcliff", "junk food"))
// res4: String = "Hello Heathcliff. Have a nice bowl of junk food."

9.8.3 Exercise: Hacking on Readers

The classic use of Readers is to build programs that accept a configuration as a parameter. Let’s ground this with a complete example of a simple login system. Our configuration will consist of two databases: a list of valid users and a list of their passwords:

final case class Db(
  usernames: Map[Int, String],
  passwords: Map[String, String]
)

Start by creating a type alias DbReader for a Reader that consumes a Db as input. This will make the rest of our code shorter.

Our type alias fixes the Db type but leaves the result type flexible:

type DbReader[A] = Reader[Db, A]

Now create methods that generate DbReaders to look up the username for an Int user ID, and look up the password for a String username. The type signatures should be as follows:

def findUsername(userId: Int): DbReader[Option[String]] =
  ???

def checkPassword(
      username: String,
      password: String): DbReader[Boolean] =
  ???

Remember: the idea is to leave injecting the configuration until last. This means setting up functions that accept the config as a parameter and check it against the concrete user info we have been given:

def findUsername(userId: Int): DbReader[Option[String]] =
  Reader(db => db.usernames.get(userId))

def checkPassword(
      username: String,
      password: String): DbReader[Boolean] =
  Reader(db => db.passwords.get(username).contains(password))

Finally create a checkLogin method to check the password for a given user ID. The type signature should be as follows:

def checkLogin(
      userId: Int,
      password: String): DbReader[Boolean] =
  ???

As you might expect, here we use flatMap to chain findUsername and checkPassword. We use pure to lift a Boolean to a DbReader[Boolean] when the username is not found:

import cats.syntax.applicative._ // for pure

def checkLogin(
      userId: Int,
      password: String): DbReader[Boolean] =
  for {
    username   <- findUsername(userId)
    passwordOk <- username.map { username =>
                    checkPassword(username, password)
                  }.getOrElse {
                    false.pure[DbReader]
                  }
  } yield passwordOk

You should be able to use checkLogin as follows:

val users = Map(
  1 -> "dade",
  2 -> "kate",
  3 -> "margo"
)

val passwords = Map(
  "dade"  -> "zerocool",
  "kate"  -> "acidburn",
  "margo" -> "secret"
)

val db = Db(users, passwords)
checkLogin(1, "zerocool").run(db)
// res7: Boolean = true
checkLogin(4, "davinci").run(db)
// res8: Boolean = false

9.8.4 When to Use Readers?

Readers provide a tool for doing dependency injection. We write steps of our program as instances of Reader, chain them together with map and flatMap, and build a function that accepts the dependency as input.

There are many ways of implementing dependency injection in Scala, from simple techniques like methods with multiple parameter lists, through implicit parameters and type classes, to complex techniques like the cake pattern and DI frameworks.

Readers are most useful in situations where:

By representing the steps of our program as Readers we can test them as easily as pure functions, plus we gain access to the map and flatMap combinators.

For more complicated problems where we have lots of dependencies, or where a program isn’t easily represented as a pure function, other dependency injection techniques tend to be more appropriate.

Kleisli Arrows

You may have noticed from console output that Reader is implemented in terms of another type called Kleisli. Kleisli arrows provide a more general form of Reader that generalise over the type constructor of the result type. We will encounter Kleislis again in Chapter 10.

9.9 The State Monad

cats.data.State allows us to pass additional state around as part of a computation. We define State instances representing atomic state operations and thread them together using map and flatMap. In this way we can model mutable state in a purely functional way, without using actual mutation.

9.9.1 Creating and Unpacking State

Boiled down to their simplest form, instances of State[S, A] represent functions of type S => (S, A). S is the type of the state and A is the type of the result.

import cats.data.State
val a = State[Int, String]{ state =>
  (state, s"The state is $state")
}

In other words, an instance of State is a function that does two things:

We can “run” our monad by supplying an initial state. State provides three methods—run, runS, and runA—that return different combinations of state and result. Each method returns an instance of Eval, which State uses to maintain stack safety. We call the value method as usual to extract the actual result:

// Get the state and the result:
val (state, result) = a.run(10).value
// state: Int = 10
// result: String = "The state is 10"

// Get the state, ignore the result:
val justTheState = a.runS(10).value
// justTheState: Int = 10

// Get the result, ignore the state:
val justTheResult = a.runA(10).value
// justTheResult: String = "The state is 10"

9.9.2 Composing and Transforming State

As we’ve seen with Reader and Writer, the power of the State monad comes from combining instances. The map and flatMap methods thread the state from one instance to another. Each individual instance represents an atomic state transformation, and their combination represents a complete sequence of changes:

val step1 = State[Int, String]{ num =>
  val ans = num + 1
  (ans, s"Result of step1: $ans")
}

val step2 = State[Int, String]{ num =>
  val ans = num * 2
  (ans, s"Result of step2: $ans")
}

val both = for {
  a <- step1
  b <- step2
} yield (a, b)
val (state, result) = both.run(20).value
// state: Int = 42
// result: Tuple2[String, String] = (
//   "Result of step1: 21",
//   "Result of step2: 42"
// )

As you can see, in this example the final state is the result of applying both transformations in sequence. State is threaded from step to step even though we don’t interact with it in the for comprehension.

The general model for using the State monad is to represent each step of a computation as an instance and compose the steps using the standard monad operators. Cats provides several convenience constructors for creating primitive steps:

val getDemo = State.get[Int]
// getDemo: IndexedStateT[[A >: Nothing <: Any] => Eval[A], Int, Int, Int] = cats.data.IndexedStateT@cb2c857
getDemo.run(10).value
// res1: Tuple2[Int, Int] = (10, 10)

val setDemo = State.set[Int](30)
// setDemo: IndexedStateT[[A >: Nothing <: Any] => Eval[A], Int, Int, Unit] = cats.data.IndexedStateT@3e09b0ba
setDemo.run(10).value
// res2: Tuple2[Int, Unit] = (30, ())

val pureDemo = State.pure[Int, String]("Result")
// pureDemo: IndexedStateT[[A >: Nothing <: Any] => Eval[A], Int, Int, String] = cats.data.IndexedStateT@2141f33b
pureDemo.run(10).value
// res3: Tuple2[Int, String] = (10, "Result")

val inspectDemo = State.inspect[Int, String](x => s"${x}!")
// inspectDemo: IndexedStateT[[A >: Nothing <: Any] => Eval[A], Int, Int, String] = cats.data.IndexedStateT@159696df
inspectDemo.run(10).value
// res4: Tuple2[Int, String] = (10, "10!")

val modifyDemo = State.modify[Int](_ + 1)
// modifyDemo: IndexedStateT[[A >: Nothing <: Any] => Eval[A], Int, Int, Unit] = cats.data.IndexedStateT@5ea720dc
modifyDemo.run(10).value
// res5: Tuple2[Int, Unit] = (11, ())

We can assemble these building blocks using a for comprehension. We typically ignore the result of intermediate stages that only represent transformations on the state:

import cats.data.State
import State._
val program: State[Int, (Int, Int, Int)] = for {
  a <- get[Int]
  _ <- set[Int](a + 1)
  b <- get[Int]
  _ <- modify[Int](_ + 1)
  c <- inspect[Int, Int](_ * 1000)
} yield (a, b, c)
// program: IndexedStateT[[A >: Nothing <: Any] => Eval[A], Int, Int, Tuple3[Int, Int, Int]] = cats.data.IndexedStateT@4898772f

val (state, result) = program.run(1).value
// state: Int = 3
// result: Tuple3[Int, Int, Int] = (1, 2, 3000)

9.9.3 Exercise: Post-Order Calculator

The State monad allows us to implement simple interpreters for complex expressions, passing the values of mutable registers along with the result. We can see a simple example of this by implementing a calculator for post-order integer arithmetic expressions.

In case you haven’t heard of post-order expressions before (don’t worry if you haven’t), they are a mathematical notation where we write the operator after its operands. So, for example, instead of writing 1 + 2 we would write:

1 2 +

Although post-order expressions are difficult for humans to read, they are easy to evaluate in code. All we need to do is traverse the symbols from left to right, carrying a stack of operands with us as we go:

This allows us to evaluate complex expressions without using parentheses. For example, we can evaluate (1 + 2) * 3) as follows:

1 2 + 3 * // see 1, push onto stack
2 + 3 *   // see 2, push onto stack
+ 3 *     // see +, pop 1 and 2 off of stack,
          //        push (1 + 2) = 3 in their place
3 3 *     // see 3, push onto stack
3 *       // see 3, push onto stack
*         // see *, pop 3 and 3 off of stack,
          //        push (3 * 3) = 9 in their place

Let’s write an interpreter for these expressions. We can parse each symbol into a State instance representing a transformation on the stack and an intermediate result. The State instances can be threaded together using flatMap to produce an interpreter for any sequence of symbols.

Start by writing a function evalOne that parses a single symbol into an instance of State. Use the code below as a template. Don’t worry about error handling for now—if the stack is in the wrong configuration, it’s OK to throw an exception.

import cats.data.State

type CalcState[A] = State[List[Int], A]

def evalOne(sym: String): CalcState[Int] = ???

If this seems difficult, think about the basic form of the State instances you’re returning. Each instance represents a functional transformation from a stack to a pair of a stack and a result. You can ignore any wider context and focus on just that one step:

State[List[Int], Int] { oldStack =>
  val newStack = someTransformation(oldStack)
  val result   = someCalculation
  (newStack, result)
}

Feel free to write your Stack instances in this form or as sequences of the convenience constructors we saw above.

The stack operation required is different for operators and operands. For clarity we’ll implement evalOne in terms of two helper functions, one for each case:

def evalOne(sym: String): CalcState[Int] =
  sym match {
    case "+" => operator(_ + _)
    case "-" => operator(_ - _)
    case "*" => operator(_ * _)
    case "/" => operator(_ / _)
    case num => operand(num.toInt)
  }

Let’s look at operand first. All we have to do is push a number onto the stack. We also return the operand as an intermediate result:

def operand(num: Int): CalcState[Int] =
  State[List[Int], Int] { stack =>
    (num :: stack, num)
  }

The operator function is a little more complex. We have to pop two operands off the stack (having the second operand at the top of the stack)i and push the result in their place. The code can fail if the stack doesn’t have enough operands on it, but the exercise description allows us to throw an exception in this case:

def operator(func: (Int, Int) => Int): CalcState[Int] =
  State[List[Int], Int] {
    case b :: a :: tail =>
      val ans = func(a, b)
      (ans :: tail, ans)

    case _ =>
      sys.error("Fail!")
  }

evalOne allows us to evaluate single-symbol expressions as follows. We call runA supplying Nil as an initial stack, and call value to unpack the resulting Eval instance:

evalOne("42").runA(Nil).value
// res10: Int = 42

We can represent more complex programs using evalOne, map, and flatMap. Note that most of the work is happening on the stack, so we ignore the results of the intermediate steps for evalOne("1") and evalOne("2"):

val program = for {
  _   <- evalOne("1")
  _   <- evalOne("2")
  ans <- evalOne("+")
} yield ans
// program: IndexedStateT[[A >: Nothing <: Any] => Eval[A], List[Int], List[Int], Int] = cats.data.IndexedStateT@68c4e73c

program.runA(Nil).value
// res11: Int = 3

Generalise this example by writing an evalAll method that computes the result of a List[String]. Use evalOne to process each symbol, and thread the resulting State monads together using flatMap. Your function should have the following signature:

def evalAll(input: List[String]): CalcState[Int] =
  ???

We implement evalAll by folding over the input. We start with a pure CalcState that returns 0 if the list is empty. We flatMap at each stage, ignoring the intermediate results as we saw in the example:

import cats.syntax.applicative._ // for pure

def evalAll(input: List[String]): CalcState[Int] =
  input.foldLeft(0.pure[CalcState]) { (a, b) =>
    a.flatMap(_ => evalOne(b))
  }

We can use evalAll to conveniently evaluate multi-stage expressions:

val multistageProgram = evalAll(List("1", "2", "+", "3", "*"))
// multistageProgram: IndexedStateT[[A >: Nothing <: Any] => Eval[A], List[Int], List[Int], Int] = cats.data.IndexedStateT@24744766

multistageProgram.runA(Nil).value
// res13: Int = 9

Because evalOne and evalAll both return instances of State, we can thread these results together using flatMap. evalOne produces a simple stack transformation and evalAll produces a complex one, but they’re both pure functions and we can use them in any order as many times as we like:

val biggerProgram = for {
  _   <- evalAll(List("1", "2", "+"))
  _   <- evalAll(List("3", "4", "+"))
  ans <- evalOne("*")
} yield ans
// biggerProgram: IndexedStateT[[A >: Nothing <: Any] => Eval[A], List[Int], List[Int], Int] = cats.data.IndexedStateT@36e9d685

biggerProgram.runA(Nil).value
// res14: Int = 21

Complete the exercise by implementing an evalInput function that splits an input String into symbols, calls evalAll, and runs the result with an initial stack.

We’ve done all the hard work now. All we need to do is split the input into terms and call runA and value to unpack the result:

def evalInput(input: String): Int =
  evalAll(input.split(" ").toList).runA(Nil).value
evalInput("1 2 + 3 4 + *")
// res15: Int = 21

9.10 Defining Custom Monads

We can define a Monad for a custom type by providing implementations of three methods: flatMap, pure, and a method we haven’t seen yet called tailRecM. Here is an implementation of Monad for Option as an example:

import cats.Monad
import scala.annotation.tailrec

val optionMonad = new Monad[Option] {
  def flatMap[A, B](opt: Option[A])
      (fn: A => Option[B]): Option[B] =
    opt.flatMap(fn)

  def pure[A](opt: A): Option[A] =
    Some(opt)

  @tailrec
  def tailRecM[A, B](a: A)(fn: A => Option[Either[A, B]]): Option[B] = {
    fn(a) match {
      case None           => None
      case Some(Left(a1)) => tailRecM(a1)(fn)
      case Some(Right(b)) => Some(b)
    }
  }
}

The tailRecM method is an optimisation used in Cats to limit the amount of stack space consumed by nested calls to flatMap. The technique comes from a 2015 paper by PureScript creator Phil Freeman. The method should recursively call itself until the result of fn returns a Right.

To motivate its use let’s use the following example: Suppose we want to write a method that calls a function until the function indicates it should stop. The function will return a monad instance because, as we know, monads represent sequencing and many monads have some notion of stopping.

We can write this method in terms of flatMap.

import cats.syntax.flatMap._ // For flatMap

def retry[F[_]: Monad, A](start: A)(f: A => F[A]): F[A] =
  f(start).flatMap{ a =>
    retry(a)(f)
  }

Unfortunately it is not stack-safe. It works for small input.

import cats.instances.option._

retry(100)(a => if(a == 0) None else Some(a - 1))
// res1: Option[Int] = None

but if we try large input we get a StackOverflowError.

retry(100000)(a => if(a == 0) None else Some(a - 1))
// KABLOOIE!!!!

We can instead rewrite this method using tailRecM.

import cats.syntax.functor._ // for map

def retryTailRecM[F[_]: Monad, A](start: A)(f: A => F[A]): F[A] =
  Monad[F].tailRecM(start){ a =>
    f(a).map(a2 => Left(a2))
  }

Now it runs successfully no matter how many time we recurse.

retryTailRecM(100000)(a => if(a == 0) None else Some(a - 1))
// res2: Option[Int] = None

It’s important to note that we have to explicitly call tailRecM. There isn’t a code transformation that will convert non-tail recursive code into tail recursive code that uses tailRecM. However there are several utilities provided by the Monad type class that makes these kinds of methods easier to write. For example, we can rewrite retry in terms of iterateWhileM and we don’t have to explicitly call tailRecM.

import cats.syntax.monad._ // for iterateWhileM

def retryM[F[_]: Monad, A](start: A)(f: A => F[A]): F[A] =
  start.iterateWhileM(f)(a => true)
retryM(100000)(a => if(a == 0) None else Some(a - 1))
// res3: Option[Int] = None

We’ll see more methods that use tailRecM in Section 12.1.

All of the built-in monads in Cats have tail-recursive implementations of tailRecM, although writing one for custom monads can be a challenge… as we shall see.

9.10.1 Exercise: Branching out Further with Monads

Let’s write a Monad for our Tree data type from last chapter. Here’s the type again:

sealed trait Tree[+A]

final case class Branch[A](left: Tree[A], right: Tree[A])
  extends Tree[A]

final case class Leaf[A](value: A) extends Tree[A]

def branch[A](left: Tree[A], right: Tree[A]): Tree[A] =
  Branch(left, right)

def leaf[A](value: A): Tree[A] =
  Leaf(value)

Verify that the code works on instances of Branch and Leaf, and that the Monad provides Functor-like behaviour for free.

Also verify that having a Monad in scope allows us to use for comprehensions, despite the fact that we haven’t directly implemented flatMap or map on Tree.

Don’t feel you have to make tailRecM tail-recursive. Doing so is quite difficult. We’ve included both tail-recursive and non-tail-recursive implementations in the solutions so you can check your work.

The code for flatMap is similar to the code for map. Again, we recurse down the structure and use the results from func to build a new Tree.

The code for tailRecM is fairly complex regardless of whether we make it tail-recursive or not.

If we follow the types, the non-tail-recursive solution falls out:

import cats.Monad

implicit val treeMonad: Monad[Tree] = new Monad[Tree] {
  def pure[A](value: A): Tree[A] =
    Leaf(value)

  def flatMap[A, B](tree: Tree[A])
      (func: A => Tree[B]): Tree[B] =
    tree match {
      case Branch(l, r) =>
        Branch(flatMap(l)(func), flatMap(r)(func))
      case Leaf(value)  =>
        func(value)
    }

  def tailRecM[A, B](a: A)(func: A => Tree[Either[A, B]]): Tree[B] = {
    flatMap(func(a)) {
      case Left(value) =>
        tailRecM(value)(func)
      case Right(value) =>
        Leaf(value)
    }
  }
}

The solution above is perfectly fine for this exercise. Its only downside is that Cats cannot make guarantees about stack safety.

The tail-recursive solution is much harder to write. We adapted this solution from this Stack Overflow post by Nazarii Bardiuk. It involves an explicit depth first traversal of the tree, maintaining an open list of nodes to visit and a closed list of nodes to use to reconstruct the tree:

import cats.Monad
import scala.annotation.tailrec

implicit val treeMonad: Monad[Tree] = new Monad[Tree] {
  def pure[A](value: A): Tree[A] =
    Leaf(value)

  def flatMap[A, B](tree: Tree[A])
      (func: A => Tree[B]): Tree[B] =
    tree match {
      case Branch(l, r) =>
        Branch(flatMap(l)(func), flatMap(r)(func))
      case Leaf(value)  =>
        func(value)
    }

  def tailRecM[A, B](arg: A)
      (func: A => Tree[Either[A, B]]): Tree[B] = {
    @tailrec
    def loop(
          open: List[Tree[Either[A, B]]],
          closed: List[Option[Tree[B]]]): List[Tree[B]] =
      open match {
        case Branch(l, r) :: next =>
          loop(l :: r :: next, None :: closed)

        case Leaf(Left(value)) :: next =>
          loop(func(value) :: next, closed)

        case Leaf(Right(value)) :: next =>
          loop(next, Some(pure(value)) :: closed)

        case Nil =>
          closed.foldLeft(Nil: List[Tree[B]]) { (acc, maybeTree) =>
            maybeTree.map(_ :: acc).getOrElse {
              acc match {
                case left :: right :: tail => branch(left, right) :: tail
              }
            }
          }
      }

    loop(List(func(arg)), Nil).head
  }
}

Regardless of which version of tailRecM we define, we can use our Monad to flatMap and map on Trees:

import cats.syntax.functor._ // for map
import cats.syntax.flatMap._ // for flatMap
branch(leaf(100), leaf(200)).
  flatMap(x => branch(leaf(x - 1), leaf(x + 1)))
// res5: Tree[Int] = Branch(
//   left = Branch(left = Leaf(value = 99), right = Leaf(value = 101)),
//   right = Branch(left = Leaf(value = 199), right = Leaf(value = 201))
// )

We can also transform Trees using for comprehensions:

for {
  a <- branch(leaf(100), leaf(200))
  b <- branch(leaf(a - 10), leaf(a + 10))
  c <- branch(leaf(b - 1), leaf(b + 1))
} yield c
// res6: Tree[Int] = Branch(
//   left = Branch(
//     left = Branch(left = Leaf(value = 89), right = Leaf(value = 91)),
//     right = Branch(left = Leaf(value = 109), right = Leaf(value = 111))
//   ),
//   right = Branch(
//     left = Branch(left = Leaf(value = 189), right = Leaf(value = 191)),
//     right = Branch(left = Leaf(value = 209), right = Leaf(value = 211))
//   )
// )

The monad for Option provides fail-fast semantics. The monad for List provides concatenation semantics. What are the semantics of flatMap for a binary tree? Every node in the tree has the potential to be replaced with a whole subtree, producing a kind of “growing” or “feathering” behaviour, reminiscent of list concatenation along two axes.

9.11 Summary

In this chapter we’ve seen monads up-close. We saw that flatMap can be viewed as an operator for sequencing computations, dictating the order in which operations must happen. From this viewpoint, Option represents a computation that can fail without an error message, Either represents computations that can fail with a message, List represents multiple possible results, and Future represents a computation that may produce a value at some point in the future.

We’ve also seen some of the custom types and data structures that Cats provides, including Id, Reader, Writer, and State. These cover a wide range of use cases.

Finally, in the unlikely event that we have to implement a custom monad, we’ve learned about defining our own instance using tailRecM. tailRecM is an odd wrinkle that is a concession to building a functional programming library that is stack-safe by default. We don’t need to understand tailRecM to understand monads, but having it around gives us benefits of which we can be grateful when writing monadic code.

10 Monad Transformers

Monads are like burritos, which means that once you acquire a taste, you’ll find yourself returning to them again and again. This is not without issues. As burritos can bloat the waist, monads can bloat the code base through nested for-comprehensions.

Imagine we are interacting with a database. We want to look up a user record. The user may or may not be present, so we return an Option[User]. Our communication with the database could fail for many reasons (network issues, authentication problems, and so on), so this result is wrapped up in an Either, giving us a final result of Either[Error, Option[User]].

To use this value we must nest flatMap calls (or equivalently, for-comprehensions):

def lookupUserName(id: Long): Either[Error, Option[String]] =
  for {
    optUser <- lookupUser(id)
  } yield {
    for { user <- optUser } yield user.name
  }

This quickly becomes very tedious.

10.1 Exercise: Composing Monads

A question arises. Given two arbitrary monads, can we combine them in some way to make a single monad? That is, do monads compose? We can try to write the code but we soon hit problems:

import cats.syntax.applicative._ // for pure
// Hypothetical example. This won't actually compile:
def compose[M1[_]: Monad, M2[_]: Monad] = {
  type Composed[A] = M1[M2[A]]

  new Monad[Composed] {
    def pure[A](a: A): Composed[A] =
      a.pure[M2].pure[M1]

    def flatMap[A, B](fa: Composed[A])
        (f: A => Composed[B]): Composed[B] =
      // Problem! How do we write flatMap?
      ???
  }
}

It is impossible to write a general definition of flatMap without knowing something about M1 or M2. However, if we do know something about one or other monad, we can typically complete this code. For example, if we fix M2 above to be Option, a definition of flatMap comes to light:

def flatMap[A, B](fa: Composed[A])
    (f: A => Composed[B]): Composed[B] =
  fa.flatMap(_.fold[Composed[B]](None.pure[M1])(f))

Notice that the definition above makes use of None—an Option-specific concept that doesn’t appear in the general Monad interface. We need this extra detail to combine Option with other monads. Similarly, there are things about other monads that help us write composed flatMap methods for them. This is the idea behind monad transformers: Cats defines transformers for a variety of monads, each providing the extra knowledge we need to compose that monad with others. Let’s look at some examples.

10.2 A Transformative Example

Cats provides transformers for many monads, each named with a T suffix: EitherT composes Either with other monads, OptionT composes Option, and so on.

Here’s an example that uses OptionT to compose List and Option. We can use OptionT[List, A], aliased to ListOption[A] for convenience, to transform a List[Option[A]] into a single monad:

import cats.data.OptionT

type ListOption[A] = OptionT[List, A]

Note how we build ListOption from the inside out: we pass List, the type of the outer monad, as a parameter to OptionT, the transformer for the inner monad.

We can create instances of ListOption using the OptionT constructor, or more conveniently using pure:

import cats.instances.list._     // for Monad
import cats.syntax.applicative._ // for pure
val result1: ListOption[Int] = OptionT(List(Option(10)))
// result1: OptionT[List, Int] = OptionT(value = List(Some(value = 10)))

val result2: ListOption[Int] = 32.pure[ListOption]
// result2: OptionT[List, Int] = OptionT(value = List(Some(value = 32)))

The map and flatMap methods combine the corresponding methods of List and Option into single operations:

result1.flatMap { (x: Int) =>
  result2.map { (y: Int) =>
    x + y
  }
}
// res1: OptionT[List, Int] = OptionT(value = List(Some(value = 42)))

This is the basis of all monad transformers. The combined map and flatMap methods allow us to use both component monads without having to recursively unpack and repack values at each stage in the computation. Now let’s look at the API in more depth.

Complexity of Imports

The imports in the code samples above hint at how everything bolts together.

We import cats.syntax.applicative to get the pure syntax. pure requires an implicit parameter of type Applicative[ListOption]. We haven’t met Applicatives yet, but all Monads are also Applicatives so we can ignore that difference for now.

In order to generate our Applicative[ListOption] we need instances of Applicative for List and OptionT. OptionT is a Cats data type so its instance is provided by its companion object. The instance for List comes from cats.instances.list.

Notice we’re not importing cats.syntax.functor or cats.syntax.flatMap. This is because OptionT is a concrete data type with its own explicit map and flatMap methods. It wouldn’t cause problems if we imported the syntax—the compiler would ignore it in favour of the explicit methods.

Remember that we’re subjecting ourselves to these shenanigans because we’re stubbornly refusing to use the universal Cats import, cats.implicits. If we did use that import, all of the instances and syntax we needed would be in scope and everything would just work.

10.3 Monad Transformers in Cats

Each monad transformer is a data type, defined in cats.data, that allows us to wrap stacks of monads to produce new monads. We use the monads we’ve built via the Monad type class. The main concepts we have to cover to understand monad transformers are:

10.3.1 The Monad Transformer Classes

By convention, in Cats a monad Foo will have a transformer class called FooT. In fact, many monads in Cats are defined by combining a monad transformer with the Id monad. Concretely, some of the available instances are:

Kleisli Arrows

In Section 9.8 we mentioned that the Reader monad was a specialisation of a more general concept called a “kleisli arrow”, represented in Cats as cats.data.Kleisli.

We can now reveal that Kleisli and ReaderT are, in fact, the same thing! ReaderT is actually a type alias for Kleisli. Hence, we were creating Readers last chapter and seeing Kleislis on the console.

10.3.2 Building Monad Stacks

All of these monad transformers follow the same convention. The transformer itself represents the inner monad in a stack, while the first type parameter specifies the outer monad. The remaining type parameters are the types we’ve used to form the corresponding monads.

For example, our ListOption type above is an alias for OptionT[List, A] but the result is effectively a List[Option[A]]. In other words, we build monad stacks from the inside out:

type ListOption[A] = OptionT[List, A]

Many monads and all transformers have at least two type parameters, so we often have to define type aliases for intermediate stages.

For example, suppose we want to wrap Either around Option. Option is the innermost type so we want to use the OptionT monad transformer. We need to use Either as the first type parameter. However, Either itself has two type parameters and monads only have one. We need a type alias to convert the type constructor to the correct shape:

// Alias Either to a type constructor with one parameter:
type ErrorOr[A] = Either[String, A]

// Build our final monad stack using OptionT:
type ErrorOrOption[A] = OptionT[ErrorOr, A]

ErrorOrOption is a monad, just like ListOption. We can use pure, map, and flatMap as usual to create and transform instances:

import cats.instances.either._ // for Monad
val a = 10.pure[ErrorOrOption]
// a: OptionT[ErrorOr, Int] = OptionT(value = Right(value = Some(value = 10)))
val b = 32.pure[ErrorOrOption]
// b: OptionT[ErrorOr, Int] = OptionT(value = Right(value = Some(value = 32)))

val c = a.flatMap(x => b.map(y => x + y))
// c: OptionT[ErrorOr, Int] = OptionT(value = Right(value = Some(value = 42)))

Things become even more confusing when we want to stack three or more monads.

For example, let’s create a Future of an Either of Option. Once again we build this from the inside out with an OptionT of an EitherT of Future. However, we can’t define this in one line because EitherT has three type parameters:

case class EitherT[F[_], E, A](stack: F[Either[E, A]]) {
  // etc...
}

The three type parameters are as follows:

This time we create an alias for EitherT that fixes Future and Error and allows A to vary:

import scala.concurrent.Future
import cats.data.{EitherT, OptionT}

type FutureEither[A] = EitherT[Future, String, A]

type FutureEitherOption[A] = OptionT[FutureEither, A]

Our mammoth stack now composes three monads and our map and flatMap methods cut through three layers of abstraction:

import cats.instances.future._ // for Monad
import scala.concurrent.Await
import scala.concurrent.ExecutionContext.Implicits.global
import scala.concurrent.duration._
val futureEitherOr: FutureEitherOption[Int] =
  for {
    a <- 10.pure[FutureEitherOption]
    b <- 32.pure[FutureEitherOption]
  } yield a + b

Kind Projector

If you frequently find yourself defining multiple type aliases when building monad stacks, you may want to try the Kind Projector compiler plugin. Kind Projector enhances Scala’s type syntax to make it easier to define partially applied type constructors. For example:

import cats.instances.option._ // for Monad

123.pure[EitherT[Option, String, _]]
// res3: EitherT[[A >: Nothing <: Any] => Option[A], String, Int] = EitherT(
//   value = Some(value = Right(value = 123))
// )

Kind Projector can’t simplify all type declarations down to a single line, but it can reduce the number of intermediate type definitions needed to keep our code readable.

10.3.3 Constructing and Unpacking Instances

As we saw above, we can create transformed monad stacks using the relevant monad transformer’s apply method or the usual pure syntax7:

// Create using apply:
val errorStack1 = OptionT[ErrorOr, Int](Right(Some(10)))
// errorStack1: OptionT[ErrorOr, Int] = OptionT(
//   value = Right(value = Some(value = 10))
// )

// Create using pure:
val errorStack2 = 32.pure[ErrorOrOption]
// errorStack2: OptionT[ErrorOr, Int] = OptionT(
//   value = Right(value = Some(value = 32))
// )

Once we’ve finished with a monad transformer stack, we can unpack it using its value method. This returns the untransformed stack. We can then manipulate the individual monads in the usual way:

// Extracting the untransformed monad stack:
errorStack1.value
// res4: Either[String, Option[Int]] = Right(value = Some(value = 10))

// Mapping over the Either in the stack:
errorStack2.value.map(_.getOrElse(-1))
// res5: Either[String, Int] = Right(value = 32)

Each call to value unpacks a single monad transformer. We may need more than one call to completely unpack a large stack. For example, to Await the FutureEitherOption stack above, we need to call value twice:

futureEitherOr
// res6: OptionT[FutureEither, Int] = OptionT(
//   value = EitherT(value = Future(Success(Right(Some(42)))))
// )

val intermediate = futureEitherOr.value
// intermediate: EitherT[[T >: Nothing <: Any] => Future[T], String, Option[Int]] = EitherT(
//   value = Future(Success(Right(Some(42))))
// )

val stack = intermediate.value
// stack: Future[Either[String, Option[Int]]] = Future(Success(Right(Some(42))))

Await.result(stack, 1.second)
// res7: Either[String, Option[Int]] = Right(value = Some(value = 42))

10.3.4 Default Instances

Many monads in Cats are defined using the corresponding transformer and the Id monad. This is reassuring as it confirms that the APIs for monads and transformers are identical. Reader, Writer, and State are all defined in this way:

type Reader[E, A] = ReaderT[Id, E, A] // = Kleisli[Id, E, A]
type Writer[W, A] = WriterT[Id, W, A]
type State[S, A]  = StateT[Id, S, A]

In other cases monad transformers are defined separately to their corresponding monads. In these cases, the methods of the transformer tend to mirror the methods on the monad. For example, OptionT defines getOrElse, and EitherT defines fold, bimap, swap, and other useful methods.

10.3.5 Usage Patterns

Widespread use of monad transformers is sometimes difficult because they fuse monads together in predefined ways. Without careful thought, we can end up having to unpack and repack monads in different configurations to operate on them in different contexts.

We can cope with this in multiple ways. One approach involves creating a single “super stack” and sticking to it throughout our code base. This works if the code is simple and largely uniform in nature. For example, in a web application, we could decide that all request handlers are asynchronous and all can fail with the same set of HTTP error codes. We could design a custom ADT representing the errors and use a fusion Future and Either everywhere in our code:

sealed abstract class HttpError
final case class NotFound(item: String) extends HttpError
final case class BadRequest(msg: String) extends HttpError
// etc...

type FutureEither[A] = EitherT[Future, HttpError, A]

The “super stack” approach starts to fail in larger, more heterogeneous code bases where different stacks make sense in different contexts. Another design pattern that makes more sense in these contexts uses monad transformers as local “glue code”. We expose untransformed stacks at module boundaries, transform them to operate on them locally, and untransform them before passing them on. This allows each module of code to make its own decisions about which transformers to use:

import cats.data.Writer

type Logged[A] = Writer[List[String], A]

// Methods generally return untransformed stacks:
def parseNumber(str: String): Logged[Option[Int]] =
  util.Try(str.toInt).toOption match {
    case Some(num) => Writer(List(s"Read $str"), Some(num))
    case None      => Writer(List(s"Failed on $str"), None)
  }

// Consumers use monad transformers locally to simplify composition:
def addAll(a: String, b: String, c: String): Logged[Option[Int]] = {
  import cats.data.OptionT

  val result = for {
    a <- OptionT(parseNumber(a))
    b <- OptionT(parseNumber(b))
    c <- OptionT(parseNumber(c))
  } yield a + b + c

  result.value
}
// This approach doesn't force OptionT on other users' code:
val result1 = addAll("1", "2", "3")
// result1: WriterT[Id, List[String], Option[Int]] = WriterT(
//   run = (List("Read 1", "Read 2", "Read 3"), Some(value = 6))
// )
val result2 = addAll("1", "a", "3")
// result2: WriterT[Id, List[String], Option[Int]] = WriterT(
//   run = (List("Read 1", "Failed on a"), None)
// )

Unfortunately, there aren’t one-size-fits-all approaches to working with monad transformers. The best approach for you may depend on a lot of factors: the size and experience of your team, the complexity of your code base, and so on. You may need to experiment and gather feedback from colleagues to determine whether monad transformers are a good fit.

10.4 Exercise: Monads: Transform and Roll Out

The Autobots, well-known robots in disguise, frequently send messages during battle requesting the power levels of their team mates. This helps them coordinate strategies and launch devastating attacks. The message sending method looks like this:

def getPowerLevel(autobot: String): Response[Int] =
  ???

Transmissions take time in Earth’s viscous atmosphere, and messages are occasionally lost due to satellite malfunction or sabotage by pesky Decepticons8. Responses are therefore represented as a stack of monads:

type Response[A] = Future[Either[String, A]]

Optimus Prime is getting tired of the nested for comprehensions in his neural matrix. Help him by rewriting Response using a monad transformer.

This is a relatively simple combination. We want Future on the outside and Either on the inside, so we build from the inside out using an EitherT of Future:

import cats.data.EitherT
import scala.concurrent.Future

type Response[A] = EitherT[Future, String, A]

Now test the code by implementing getPowerLevel to retrieve data from a set of imaginary allies. Here’s the data we’ll use:

val powerLevels = Map(
  "Jazz"      -> 6,
  "Bumblebee" -> 8,
  "Hot Rod"   -> 10
)

If an Autobot isn’t in the powerLevels map, return an error message reporting that they were unreachable. Include the name in the message for good effect.

import cats.data.EitherT
import scala.concurrent.Future
val powerLevels = Map(
  "Jazz"      -> 6,
  "Bumblebee" -> 8,
  "Hot Rod"   -> 10
)
import cats.instances.future._ // for Monad
import scala.concurrent.ExecutionContext.Implicits.global

type Response[A] = EitherT[Future, String, A]

def getPowerLevel(ally: String): Response[Int] = {
  powerLevels.get(ally) match {
    case Some(avg) => EitherT.right(Future(avg))
    case None      => EitherT.left(Future(s"$ally unreachable"))
  }
}

Two autobots can perform a special move if their combined power level is greater than 15. Write a second method, canSpecialMove, that accepts the names of two allies and checks whether a special move is possible. If either ally is unavailable, fail with an appropriate error message:

def canSpecialMove(ally1: String, ally2: String): Response[Boolean] =
  ???

We request the power level from each ally and use map and flatMap to combine the results:

def canSpecialMove(ally1: String, ally2: String): Response[Boolean] =
  for {
    power1 <- getPowerLevel(ally1)
    power2 <- getPowerLevel(ally2)
  } yield (power1 + power2) > 15

Finally, write a method tacticalReport that takes two ally names and prints a message saying whether they can perform a special move:

def tacticalReport(ally1: String, ally2: String): String =
  ???

We use the value method to unpack the monad stack and Await and fold to unpack the Future and Either:

import scala.concurrent.Await
import scala.concurrent.ExecutionContext.Implicits.global
import scala.concurrent.duration._

def canSpecialMove(ally1: String, ally2: String): Response[Boolean] =
  for {
    power1 <- getPowerLevel(ally1)
    power2 <- getPowerLevel(ally2)
  } yield (power1 + power2) > 15

def tacticalReport(ally1: String, ally2: String): String = {
  val stack = canSpecialMove(ally1, ally2).value

  Await.result(stack, 1.second) match {
    case Left(msg) =>
      s"Comms error: $msg"
    case Right(true)  =>
      s"$ally1 and $ally2 are ready to roll out!"
    case Right(false) =>
      s"$ally1 and $ally2 need a recharge."
  }
}

You should be able to use report as follows:

tacticalReport("Jazz", "Bumblebee")
// res13: String = "Jazz and Bumblebee need a recharge."
tacticalReport("Bumblebee", "Hot Rod")
// res14: String = "Bumblebee and Hot Rod are ready to roll out!"
tacticalReport("Jazz", "Ironhide")
// res15: String = "Comms error: Ironhide unreachable"

10.5 Summary

In this chapter we introduced monad transformers, which eliminate the need for nested for comprehensions and pattern matching when working with “stacks” of nested monads.

Each monad transformer, such as FutureT, OptionT or EitherT, provides the code needed to merge its related monad with other monads. The transformer is a data structure that wraps a monad stack, equipping it with map and flatMap methods that unpack and repack the whole stack.

The type signatures of monad transformers are written from the inside out, so an EitherT[Option, String, A] is a wrapper for an Option[Either[String, A]]. It is often useful to use type aliases when writing transformer types for deeply nested monads.

With this look at monad transformers, we have now covered everything we need to know about monads and the sequencing of computations using flatMap. In the next chapter we will switch tack and discuss two new type classes, Semigroupal and Applicative, that support new kinds of operation such as zipping independent values within a context.

11 Semigroupal and Applicative

In previous chapters we saw how functors and monads let us sequence operations using map and flatMap. While functors and monads are both immensely useful abstractions, there are certain types of program flow that they cannot represent.

One such example is form validation. When we validate a form we want to return all the errors to the user, not stop on the first error we encounter. If we model this with a monad like Either, we fail fast and lose errors. For example, the code below fails on the first call to parseInt and doesn’t go any further:

import cats.syntax.either._ // for catchOnly

def parseInt(str: String): Either[String, Int] =
  Either.catchOnly[NumberFormatException](str.toInt).
    leftMap(_ => s"Couldn't read $str")
for {
  a <- parseInt("a")
  b <- parseInt("b")
  c <- parseInt("c")
} yield (a + b + c)
// res0: Either[String, Int] = Left(value = "Couldn't read a")

Another example is the concurrent evaluation of Futures. If we have several long-running independent tasks, it makes sense to execute them concurrently. However, monadic comprehension only allows us to run them in sequence. map and flatMap aren’t quite capable of capturing what we want because they make the assumption that each computation is dependent on the previous one:

// context2 is dependent on value1:
context1.flatMap(value1 => context2)

The calls to parseInt and Future.apply above are independent of one another, but map and flatMap can’t exploit this. We need a weaker construct—one that doesn’t guarantee sequencing—to achieve the result we want. In this chapter we will look at three type classes that support this pattern:

Applicatives are often formulated in terms of function application, instead of the semigroupal formulation that is emphasised in Cats. This alternative formulation provides a link to other libraries and languages such as Scalaz and Haskell. We’ll take a look at different formulations of Applicative, as well as the relationships between Semigroupal, Functor, Applicative, and Monad, towards the end of the chapter.

11.1 Semigroupal

cats.Semigroupal is a type class that allows us to combine contexts9. If we have two objects of type F[A] and F[B], a Semigroupal[F] allows us to combine them to form an F[(A, B)]. Its definition in Cats is:

trait Semigroupal[F[_]] {
  def product[A, B](fa: F[A], fb: F[B]): F[(A, B)]
}

As we discussed at the beginning of this chapter, the parameters fa and fb are independent of one another: we can compute them in either order before passing them to product. This is in contrast to flatMap, which imposes a strict order on its parameters. This gives us more freedom when defining instances of Semigroupal than we get when defining Monads.

11.1.1 Joining Two Contexts

While Semigroup allows us to join values, Semigroupal allows us to join contexts. Let’s join some Options as an example:

import cats.Semigroupal
import cats.instances.option._ // for Semigroupal
Semigroupal[Option].product(Some(123), Some("abc"))
// res1: Option[Tuple2[Int, String]] = Some(value = (123, "abc"))

If both parameters are instances of Some, we end up with a tuple of the values within. If either parameter evaluates to None, the entire result is None:

Semigroupal[Option].product(None, Some("abc"))
// res2: Option[Tuple2[Nothing, String]] = None
Semigroupal[Option].product(Some(123), None)
// res3: Option[Tuple2[Int, Nothing]] = None

11.1.2 Joining Three or More Contexts

The companion object for Semigroupal defines a set of methods on top of product. For example, the methods tuple2 through tuple22 generalise product to different arities:

import cats.instances.option._ // for Semigroupal
Semigroupal.tuple3(Option(1), Option(2), Option(3))
// res4: Option[Tuple3[Int, Int, Int]] = Some(value = (1, 2, 3))
Semigroupal.tuple3(Option(1), Option(2), Option.empty[Int])
// res5: Option[Tuple3[Int, Int, Int]] = None

The methods map2 through map22 apply a user-specified function to the values inside 2 to 22 contexts:

Semigroupal.map3(Option(1), Option(2), Option(3))(_ + _ + _)
// res6: Option[Int] = Some(value = 6)

Semigroupal.map2(Option(1), Option.empty[Int])(_ + _)
// res7: Option[Int] = None

There are also methods contramap2 through contramap22 and imap2 through imap22, that require instances of Contravariant and Invariant respectively.

11.1.3 Semigroupal Laws

There is only one law for Semigroupal: the product method must be associative.

product(a, product(b, c)) == product(product(a, b), c)

11.2 Apply Syntax

Cats provides a convenient apply syntax that provides a shorthand for the methods described above. We import the syntax from cats.syntax.apply. Here’s an example:

import cats.instances.option._ // for Semigroupal
import cats.syntax.apply._     // for tupled and mapN

The tupled method is implicitly added to the tuple of Options. It uses the Semigroupal for Option to zip the values inside the Options, creating a single Option of a tuple:

(Option(123), Option("abc")).tupled
// res8: Option[Tuple2[Int, String]] = Some(value = (123, "abc"))

We can use the same trick on tuples of up to 22 values. Cats defines a separate tupled method for each arity:

(Option(123), Option("abc"), Option(true)).tupled
// res9: Option[Tuple3[Int, String, Boolean]] = Some(
//   value = (123, "abc", true)
// )

In addition to tupled, Cats’ apply syntax provides a method called mapN that accepts an implicit Functor and a function of the correct arity to combine the values.

final case class Cat(name: String, born: Int, color: String)
(
  Option("Garfield"),
  Option(1978),
  Option("Orange & black")
).mapN(Cat.apply)
// res10: Option[Cat] = Some(
//   value = Cat(name = "Garfield", born = 1978, color = "Orange & black")
// )

Of all the methods mentioned here, it is most common to use mapN.

Internally mapN uses the Semigroupal to extract the values from the Option and the Functor to apply the values to the function.

It’s nice to see that this syntax is type checked. If we supply a function that accepts the wrong number or types of parameters, we get a compile error:

val add: (Int, Int) => Int = (a, b) => a + b
// add: Function2[Int, Int, Int] = repl.MdocSession$MdocApp0$$$Lambda$18724/0x0000000804cff040@518e803d
(Option(1), Option(2), Option(3)).mapN(add)
// error: 
// ':' expected, but '(' found
// error: 
// ':' expected, but '(' found
// error: 
// ':' expected, but '(' found
// error: 
// end of statement expected but '.' found
// error: 
// Found:    (repl.MdocSession.MdocApp0.add : (Int, Int) => Int)
// Required: (Int, Int, Int) => Any
(Option("cats"), Option(true)).mapN(add)
// error: 
// ':' expected, but '(' found
// error: 
// ':' expected, but '(' found
// error: 
// ':' expected, but '(' found
// error: 
// end of statement expected but '.' found
// error:
// Found:    (repl.MdocSession.MdocApp0.add : (Int, Int) => Int)
// Required: (String, Boolean) => Any
// (Option("cats"), Option(true)).mapN(add)
//                                     ^^^

11.2.1 Fancy Functors and Apply Syntax

Apply syntax also has contramapN and imapN methods that accept Contravariant and Invariant functors (Section 8.6). For example, we can combine Monoids using Invariant. Here’s an example:

import cats.Monoid
import cats.instances.int._        // for Monoid
import cats.instances.invariant._  // for Semigroupal
import cats.instances.list._       // for Monoid
import cats.instances.string._     // for Monoid
import cats.syntax.apply._         // for imapN

final case class Cat(
  name: String,
  yearOfBirth: Int,
  favoriteFoods: List[String]
)

val tupleToCat: (String, Int, List[String]) => Cat =
  Cat.apply _

val catToTuple: Cat => (String, Int, List[String]) =
  cat => (cat.name, cat.yearOfBirth, cat.favoriteFoods)

implicit val catMonoid: Monoid[Cat] = (
  Monoid[String],
  Monoid[Int],
  Monoid[List[String]]
).imapN(tupleToCat)(catToTuple)

Our Monoid allows us to create “empty” Cats, and add Cats together using the syntax from Chapter 7:

import cats.syntax.semigroup._ // for |+|

val garfield   = Cat("Garfield", 1978, List("Lasagne"))
val heathcliff = Cat("Heathcliff", 1988, List("Junk Food"))
garfield |+| heathcliff
// res14: Cat = Cat(
//   name = "GarfieldHeathcliff",
//   yearOfBirth = 3966,
//   favoriteFoods = List("Lasagne", "Junk Food")
// )

11.3 Semigroupal Applied to Different Types

Semigroupal doesn’t always provide the behaviour we expect, particularly for types that also have instances of Monad. We have seen the behaviour of the Semigroupal for Option. Let’s look at some examples for other types.

Future

The semantics for Future provide parallel as opposed to sequential execution:

import cats.Semigroupal
import cats.instances.future._ // for Semigroupal
import scala.concurrent._
import scala.concurrent.duration._
import scala.concurrent.ExecutionContext.Implicits.global

val futurePair = Semigroupal[Future].
  product(Future("Hello"), Future(123))
Await.result(futurePair, 1.second)
// res0: Tuple2[String, Int] = ("Hello", 123)

The two Futures start executing the moment we create them, so they are already calculating results by the time we call product. We can use apply syntax to zip fixed numbers of Futures:

import cats.syntax.apply._ // for mapN

case class Cat(
  name: String,
  yearOfBirth: Int,
  favoriteFoods: List[String]
)

val futureCat = (
  Future("Garfield"),
  Future(1978),
  Future(List("Lasagne"))
).mapN(Cat.apply)
Await.result(futureCat, 1.second)
// res1: Cat = Cat(
//   name = "Garfield",
//   yearOfBirth = 1978,
//   favoriteFoods = List("Lasagne")
// )

List

Combining Lists with Semigroupal produces some potentially unexpected results. We might expect code like the following to zip the lists, but we actually get the cartesian product of their elements:

import cats.Semigroupal
import cats.instances.list._ // for Semigroupal
Semigroupal[List].product(List(1, 2), List(3, 4))
// res2: List[Tuple2[Int, Int]] = List((1, 3), (1, 4), (2, 3), (2, 4))

This is perhaps surprising. Zipping lists tends to be a more common operation. We’ll see why we get this behaviour in a moment.

Either

We opened this chapter with a discussion of fail-fast versus accumulating error-handling. We might expect product applied to Either to accumulate errors instead of fail fast. Again, perhaps surprisingly, we find that product implements the same fail-fast behaviour as flatMap:

import cats.instances.either._ // for Semigroupal

type ErrorOr[A] = Either[Vector[String], A]
Semigroupal[ErrorOr].product(
  Left(Vector("Error 1")),
  Left(Vector("Error 2"))
)
// res3: Either[Vector[String], Tuple2[Nothing, Nothing]] = Left(
//   value = Vector("Error 1")
// )

In this example product sees the first failure and stops, even though it is possible to examine the second parameter and see that it is also a failure.

11.3.1 Semigroupal Applied to Monads

The reason for the surprising results for List and Either is that they are both monads. If we have a monad we can implement product as follows.

import cats.Monad
import cats.syntax.functor._ // for map
import cats.syntax.flatMap._ // for flatmap

def product[F[_]: Monad, A, B](fa: F[A], fb: F[B]): F[(A,B)] =
  fa.flatMap(a => 
    fb.map(b =>
      (a, b)
    )
  )

It would be very strange if we had different semantics for product depending on how we implemented it. To ensure consistent semantics, Cats’ Monad (which extends Semigroupal) provides a standard definition of product in terms of map and flatMap as we showed above.

Even our results for Future are a trick of the light. flatMap provides sequential ordering, so product provides the same. The parallel execution we observe occurs because our constituent Futures start running before we call product. This is equivalent to the classic create-then-flatMap pattern:

val a = Future("Future 1")
val b = Future("Future 2")

for {
  x <- a
  y <- b
} yield (x, y)

So why bother with Semigroupal at all? The answer is that we can create useful data types that have instances of Semigroupal (and Applicative) but not Monad. This frees us to implement product in different ways. We’ll examine this further in a moment when we look at an alternative data type for error handling.

11.3.1.1 Exercise: The Product of Lists

Why does product for List produce the Cartesian product? We saw an example above. Here it is again.

Semigroupal[List].product(List(1, 2), List(3, 4))
// res5: List[Tuple2[Int, Int]] = List((1, 3), (1, 4), (2, 3), (2, 4))

We can also write this in terms of tupled.

(List(1, 2), List(3, 4)).tupled
// res6: List[Tuple2[Int, Int]] = List((1, 3), (1, 4), (2, 3), (2, 4))

This exercise is checking that you understood the definition of product in terms of flatMap and map.

import cats.syntax.functor._ // for map
import cats.syntax.flatMap._ // for flatMap

def product[F[_]: Monad, A, B](x: F[A], y: F[B]): F[(A, B)] =
  x.flatMap(a => y.map(b => (a, b)))

This code is equivalent to a for comprehension:

def product[F[_]: Monad, A, B](x: F[A], y: F[B]): F[(A, B)] =
  for {
    a <- x
    b <- y
  } yield (a, b)

The semantics of flatMap are what give rise to the behaviour for List and Either:

import cats.instances.list._ // for Semigroupal
product(List(1, 2), List(3, 4))
// res9: List[Tuple2[Int, Int]] = List((1, 3), (1, 4), (2, 3), (2, 4))

11.4 Parallel

In the previous section we saw that when call product on a type that has a Monad instance we get sequential semantics. This makes sense from the point-of-view of keeping consistency with implementations of product in terms of flatMap and map. However it’s not always what we want. The Parallel type class, and its associated syntax, allows us to access alternate semantics for certain monads.

We’ve seen how the product method on Either stops at the first error.

import cats.Semigroupal
import cats.instances.either._ // for Semigroupal

type ErrorOr[A] = Either[Vector[String], A]
val error1: ErrorOr[Int] = Left(Vector("Error 1"))
val error2: ErrorOr[Int] = Left(Vector("Error 2"))
Semigroupal[ErrorOr].product(error1, error2)
// res0: Either[Vector[String], Tuple2[Int, Int]] = Left(
//   value = Vector("Error 1")
// )

We can also write this using tupled as a short-cut.

import cats.syntax.apply._ // for tupled
import cats.instances.vector._ // for Semigroup on Vector
(error1, error2).tupled
// res1: Either[Vector[String], Tuple2[Int, Int]] = Left(
//   value = Vector("Error 1")
// )

To collect all the errors we simply replace tupled with its “parallel” version called parTupled.

import cats.syntax.parallel._ // for parTupled
(error1, error2).parTupled
// res2: Either[Vector[String], Tuple2[Int, Int]] = Left(
//   value = Vector("Error 1", "Error 2")
// )

Notice that both errors are returned! This behaviour is not special to using Vector as the error type. Any type that has a Semigroup instance will work. For example, here we use List instead.

import cats.instances.list._ // for Semigroup on List

type ErrorOrList[A] = Either[List[String], A]
val errStr1: ErrorOrList[Int] = Left(List("error 1"))
val errStr2: ErrorOrList[Int] = Left(List("error 2"))
(errStr1, errStr2).parTupled
// res3: Either[List[String], Tuple2[Int, Int]] = Left(
//   value = List("error 1", "error 2")
// )

There are many syntax methods provided by Parallel for methods on Semigroupal and related types, but the most commonly used is parMapN. Here’s an example of parMapN in an error handling situation.

val success1: ErrorOr[Int] = Right(1)
val success2: ErrorOr[Int] = Right(2)
val addTwo = (x: Int, y: Int) => x + y
(error1, error2).parMapN(addTwo)
(success1, success2).parMapN(addTwo)
// res4: Either[Vector[String], Int] = Right(value = 3)

Let’s dig into how Parallel works. The definition below is the core of Parallel.

trait Parallel[M[_]] {
  type F[_]
  
  def applicative: Applicative[F]
  def monad: Monad[M]
  def parallel: ~>[M, F]
}

This tells us if there is a Parallel instance for some type constructor M then:

We haven’t seen ~> before. It’s a type alias for FunctionK and is what performs the conversion from M to F. A normal function A => B converts values of type A to values of type B. Remember that M and F are not types; they are type constructors. A FunctionK M ~> F is a function from a value with type M[A] to a value with type F[A]. Let’s see a quick example by defining a FunctionK that converts an Option to a List.

import cats.arrow.FunctionK

object optionToList extends FunctionK[Option, List] {
  def apply[A](fa: Option[A]): List[A] =
    fa match {
      case None    => List.empty[A]
      case Some(a) => List(a)
    }
}
optionToList(Some(1))
// res5: List[Int] = List(1)
optionToList(None)
// res6: List[Nothing] = List()

As the type parameter A is generic a FunctionK cannot inspect any values contained with the type constructor M. The conversion must be performed purely in terms of the structure of the type constructors M and F. We can see in optionToList above this is indeed the case.

So in summary, Parallel allows us to take a type that has a monad instance and convert it to some related type that instead has an applicative (or semigroupal) instance. This related type will have some useful alternate semantics. We’ve seen the case above where the related applicative for Either allows for accumulation of errors instead of fail-fast semantics.

Now we’ve seen Parallel it’s time to finally learn about Applicative.

11.4.0.1 Exercise: Parallel List

Does List have a Parallel instance? If so, what does the Parallel instance do?

List does have a Parallel instance, and it zips the List insted of creating the cartesian product.

We can see by writing a little bit of code.

import cats.instances.list._
(List(1, 2), List(3, 4)).tupled
(List(1, 2), List(3, 4)).parTupled
// res7: List[Tuple2[Int, Int]] = List((1, 3), (2, 4))

11.5 Apply and Applicative

Semigroupals aren’t mentioned frequently in the wider functional programming literature. They provide a subset of the functionality of a related type class called an applicative functor (“applicative” for short).

Semigroupal and Applicative effectively provide alternative encodings of the same notion of joining contexts. Both encodings are introduced in the same 2008 paper by Conor McBride and Ross Paterson10.

Cats models applicatives using two type classes. The first, cats.Apply, extends Semigroupal and Functor and adds an ap method that applies a parameter to a function within a context. The second, cats.Applicative, extends Apply and adds the pure method introduced in Chapter 9. Here’s a simplified definition in code:

trait Apply[F[_]] extends Semigroupal[F] with Functor[F] {
  def ap[A, B](ff: F[A => B])(fa: F[A]): F[B]

  def product[A, B](fa: F[A], fb: F[B]): F[(A, B)] =
    ap(map(fa)(a => (b: B) => (a, b)))(fb)
}

trait Applicative[F[_]] extends Apply[F] {
  def pure[A](a: A): F[A]
}

Breaking this down, the ap method applies a parameter fa to a function ff within a context F[_]. The product method from Semigroupal is defined in terms of ap and map.

Don’t worry too much about the implementation of product—it’s difficult to read and the details aren’t particuarly important. The main point is that there is a tight relationship between product, ap, and map that allows any one of them to be defined in terms of the other two.

Applicative also introduces the pure method. This is the same pure we saw in Monad. It constructs a new applicative instance from an unwrapped value. In this sense, Applicative is related to Apply as Monoid is related to Semigroup.

11.5.1 The Hierarchy of Sequencing Type Classes

With the introduction of Apply and Applicative, we can zoom out and see a whole family of type classes that concern themselves with sequencing computations in different ways. Figure 10 shows the relationship between the type classes covered in this book11.

Monad type class hierarchy
Figure 10: Monad type class hierarchy

Each type class in the hierarchy represents a particular set of sequencing semantics, introduces a set of characteristic methods, and defines the functionality of its supertypes in terms of them:

Because of the lawful nature of the relationships between the type classes, the inheritance relationships are constant across all instances of a type class. Apply defines product in terms of ap and map; Monad defines product, ap, and map, in terms of pure and flatMap.

To illustrate this let’s consider two hypothetical data types:

What can we say about these two data types without knowing more about their implementation?

We know strictly more about Foo than Bar: Monad is a subtype of Applicative, so we can guarantee properties of Foo (namely flatMap) that we cannot guarantee with Bar. Conversely, we know that Bar may have a wider range of behaviours than Foo. It has fewer laws to obey (no flatMap), so it can implement behaviours that Foo cannot.

This demonstrates the classic trade-off of power (in the mathematical sense) versus constraint. The more constraints we place on a data type, the more guarantees we have about its behaviour, but the fewer behaviours we can model.

Monads happen to be a sweet spot in this trade-off. They are flexible enough to model a wide range of behaviours and restrictive enough to give strong guarantees about those behaviours. However, there are situations where monads aren’t the right tool for the job. Sometimes we want Thai food, and burritos just won’t satisfy.

Whereas monads impose a strict sequencing on the computations they model, applicatives and semigroupals impose no such restriction. This puts them in a different sweet spot in the hierarchy. We can use them to represent classes of parallel / independent computations that monads cannot.

We choose our semantics by choosing our data structures. If we choose a monad, we get strict sequencing. If we choose an applicative, we lose the ability to flatMap. This is the trade-off enforced by the consistency laws. So choose your types carefully!

11.6 Summary

While monads and functors are the most widely used sequencing data types we’ve covered in this book, semigroupals and applicatives are the most general. These type classes provide a generic mechanism to combine values and apply functions within a context, from which we can fashion monads and a variety of other combinators.

Semigroupal and Applicative are most commonly used as a means of combining independent values such as the results of validation rules. Cats provides the Validated type for this specific purpose, along with apply syntax as a convenient way to express the combination of rules.

We have almost covered all of the functional programming concepts on our agenda for this book. The next chapter covers Traverse and Foldable, two powerful type classes for converting between data types. After that we’ll look at several case studies that bring together all of the concepts from Part I.

12 Foldable and Traverse

In this chapter we’ll look at two type classes that capture iteration over collections:

We’ll start by looking at Foldable, and then examine cases where folding becomes complex and Traverse becomes convenient.

12.1 Foldable

The Foldable type class captures the foldLeft and foldRight methods we’re used to in sequences like Lists, Vectors, and Streams. Using Foldable, we can write generic folds that work with a variety of sequence types. We can also invent new sequences and plug them into our code. Foldable gives us great use cases for Monoids and the Eval monad.

12.1.1 Folds and Folding

Let’s start with a quick recap of the general concept of folding. We supply an accumulator value and a binary function to combine it with each item in the sequence:

def show[A](list: List[A]): String =
  list.foldLeft("nil")((accum, item) => s"$item then $accum")
show(Nil)
// res0: String = "nil"

show(List(1, 2, 3))
// res1: String = "3 then 2 then 1 then nil"

The foldLeft method works recursively down the sequence. Our binary function is called repeatedly for each item, the result of each call becoming the accumulator for the next. When we reach the end of the sequence, the final accumulator becomes our final result.

Depending on the operation we’re performing, the order in which we fold may be important. Because of this there are two standard variants of fold:

Figure 11 illustrates each direction.

fold Created with Sketch. 1 1 2 2 3 3 0 3 5 6 + + + 1 2 3 1 2 3 0 1 3 + + + 6
Figure 11: Illustration of foldLeft and foldRight

foldLeft and foldRight are equivalent if our binary operation is associative. For example, we can sum a List[Int] by folding in either direction, using 0 as our accumulator and addition as our operation:

List(1, 2, 3).foldLeft(0)(_ + _)
// res2: Int = 6
List(1, 2, 3).foldRight(0)(_ + _)
// res3: Int = 6

If we provide a non-associative operator the order of evaluation makes a difference. For example, if we fold using subtraction, we get different results in each direction:

List(1, 2, 3).foldLeft(0)(_ - _)
// res4: Int = -6
List(1, 2, 3).foldRight(0)(_ - _)
// res5: Int = 2

12.1.2 Exercise: Reflecting on Folds

Try using foldLeft and foldRight with an empty list as the accumulator and :: as the binary operator. What results do you get in each case?

Folding from left to right reverses the list:

List(1, 2, 3).foldLeft(List.empty[Int])((a, i) => i :: a)
// res6: List[Int] = List(3, 2, 1)

Folding right to left copies the list, leaving the order intact:

List(1, 2, 3).foldRight(List.empty[Int])((i, a) => i :: a)
// res7: List[Int] = List(1, 2, 3)

Note that we have to carefully specify the type of the accumulator to avoid a type error. We use List.empty[Int] to avoid inferring the accumulator type as Nil.type or List[Nothing]:

List(1, 2, 3).foldRight(Nil)(_ :: _)
// error:
// Found:    List[Int]
// Required: scala.collection.immutable.Nil.type
// List(1, 2, 3).foldRight(Nil)(_ :: _)
//                              ^^^^^^

12.1.3 Exercise: Scaf-fold-ing Other Methods

foldLeft and foldRight are very general methods. We can use them to implement many of the other high-level sequence operations we know. Prove this to yourself by implementing substitutes for List's map, flatMap, filter, and sum methods in terms of foldRight.

Here are the solutions:

def map[A, B](list: List[A])(func: A => B): List[B] =
  list.foldRight(List.empty[B]) { (item, accum) =>
    func(item) :: accum
  }
map(List(1, 2, 3))(_ * 2)
// res9: List[Int] = List(2, 4, 6)
def flatMap[A, B](list: List[A])(func: A => List[B]): List[B] =
  list.foldRight(List.empty[B]) { (item, accum) =>
    func(item) ::: accum
  }
flatMap(List(1, 2, 3))(a => List(a, a * 10, a * 100))
// res10: List[Int] = List(1, 10, 100, 2, 20, 200, 3, 30, 300)
def filter[A](list: List[A])(func: A => Boolean): List[A] =
  list.foldRight(List.empty[A]) { (item, accum) =>
    if(func(item)) item :: accum else accum
  }
filter(List(1, 2, 3))(_ % 2 == 1)
// res11: List[Int] = List(1, 3)

We’ve provided two definitions of sum, one using scala.math.Numeric (which recreates the built-in functionality accurately)…

import scala.math.Numeric

def sumWithNumeric[A](list: List[A])
      (implicit numeric: Numeric[A]): A =
  list.foldRight(numeric.zero)(numeric.plus)
sumWithNumeric(List(1, 2, 3))
// res12: Int = 6

and one using cats.Monoid (which is more appropriate to the content of this book):

import cats.Monoid

def sumWithMonoid[A](list: List[A])
      (implicit monoid: Monoid[A]): A =
  list.foldRight(monoid.empty)(monoid.combine)

import cats.instances.int._ // for Monoid
sumWithMonoid(List(1, 2, 3))
// res13: Int = 6

12.1.4 Foldable in Cats

Cats’ Foldable abstracts foldLeft and foldRight into a type class. Instances of Foldable define these two methods and inherit a host of derived methods. Cats provides out-of-the-box instances of Foldable for a handful of Scala data types: List, Vector, LazyList, and Option.

We can summon instances as usual using Foldable.apply and call their implementations of foldLeft directly. Here is an example using List:

import cats.Foldable
import cats.instances.list._ // for Foldable

val ints = List(1, 2, 3)
Foldable[List].foldLeft(ints, 0)(_ + _)
// res0: Int = 6

Other sequences like Vector and LazyList work in the same way. Here is an example using Option, which is treated like a sequence of zero or one elements:

import cats.instances.option._ // for Foldable

val maybeInt = Option(123)
Foldable[Option].foldLeft(maybeInt, 10)(_ * _)
// res1: Int = 1230

12.1.4.1 Folding Right

Foldable defines foldRight differently to foldLeft, in terms of the Eval monad:

def foldRight[A, B](fa: F[A], lb: Eval[B])
                     (f: (A, Eval[B]) => Eval[B]): Eval[B]

Using Eval means folding is always stack safe, even when the collection’s default definition of foldRight is not. For example, the default implementation of foldRight for LazyList is not stack safe. The longer the lazy list, the larger the stack requirements for the fold. A sufficiently large lazy list will trigger a StackOverflowError:

import cats.Eval
import cats.Foldable

def bigData = (1 to 100000).to(LazyList)
bigData.foldRight(0L)(_ + _)
// java.lang.StackOverflowError ...

Using Foldable forces us to use stack safe operations, which fixes the overflow exception:

import cats.instances.lazyList._ // for Foldable
val eval: Eval[Long] =
  Foldable[LazyList].
    foldRight(bigData, Eval.now(0L)) { (num, eval) =>
      eval.map(_ + num)
    }
eval.value
// res3: Long = 5000050000L

Stack Safety in the Standard Library

Stack safety isn’t typically an issue when using the standard library. The most commonly used collection types, such as List and Vector, provide stack safe implementations of foldRight:

(1 to 100000).toList.foldRight(0L)(_ + _)
(1 to 100000).toVector.foldRight(0L)(_ + _)
// res4: Long = 5000050000L

We’ve called out Stream because it is an exception to this rule. Whatever data type we’re using, though, it’s useful to know that Eval has our back.

12.1.4.2 Folding with Monoids

Foldable provides us with a host of useful methods defined on top of foldLeft. Many of these are facsimiles of familiar methods from the standard library: find, exists, forall, toList, isEmpty, nonEmpty, and so on:

Foldable[Option].nonEmpty(Option(42))
// res5: Boolean = true

Foldable[List].find(List(1, 2, 3))(_ % 2 == 0)
// res6: Option[Int] = Some(value = 2)

In addition to these familiar methods, Cats provides two methods that make use of Monoids:

For example, we can use combineAll to sum over a List[Int]:

import cats.instances.int._ // for Monoid
Foldable[List].combineAll(List(1, 2, 3))
// res7: Int = 6

Alternatively, we can use foldMap to convert each Int to a String and concatenate them:

import cats.instances.string._ // for Monoid
Foldable[List].foldMap(List(1, 2, 3))(_.toString)
// res8: String = "123"

Finally, we can compose Foldables to support deep traversal of nested sequences:

import cats.instances.vector._ // for Monoid

val ints = List(Vector(1, 2, 3), Vector(4, 5, 6))
(Foldable[List] compose Foldable[Vector]).combineAll(ints)
// res10: Int = 21

12.1.4.3 Syntax for Foldable

Every method in Foldable is available in syntax form via cats.syntax.foldable. In each case, the first argument to the method on Foldable becomes the receiver of the method call:

import cats.syntax.foldable._ // for combineAll and foldMap
List(1, 2, 3).combineAll
// res11: Int = 6

List(1, 2, 3).foldMap(_.toString)
// res12: String = "123"

Explicits over Implicits

Remember that Scala will only use an instance of Foldable if the method isn’t explicitly available on the receiver. For example, the following code will use the version of foldLeft defined on List:

List(1, 2, 3).foldLeft(0)(_ + _)
// res13: Int = 6

whereas the following generic code will use Foldable:

def sum[F[_]: Foldable](values: F[Int]): Int =
  values.foldLeft(0)(_ + _)

We typically don’t need to worry about this distinction. It’s a feature! We call the method we want and the compiler uses a Foldable when needed to ensure our code works as expected. If we need a stack-safe implementation of foldRight, using Eval as the accumulator is enough to force the compiler to select the method from Cats.

12.2 Traverse

foldLeft and foldRight are flexible iteration methods but they require us to do a lot of work to define accumulators and combinator functions. The Traverse type class is a higher level tool that leverages Applicatives to provide a more convenient, more lawful, pattern for iteration.

12.2.1 Traversing with Futures

We can demonstrate Traverse using the Future.traverse and Future.sequence methods in the Scala standard library. These methods provide Future-specific implementations of the traverse pattern. As an example, suppose we have a list of server hostnames and a method to poll a host for its uptime:

import scala.concurrent._
import scala.concurrent.duration._
import scala.concurrent.ExecutionContext.Implicits.global

val hostnames = List(
  "alpha.example.com",
  "beta.example.com",
  "gamma.demo.com"
)

def getUptime(hostname: String): Future[Int] =
  Future(hostname.length * 60) // just for demonstration

Now, suppose we want to poll all of the hosts and collect all of their uptimes. We can’t simply map over hostnames because the result—a List[Future[Int]]—would contain more than one Future. We need to reduce the results to a single Future to get something we can block on. Let’s start by doing this manually using a fold:

val allUptimes: Future[List[Int]] =
  hostnames.foldLeft(Future(List.empty[Int])) {
    (accum, host) =>
      val uptime = getUptime(host)
      for {
        accum  <- accum
        uptime <- uptime
      } yield accum :+ uptime
  }
Await.result(allUptimes, 1.second)
// res0: List[Int] = List(1020, 960, 840)

Intuitively, we iterate over hostnames, call func for each item, and combine the results into a list. This sounds simple, but the code is fairly unwieldy because of the need to create and combine Futures at every iteration. We can improve on things greatly using Future.traverse, which is tailor-made for this pattern:

val allUptimes: Future[List[Int]] =
  Future.traverse(hostnames)(getUptime)
Await.result(allUptimes, 1.second)
// res2: List[Int] = List(1020, 960, 840)

This is much clearer and more concise—let’s see how it works. If we ignore distractions like CanBuildFrom and ExecutionContext, the implementation of Future.traverse in the standard library looks like this:

def traverse[A, B](values: List[A])
    (func: A => Future[B]): Future[List[B]] =
  values.foldLeft(Future(List.empty[B])) { (accum, host) =>
    val item = func(host)
    for {
      accum <- accum
      item  <- item
    } yield accum :+ item
  }

This is essentially the same as our example code above. Future.traverse is abstracting away the pain of folding and defining accumulators and combination functions. It gives us a clean high-level interface to do what we want:

The standard library also provides another method, Future.sequence, that assumes we’re starting with a List[Future[B]] and don’t need to provide an identity function:

object Future {
  def sequence[B](futures: List[Future[B]]): Future[List[B]] =
    traverse(futures)(identity)

  // etc...
}

In this case the intuitive understanding is even simpler:

Future.traverse and Future.sequence solve a very specific problem: they allow us to iterate over a sequence of Futures and accumulate a result. The simplified examples above only work with Lists, but the real Future.traverse and Future.sequence work with any standard Scala collection.

Cats’ Traverse type class generalises these patterns to work with any type of Applicative: Future, Option, Validated, and so on. We’ll approach Traverse in the next sections in two steps: first we’ll generalise over the Applicative, then we’ll generalise over the sequence type. We’ll end up with an extremely valuable tool that trivialises many operations involving sequences and other data types.

12.2.2 Traversing with Applicatives

If we squint, we’ll see that we can rewrite traverse in terms of an Applicative. Our accumulator from the example above:

Future(List.empty[Int])

is equivalent to Applicative.pure:

import cats.Applicative
import cats.instances.future._   // for Applicative
import cats.syntax.applicative._ // for pure

List.empty[Int].pure[Future]

Our combinator, which used to be this:

def oldCombine(
  accum : Future[List[Int]],
  host  : String
): Future[List[Int]] = {
  val uptime = getUptime(host)
  for {
    accum  <- accum
    uptime <- uptime
  } yield accum :+ uptime
}

is now equivalent to Semigroupal.combine:

import cats.syntax.apply._ // for mapN

// Combining accumulator and hostname using an Applicative:
def newCombine(accum: Future[List[Int]],
      host: String): Future[List[Int]] =
  (accum, getUptime(host)).mapN(_ :+ _)

By substituting these snippets back into the definition of traverse we can generalise it to to work with any Applicative:


def listTraverse[F[_]: Applicative, A, B]
      (list: List[A])(func: A => F[B]): F[List[B]] =
  list.foldLeft(List.empty[B].pure[F]) { (accum, item) =>
    (accum, func(item)).mapN(_ :+ _)
  }

def listSequence[F[_]: Applicative, B]
      (list: List[F[B]]): F[List[B]] =
  listTraverse(list)(identity)

We can use listTraverse to re-implement our uptime example:

val totalUptime = listTraverse(hostnames)(getUptime)
Await.result(totalUptime, 1.second)
// res5: List[Int] = List(1020, 960, 840)

or we can use it with other Applicative data types as shown in the following exercises.

12.2.2.1 Exercise: Traversing with Vectors

What is the result of the following?

import cats.instances.vector._ // for Applicative

listSequence(List(Vector(1, 2), Vector(3, 4)))

The argument is of type List[Vector[Int]], so we’re using the Applicative for Vector and the return type is going to be Vector[List[Int]].

Vector is a monad, so its semigroupal combine function is based on flatMap. We’ll end up with a Vector of Lists of all the possible combinations of List(1, 2) and List(3, 4):

listSequence(List(Vector(1, 2), Vector(3, 4)))
// res7: Vector[List[Int]] = Vector(
//   List(1, 3),
//   List(1, 4),
//   List(2, 3),
//   List(2, 4)
// )

What about a list of three parameters?

listSequence(List(Vector(1, 2), Vector(3, 4), Vector(5, 6)))

With three items in the input list, we end up with combinations of three Ints: one from the first item, one from the second, and one from the third:

listSequence(List(Vector(1, 2), Vector(3, 4), Vector(5, 6)))
// res9: Vector[List[Int]] = Vector(
//   List(1, 3, 5),
//   List(1, 3, 6),
//   List(1, 4, 5),
//   List(1, 4, 6),
//   List(2, 3, 5),
//   List(2, 3, 6),
//   List(2, 4, 5),
//   List(2, 4, 6)
// )

12.2.2.2 Exercise: Traversing with Options

Here’s an example that uses Options:

import cats.instances.option._ // for Applicative

def process(inputs: List[Int]) =
  listTraverse(inputs)(n => if(n % 2 == 0) Some(n) else None)

What is the return type of this method? What does it produce for the following inputs?

process(List(2, 4, 6))
process(List(1, 2, 3))

The arguments to listTraverse are of types List[Int] and Int => Option[Int], so the return type is Option[List[Int]]. Again, Option is a monad, so the semigroupal combine function follows from flatMap. The semantics are therefore fail-fast error handling: if all inputs are even, we get a list of outputs. Otherwise we get None:

process(List(2, 4, 6))
// res12: Option[List[Int]] = Some(value = List(2, 4, 6))
process(List(1, 2, 3))
// res13: Option[List[Int]] = None

12.2.2.3 Exercise: Traversing with Validated

Finally, here is an example that uses Validated:

import cats.data.Validated
import cats.instances.list._ // for Monoid

type ErrorsOr[A] = Validated[List[String], A]

def process(inputs: List[Int]): ErrorsOr[List[Int]] =
  listTraverse(inputs) { n =>
    if(n % 2 == 0) {
      Validated.valid(n)
    } else {
      Validated.invalid(List(s"$n is not even"))
    }
  }

What does this method produce for the following inputs?

process(List(2, 4, 6))
process(List(1, 2, 3))

The return type here is ErrorsOr[List[Int]], which expands to Validated[List[String], List[Int]]. The semantics for semigroupal combine on validated are accumulating error handling, so the result is either a list of even Ints, or a list of errors detailing which Ints failed the test:

process(List(2, 4, 6))
// res17: Validated[List[String], List[Int]] = Valid(a = List(2, 4, 6))
process(List(1, 2, 3))
// res18: Validated[List[String], List[Int]] = Invalid(
//   e = List("1 is not even", "3 is not even")
// )

12.2.3 Traverse in Cats

Our listTraverse and listSequence methods work with any type of Applicative, but they only work with one type of sequence: List. We can generalise over different sequence types using a type class, which brings us to Cats’ Traverse. Here’s the abbreviated definition:

package cats

trait Traverse[F[_]] {
  def traverse[G[_]: Applicative, A, B]
      (inputs: F[A])(func: A => G[B]): G[F[B]]

  def sequence[G[_]: Applicative, B]
      (inputs: F[G[B]]): G[F[B]] =
    traverse(inputs)(identity)
}

Cats provides instances of Traverse for List, Vector, Stream, Option, Either, and a variety of other types. We can summon instances as usual using Traverse.apply and use the traverse and sequence methods as described in the previous section:

import cats.Traverse
import cats.instances.future._ // for Applicative
import cats.instances.list._   // for Traverse

val totalUptime: Future[List[Int]] =
  Traverse[List].traverse(hostnames)(getUptime)
Await.result(totalUptime, 1.second)
// res0: List[Int] = List(1020, 960, 840)
val numbers = List(Future(1), Future(2), Future(3))

val numbers2: Future[List[Int]] =
  Traverse[List].sequence(numbers)
Await.result(numbers2, 1.second)
// res1: List[Int] = List(1, 2, 3)

There are also syntax versions of the methods, imported via cats.syntax.traverse:

import cats.syntax.traverse._ // for sequence and traverse
val numbers3 = hostnames.traverse(getUptime)
// numbers3: Future[List[Int]] = Future(Success(List(1020, 960, 840)))
val numbers4 = numbers.sequence
// numbers4: Future[List[Int]] = Future(Success(List(1, 2, 3)))

Await.result(numbers3, 1.second)
// res2: List[Int] = List(1020, 960, 840)
Await.result(numbers4, 1.second)
// res3: List[Int] = List(1, 2, 3)

As you can see, this is much more compact and readable than the foldLeft code we started with earlier this chapter!

12.3 Summary

In this chapter we were introduced to Foldable and Traverse, two type classes for iterating over sequences.

Foldable abstracts the foldLeft and foldRight methods we know from collections in the standard library. It adds stack-safe implementations of these methods to a handful of extra data types, and defines a host of situationally useful additions. That said, Foldable doesn’t introduce much that we didn’t already know.

The real power comes from Traverse, which abstracts and generalises the traverse and sequence methods we know from Future. Using these methods we can turn an F[G[A]] into a G[F[A]] for any F with an instance of Traverse and any G with an instance of Applicative. In terms of the reduction we get in lines of code, Traverse is one of the most powerful patterns in this book. We can reduce folds of many lines down to a single foo.traverse.


…and with that, we’ve finished all of the theory in this book. There’s plenty more to come, though, as we put everything we’ve learned into practice in a series of in-depth case studies in Part II!

13 Indexed Types

In this chapter we look at indexed types. An indexed type is a type constructor, so a type like F[_], along with a set of types that can fill in the constructor’s type parameters. Let’s say this set of types is Int, String, and Option[Double]. Then, for a type constructor F we can construct an indexed type from the set F[Int], F[String], and F[Option[Double]]. The types Int, String, and Option[Double] act as indices into the set F[Int], F[String], and F[Option[Double]], hence the name. The type constructor F can be either data and codata.

The description above is very abstract, and doesn’t help us understand how indexed types are useful. We’ll see a lot of details and examples in this chapter, but let’s start with a more useful high-level overview. We can think of indexed types as working with proofs that a type parameter is equal to a particular element from the set of indices. Indexed data provides this evidence when we destructure it, while indexed codata requires this evidence when we call methods. Remember the definition of algebras we gave in Section 5.2, where we said an algebra consists of three different kinds of methods: constructors, combinators, and interpreters. Indexed types allows us to do two things:

Indexed data are more usually known as generalized algebraic data types. Indexed codata are sometimes known as typestate. Both can make use of what is known as phantom types. Indeed, an early name for indexed data was first-class phantom types. As you might expect, indexed data and indexed codata are dual to one another.

13.1 Phantom Types

Phantom types are a basic building block of indexed types, so we’ll start with an example of them. A phantom type is simply a type parameter that doesn’t correspond to any value. In the example below, the type parameter A is a phantom type, because there is no value of type A, while B is not because there is a value of that type.

final case class PhantomExample[A, B](value: B)

Phantom types are used to shift constraints to compile time. A simple example involves units of measurement. Most of the world has standardized on SI units, such as metres and litres. However, other measuring systems, such as Imperial units, remain in use some countries or in some niches within countries that otherwise use metric. Differences between different measurement systems can cause problems. A dramatic example is the loss of the Mars orbiter, caused by two software components using incompatible measurements (one using metric, and one using US customary measurements.)

With phantom types we can annotate measurements with their units, which in turn can prevent us ever using incompatible units. Let’s work with just length, which is sufficient to show the idea. We’ll start by defining a length type with a phantom type recording the unit, and a method that allows us to add together lengths.

final case class Length[Unit](value: Double) {
  def +(that: Length[Unit]): Length[Unit] =
    Length[Unit](this.value + that.value)
}

We’ll need to define a few unit types to use this, and some Lengths using these units.

trait Metres
trait Feet

val threeMetres = Length[Metres](3)
val threeFeetAndRising = Length[Feet](3)

Now we can add Lengths together if they have the same unit.

threeMetres + threeMetres
// res0: Length[Metres] = Length(value = 6.0)

However if we try to add Lengths with different units the code will not compile.

threeMetres + threeFeetAndRising
// error:
// Found:    (repl.MdocSession.MdocApp.threeFeetAndRising :
//   repl.MdocSession.MdocApp.Length[repl.MdocSession.MdocApp.Feet])
// Required: repl.MdocSession.MdocApp.Length[repl.MdocSession.MdocApp.Metres]
// threeMetres + threeFeetAndRising
//               ^^^^^^^^^^^^^^^^^^

There is one big problem with phantom types on their own: there is no way to use the information stored in the phantom type in further processing. For example, force times length gives torque (with the SI unit of newton metres). However we cannot define a * method on Length that can only be called if the Unit is Metre using just the tool of phantom types. Similarly, we cannot define, say, a toString method that uses the Unit type to appropriately print the result. Solving these problems leads us to indexed codata, so let’s now look at that.

13.2 Indexed Codata

The basic idea of indexed codata is to prevent methods being called unless certain conditions, encoded in types, are met. More precisely, methods are guarded by type equalities that callers must prove they satisfy to call a method. The contextual abstraction features, given instances and using clauses, are used to implement this in Scala.

We’ll start our exploration of indexed codata with a very simple example. We are going to define a switch that can only be turned on when it is off, and off when it is on. Since this is codata, we start with an interface.

trait Switch {
  def on: Switch
  def off: Switch
}

There are no constraints on this interface as defined; we can turn any switch on, even if it is already on, and vice versa. The first step to implement such a constraint is to add a type parameter, which will hold the state of the Switch. This type parameter doesn’t correspond to any data we store in Switch, so it is a phantom type.

trait Switch[A] {
  def on: Switch[A]
  def off: Switch[A]
}

We are now going to add constraints that say we can only call certain methods when this type parameter corresponds to particular concrete types. It is in this way that indexed codata goes beyond what phantom types alone can do: we can inspect, at compile-time, the type of a type parameter and make decisions based on this type.

Implementing these constraints has two parts. The first is defining types to represent on and off.

trait On
trait Off

The second step is to add the constraints to the relevant methods on Switch. Here is how we do it.

trait Switch[A] {
  def on(using ev: A =:= Off): Switch[On]
  def off(using ev: A =:= On): Switch[Off]
}

We can create an implementation to show it really works.

final case class SimpleSwitch[A]() extends Switch[A] {
  def on(using ev: A =:= Off): Switch[On] =
    SimpleSwitch()
  def off(using ev: A =:= On): Switch[Off] =
    SimpleSwitch()
}
object SimpleSwitch {
  val on: Switch[On] = SimpleSwitch()
  val off: Switch[Off] = SimpleSwitch()
}

Here are some examples of using it correctly

SimpleSwitch.on.off
// res2: Switch[Off] = SimpleSwitch()
SimpleSwitch.off.on
// res3: Switch[On] = SimpleSwitch()

Incorrect uses fail to compile.

SimpleSwitch.on.on
// error: 
// Cannot prove that MdocApp1.this.On =:= MdocApp1.this.Off.

The constraint is made of two parts: using clauses, which we learned about in Section 4, and the A =:= B construction, which is new. =:= represents a type equality. If a given instance A =:= B exists, then the type A is equal to the type B. (Note we can write this with the more familiar prefix notation =:=[A, B] if we prefer.) We never create these instances ourselves; instead the compiler creates them for us. In the method on we are asking the compiler to construct an instance A =:= Off, which can only be done if A is Off. This in turn means we can only call the method when the Switch is Off. This is the core idea of indexed codata: we raise states into types, and restrict method calls to a subset of states.

This is a different use of contextual abstraction to type classes. Type classes associate operations with types. What we’re doing here is proving some property of a type with respect to another type. More precisely we’re proving that a type parameter is equal to a particular type. The given instance only exists when the compiler can prove this is the case. Hence these given instances are sometimes called evidence or witnesses. This different view subsumes type classes, as we can think of type classes as evidence that a type implements a certain interface.

Exercise: Torque

In Section 13.1 we saw we could use phantom types to represent units. We also ran into a limitation: we had no way to inspect the phantom types and hence make decisions based on them. Now, with indexed codata, we can do solve this problem.

Below if the definition of Length we previously used. Your mission is to:

  1. implement a type Force, parameterized by a phantom type that represents the units of force;
  2. implement a type Torque, parameterized by a phantom type that represents the units of torque;
  3. define types Newtons and NewtonMetres to represent force in SI units;
  4. implement a method * on Force that accepts a Length and returns a Torque. It can only be called if the Force is in Newtons and the Length is in Metres. In this case the Torque is in NewtonMetres. (Torque is force times length.)
final case class Length[Unit](value: Double) {
  def +(that: Length[Unit]): Length[Unit] =
    Length[Unit](this.value + that.value)
}

Defining Force, Torque, and the unit types is a repeat of the pattern we saw in the example code.

trait Newtons
trait NewtonMetres

final case class Force[Unit](value: Double)
final case class Torque[Unit](value: Double)

To define the * method on Force we need constraints that specify Force's Unit type is Newtons, and Length's Unit type is Metres. These are both type equalities, so we can express them with =:=.

final case class Force[Unit](value: Double) {
  def *[L](length: Length[L])(using Unit =:= Newtons, L =:= Metres): Torque[NewtonMetres] =
    Torque(this.value * length.value)
}

13.2.1 API Protocols

An API protocol defines the order in which methods must be called. The protocol in the case of Switch is that we can only call off after calling on and vice versa. This protocol is a simple finite state machine, and illustrated in Figure 12. Many common types have similar protocols. For example, files can only be read once they are opened and cannot be read once they have been closed.

Figure 12: The switch API protocol

Indexed codata allows us to enforce API protocols at compile-time. Often these protocols are finite-state machines. We can represent these protocols with a single type parameter that represents the state, as we did with Switch. We can also use multiple type parameters if that makes for a more convenient representation.

Let’s see an example using multiple type parameters. We’re going to build an API that represents a very limited subset of HTML, the language the defines web pages. An example of HTML is below.

<!DOCTYPE html>
<html>
  <head><title>Our Amazing Web Page</title></head>
  <body>
    <h1>This Is Our Amazing Web Page</h1>
    <p>Please be in awe of its <strong>amazingness</strong></p>
  </body>
</html>

In HTML the content of the page is marked up with tags, like <h1>, that give it meaning. For example, <h1> means a heading at level one, and <p> means a paragraph. An opening tag is closed by a corresponding closing tag, such as </h1> for <h1> and </p> for <p>.

There are several rules for valid HTML12. We’re going to focus on the following:

  1. Within the html tag there can only be a head and a body tag, in that order.
  2. Within the head tag there must be exactly one title, and there can be any other number of allowed tags (of which we’re only going to model link).
  3. Within the body there can be any number of allowed tags (of which we are only going to model h1 and p).

We’re going to use a Church-encoded representation for HTML, so tags are created by method calls. Figure 13 shows the finite state machine representation of the API protocol. I find it easier to read as a regular expression, which we can write down as

head link* title link* body (h1 | p)*

Figure 13: The HTML API protocol

As the code is fairly repetitive I will just present all the code and then discuss the important parts. Here’s the implementation.

sealed trait StructureState
trait Empty extends StructureState
trait InHead extends StructureState
trait InBody extends StructureState

sealed trait TitleState
trait WithoutTitle extends TitleState
trait WithTitle extends TitleState

// Not a case class so external users cannot copy it
// and break invariants
final class Html[S <: StructureState, T <: TitleState](
    head: Vector[String],
    body: Vector[String]
) {
  // Head tags ---------------------------------------------

  def head(using S =:= Empty): Html[InHead, WithoutTitle] =
    Html(head, body)

  def title(
      text: String
  )(using S =:= InHead, T =:= WithoutTitle): Html[InHead, WithTitle] =
    Html(head :+ s"<title>$text</title>", this.body)

  def link(rel: String, href: String)(using S =:= InHead): Html[InHead, T] =
    Html(head :+ s"<link rel=\"$rel\" href=\"$href\"/>", body)

  // Body tags ---------------------------------------------

  def body(using S =:= InHead, T =:= WithTitle): Html[InBody, WithTitle] =
    Html(head, body)

  def h1(text: String)(using S =:= InBody): Html[InBody, T] =
    Html(head, body :+ s"<h1>$text</h1>")

  def p(text: String)(using S =:= InBody): Html[InBody, T] =
    Html(head, body :+ s"<p>$text</p>")

  // Interpreter ------------------------------------------

  override def toString(): String = {
    val h = head.mkString("  <head>\n    ", "\n    ", "\n  </head>")
    val b = body.mkString("  <body>\n    ", "\n    ", "\n  </body>")

    s"\n<html>\n$h\n$b\n</html>"
  }
}
object Html {
  val empty: Html[Empty, WithoutTitle] = Html(Vector.empty, Vector.empty)
}

The key point is that we factor the state into two components. StructureState represents where in the overall structure we are (inside the head, inside the body, or inside neither). TitleState represents the state when defining the elements inside the head, specifically whether we have a title element or not. We could certainly represent this with one state type variable, but I find the factored representation both easier to work with and easier for other developers to understand.

Here’s an example in use.

Html.empty.head
  .link("stylesheet", "styles.css")
  .title("Our Amazing Webpage")
  .body
  .h1("Where Amazing Exists")
  .p("Right here")
  .toString
// res6: String = """
// <html>
//   <head>
//     <link rel="stylesheet" href="styles.css"/>
//     <title>Our Amazing Webpage</title>
//   </head>
//   <body>
//     <h1>Where Amazing Exists</h1>
//     <p>Right here</p>
//   </body>
// </html>"""

Here’s an example of the type system preventing an invalid construction, in this case the lack of a title.

Html.empty.head
  .link("stylesheet", "styles.css")
  .body
  .h1("This Shouldn't Work")
// error: 
// Cannot prove that MdocApp2.this.WithoutTitle =:= MdocApp2.this.WithTitle.

These error messages are not great. We’ll address this in Chapter 16.

We can implement more complex protocols, such as those that can be represented by context-free or even context-sensitive grammars, using the same technique.

Exercise: HTML API Design

I don’t particularly like the HTML API we developed above, as the flat method call structure doesn’t match the nesting in the HTML structure we’re creating. I would prefer to write the following.

Html.empty
  .head(_.title("Our Amazing Webpage"))
  .body(_.h1("Where Amazing Happens").p("Right here"))
  .toString

We still require the head is specified before the body, but now the nesting of the method calls matches the nesting of the structure. Notice we’re still using a Church-encoded representation.

Can you think of how to implement this? You’ll need to use indexed codata, and perhaps a bit of inspiration. This is a very open ended question, so don’t worry if you struggle with it!

Here’s how I implemented it. The structure is very similar to the original implementation, but where we factored the state into type parameters I also factored the implementation into types. Notice how we use Head and Body to accumulate the set of tags that make up the head and body respectively. We still need to use indexed codata in some place, but we can avoid it in others. For example, the head method simply requires a function of type Head[WithoutTitle] => Head[WithTitle].

sealed trait StructureState
trait NeedsHead extends StructureState
trait NeedsBody extends StructureState
trait Complete extends StructureState

sealed trait TitleState
trait WithoutTitle extends TitleState
trait WithTitle extends TitleState

final class Head[S <: TitleState](contents: Vector[String]) {
  def title(text: String)(using S =:= WithoutTitle): Head[WithTitle] =
    Head(contents :+ s"<title>$text</title>")

  def link(rel: String, href: String): Head[S] =
    Head(contents :+ s"<link rel=\"$rel\" href=\"$href\"/>")

  override def toString(): String =
    contents.mkString("  <head>\n    ", "\n    ", "\n  </head>")
}
object Head {
  val empty: Head[WithoutTitle] = Head(Vector.empty)
}

final class Body(contents: Vector[String]) {
  def h1(text: String): Body =
    Body(contents :+ s"<h1>$text</h1>")

  def p(text: String): Body =
    Body(contents :+ s"<p>$text</p>")

  override def toString(): String =
    contents.mkString("  <body>\n    ", "\n    ", "\n  </body>")
}
object Body {
  val empty: Body = Body(Vector.empty)
}

final class Html[S <: StructureState](
    head: Head[?],
    body: Body
) {
  def head(f: Head[WithoutTitle] => Head[WithTitle])(using
      S =:= NeedsHead
  ): Html[NeedsBody] =
    Html(f(Head.empty), body)

  def body(f: Body => Body)(using S =:= NeedsBody): Html[Complete] =
    Html(head, f(Body.empty))

  override def toString(): String = {
    s"\n<html>\n${head.toString()}\n${body.toString()}\n</html>"

  }
}
object Html {
  val empty: Html[NeedsHead] = Html(Head.empty, Body.empty)
}

As always, we should show that is works. Here’s the output from the motivating example.

Html.empty
  .head(_.title("Our Amazing Webpage"))
  .body(_.h1("Where Amazing Happens").p("Right here"))
  .toString()
// res9: String = """
// <html>
//   <head>
//     <title>Our Amazing Webpage</title>
//   </head>
//   <body>
//     <h1>Where Amazing Happens</h1>
//     <p>Right here</p>
//   </body>
// </html>"""

13.2.2 Beyond Equality Constraints

Indexed data is all about equality constraints: proofs that some type parameter is equal to some type. However we can go beyond equality constraints with contextual abstraction. We can use <:< for evidence of a subtyping relationship, and NotGiven for evidence that no given instance exists (with which we can test that types are not equal, for example). Beyond that, we can view any given instance as evidence.

Let’s return to our example of length, force, and torque to see how this is useful. In the exercise where we defined torque as force times length, we fixed the computation to have SI units. The example code is below.

final case class Force[Unit](value: Double) {
  def *[L](length: Length[L])(using Unit =:= Newtons, L =:= Metres): Torque[NewtonMetres] =
    Torque(this.value * length.value)
}

This is a reasonable thing to do, as other units are insane, but there are a lot of insane people out there. To accommodate other unit types we can create given instances that represent the result types of operations of interest. In this case we want to represent the result of multiplying a length unit by the force unit. In code we can write the following.

// Weird units
trait Feet
trait Pounds
trait PoundsFeet

// An instance exists if A * B = C
trait Multiply[A, B, C]
object Multiply {
  given Multiply[Metres, Newtons, NewtonMetres] = new Multiply {}
  given Multiply[Feet, Pounds, PoundsFeet] = new Multiply {}
}

Now we can define * methods on Length and Force in terms of Multiply.

final case class Length[L](value: Double) {
  def *[F, T](that: Force[F])(using Multiply[L, F, T]): Torque[T] =
    Torque(this.value * that.value)
}

final case class Force[F](value: Double) {
  def *[L, T](that: Length[L])(using Multiply[F, L, T]): Torque[T] =
    Torque(this.value * that.value)
}

Here’s an example showing it works.

Length[Metres](3) * Force[Newtons](4)
// res11: Torque[NewtonMetres] = Torque(value = 12.0)

Length[Feet](3) * Force[Pounds](4)
// res12: Torque[PoundsFeet] = Torque(value = 12.0)

Note that’s it hard to think of Multiply as a type class, as it does not provide any methods. Viewing it as evidence, however, does make sense.

Exercise: Commutivitiy

In the example above we defined a Multiply type class to represent that metres times newtons gives newton metres. Multiplication is commutative. If A × B = C, then B × A = C. However we have not represented this, and if we try newtons times metres, as in the example below, the code will fail.

Force[Newtons](3) * Length[Metres](4)
// error:
// No given instance of type MdocApp4.this.Multiply[MdocApp4.this.Newtons, MdocApp4.this.Metres, Any] was found for parameter x$2 of method * in class Force
// Force[Newtons](3) * Length[Metres](4)
//                                     ^

Add evidence to Multiply that if Multiply[A, B, C] exists, then so does Multiply[B, A, C], and show that it solves this problem.

To solve this I defined a given instance called commutative, as shown below.

// An instance exists if A * B = C
trait Multiply[A, B, C]
object Multiply {
  given Multiply[Metres, Newtons, NewtonMetres] = new Multiply {}
  
  // A * B == B * A
  given commutative[A, B, C](using Multiply[A, B, C]): Multiply[B, A, C] =
    new Multiply {}
}

Now the example works as expected.

Force[Newtons](3) * Length[Metres](4)
// res15: Torque[NewtonMetres] = Torque(value = 12.0)

Now that we have learned about indexed codata, we’ll turn to its dual, indexed data.

13.3 Indexed Data

The key idea of indexed data is to encode type equalities in data. When we come to inspect the data (usually, via structural recursion) we discover these equalities, which in turn limit what values we can produce. Notice, again, the duality with codata. Indexed codata limits methods we can call. Indexed data limits values we can produce. Also, remember that indexed data is often known as generalized algebraic data types. We are using the simpler term indexed data to emphasise the relationship to indexed codata, and also because it’s much easier to type!

Concretely, indexed data in Scala occurs when:

  1. we define a sum type with at least one type parameter; and
  2. cases within the sum instantiate that type parameter with a concrete type.

Let’s see an example. Imagine we are implementing a programming language. We need some representation of values within the language. Suppose our language supports strings, integers, and doubles, which we will represent with the corresponding Scala types. The code below shows how we can implement this as a standard algebraic data type.

enum Value {
  case VString(value: String)
  case VInt(value: Int)
  case VDouble(value: Double)
}

Using indexed data we can use the alternate implementation below.

enum Value[A] {
  case VString(value: String) extends Value[String]
  case VInt(value: Int) extends Value[Int]
  case VDouble(value: Double) extends Value[Double]
}

This is indexed data, as it meets the criteria above: we have a type parameter A that is instantiated with a concrete type in the cases VString, VInt, and VDouble. It’s quite easy to use indexed data in Scala, and people often do so not knowing that it is anything special. The natural next question is why is this useful? It will take a more involved example to show why, so let us now dive into one that makes good use of indexed data.

13.3.1 The Probability Monad

For our case study of indexed data we will create a probability monad. This is a composable abstraction for defining probability distributions. The probability monad has a lot of uses. The most relevant to most developers is generating data for property-based tests, so we’ll focus on this use case. However, it can also be used, for example, for statistical inference or for creating generative art. See the conclusions (Section 13.4) for some pointers to these uses.

Let’s start with an example of generating random data. Doodle is a Scala library for graphics and visualization. A core part of the library is representing colors. Doodle has two different representations of colors, RGB and OkLCH, with conversions between the two. These conversions involve some somewhat tricky mathematics. Testing these conversions is an excellent use of property-based testing. If we can generate many, say, random RGB colors, we can test the conversion by checking the roundrip from RGB to OkLCH and back results in the original color13.

To create an RGB color we need three unsigned bytes, so our first task is to define how we generate a random byte. Doodle happens to have an implementation of the probability monad that we will use. Here is how we can do it.

import cats.syntax.all.*
import doodle.core.Color
import doodle.core.UnsignedByte
import doodle.random.{*, given}

val randomByte: Random[UnsignedByte] = 
  Random.int(0, 255).map(UnsignedByte.clip)

Note that once again we see the interpreter strategy. A Random[A] is a value representing a program that will generate a random value of type A when it runs.

With three random unsigned bytes we can create a random RGB color.

val randomRGB: Random[Color] =
  (randomByte, randomByte, randomByte)
    .mapN((r, g, b) => Color.rgb(r, g, b))

We might want to check our code by generating a few random values.

randomRGB.replicateA(2).run
// res1: List[Color] = List(
//   Rgb(
//     r = UnsignedByte(value = -25),
//     g = UnsignedByte(value = 22),
//     b = UnsignedByte(value = 43),
//     a = Normalized(get = 1.0)
//   ),
//   Rgb(
//     r = UnsignedByte(value = 89),
//     g = UnsignedByte(value = -123),
//     b = UnsignedByte(value = 110),
//     a = Normalized(get = 1.0)
//   )
// )

It seems to be working.

Once we have a source of random data we can write tests using it. We can easily generate more data than is feasible for a programmer to write by hand, and therefore have a higher degree of certainty that our code is correct than we would get with manual testing. The details of writing the tests are not important to us here, so let’s move on.

We have seen is an illustration of using the probability monad to generate random data. The probability monad works the same way as every other algebra: we have constructors (Random.int), combinators (map, and mapN), and interpreters (run). Being a monad means the algebra has some specific structure. For example, it tells us that we have pure and flatMap available, from which we can derive mapN.

Let’s sketch an plausible interface for our probability monad.

trait Random[A] {
  def flatMap[B](f: A => Random[B]): Random[B]
}
object Random {
  def pure[A](value: A): Random[A] = ???
  
  // Generate a uniformly distributed random  Double greater
  // than or equal to zero and less than one.
  val double: Random[Double] = ???
  
  // Generate a uniformly distributed random  Int
  val int: Random[Int] = ???
}

The interface has the minimum requirements to be a monad, and a few other constructors. We can make progress on the implementation by applying the reification strategy, introduced in Section 5.2.

enum Random[A] {
  def flatMap[B](f: A => Random[B]): Random[B] =
    RFlatMap(this, f)

  case RFlatMap[A, B](source: Random[A], f: A => Random[B])
      extends Random[B]
  case RPure(value: A)
  case RDouble extends Random[Double]
  case RInt extends Random[Int]
}
object Random {
  import Random.{RPure, RDouble, RInt}

  def pure[A](value: A): Random[A] = RPure(value)

  // Generate a uniformly distributed random  Double greater
  // than or equal to zero and less than one.
  val double: Random[Double] = RDouble

  // Generate a uniformly distributed random  Int
  val int: Random[Int] = RInt
}

The next step is to implement an interpreter, which is a standard structural recursion. The interpreter has a parameter that provides a source of random numbers.

def run(rng: scala.util.Random = scala.util.Random): A =
  this match {
    case RFlatMap(source, f) => f(source.run(rng)).run(rng)
    case RPure(value)        => value
    case RDouble             => rng.nextDouble()
    case RInt                => rng.nextInt()
  }

This is an example of indexed data, as the cases RDouble and RInt provide a concrete type for the type parameter A. This means that these cases in the interpreter can produce values of that concrete type. If we did not use indexed data we could only generate values of type A, which the programmer would have to supply to use like in the RPure case.

To finish this implementation we should implement the Monad type class, which would give us mapN and other methods for free. However, this is outside the scope of this case study, which is focused on indexed data. I encourage you to do this yourself if you feel you would benefit from the practice.

Note that indexed data can mix concrete and generic types. Let’s say we add a product method to Random.

enum Random[A] {
  // ...

  def product[B](that: Random[B]): Random[(A, B)] =
    RProduct(this, that)

  case RProduct[A, B](left: Random[A], right: Random[B]) extends Random[(A, B)]
  // .. other cases here
}

The right-hand side of the RProduct case instantiates the type parameter to (A, B), which mixes the concrete tuple type with the generic types A and B

There are a few tricks to using indexed data that are essential in Scala 2, and can sometimes be useful in Scala 3. Take the following translation of the probability monad into Scala 2. (I’ve placed a using directive in this code, so if you paste it into a file and run it with the Scala CLI it will use the latest version of Scala 2.13.)

//> using scala 2.13

sealed trait Random[A] {
  import Random._

  def flatMap[B](f: A => Random[B]): Random[B] =
    RFlatMap(this, f)

  def product[B](that: Random[B]): Random[(A, B)] =
    RProduct(this, that)

  def run(rng: scala.util.Random = scala.util.Random): A =
    this match {
      case RFlatMap(source, f) => f(source.run(rng)).run(rng)
      case RProduct(l, r)      => (l.run(rng), r.run(rng))
      case RPure(value)        => value
      case RDouble             => rng.nextDouble()
      case RInt                => rng.nextInt()
    }

}
object Random {
  final case class RFlatMap[A, B](source: Random[A], f: A => Random[B])
      extends Random[B]
  final case class RProduct[A, B](left: Random[A], right: Random[B])
      extends Random[(A, B)]
  final case class RPure[A](value: A) extends Random[A]
  case object RDouble extends Random[Double]
  case object RInt extends Random[Int]

  def pure[A](value: A): Random[A] = RPure(value)

  // Generate a uniformly distributed random  Double greater
  // than or equal to zero and less than one.
  val double: Random[Double] = RDouble

  // Generate a uniformly distributed random  Int
  val int: Random[Int] = RInt
}

In Scala 2 this generates a lot of type errors like

[error] constructor cannot be instantiated to expected type;
[error]  found   : Random.RProduct[A(in class RProduct),B]
[error]  required: Random[A(in trait Random)]
[error]       case RProduct(l, r)      => (l.run(rng), r.run(rng))
[error]            ^^^^^^^^

To solve this we need to create a nested method with a fresh type parameter in the interpreter, as shown below. With this change Scala 2’s type inference works and it can successfully compile the code.

def run(rng: scala.util.Random = scala.util.Random): A = {
  def loop[A](random: Random[A]): A =
    random match {
      case RFlatMap(source, f)   => loop(f(loop(source)))
      case RProduct(left, right) => (loop(left), loop(right))
      case RPure(value)          => value
      case RDouble               => rng.nextDouble()
      case RInt                  => rng.nextInt()
    }

  loop(this)
}

The other trick is for when we want to use pattern matches that match type tags. This means the form like

case r: RPure[A] => ???

rather than

case RPure(value) => ???

For cases like RProduct it is not clear how to write these pattern matches, as the type parameters A and B for RProduct don’t correspond to the type parameter A on Random. The solution is use lower case names from the type parameters. Concretely, this means we can write

case r: RProduct[a, b] => ???

The type parameters a and b are existential types; we know they exist but we don’t know what concrete type they correspond to. I’ve found this is occasionally necessary in Scala 2, but very rare in Scala 3.

13.4 Conclusions

In this chapter we looked at indexed data and indexed codata. The key idea of indexed types is to encode equality constraints that a type parameter equals some type. With indexed data these constraints are encoded in the data and we discover them when we destructure the data. In this way indexed data is a producer of equalities. With indexed codata these constraints must be shown to hold when methods are called. Hence indexed codata is a consumer of equalities. We also saw that we can go beyond equalities constraints with contextual abstraction, by encoding other types of constraints in given instances.

Indexed types build on phantom types. The earliest reference I’ve found to phantom types is Leijen and Meijer [2000]. Type equalities were added soon afterwards, creating what we now know as generalized algebraic data types or indexed data [Cheney and Hinze 2003; Xi et al. 2003; Sheard and Pasalic 2008]. Most work on generalized algebraic data types is concerned with type inference algorithms (e.g. [Peyton Jones et al. 2006]), which is not so relevant to the working programmer. Lin and Sheard [2010] is not different in this respect, but it does have a particularly clear breakdown of how GADTs are used in the most common case.

Interest in indexed codata is much more recent [Thibodeau et al. 2016], reflecting the general lack of attention that codata has received in programming language research (or at least the parts that I read.) Scala has excellent support for indexed codata but, even so, we can see in Scala a lack of symmetry in the support for indexed data and codata. While indexed data is built into the language, indexed codata is something we must built ourselves from contextual abstractions. This is not necessarily a bad thing, as contextual abstraction allows us to go beyond the simple type equalities of indexed data and codata. Recent research has looked to address this asymmetry. For example, Ostermann and Jabs [2018] considers indexed data and indexed codata as related by transposition of a matrix defining the API and Zhang et al. [2022] develops a system, implemented in Scala, that translates between data and codata.

In a case study we used indexed codata to implement an API protocol: a restriction on the order in which methods can be called. We can view this as an elaboration on the basic algebra or combinator library strategy we have seen in some many different case studies. We can also relate it to work in the object-oriented programming (OOP) community. It is worth doing so to show that these problems bridge programming communities and sometimes disparate communities discover very similar solutions.

In the OOP world a combinator library is called a fluent interface. The same article that introduces the term fluent interface also mentions the need for API protocols: “choose your return type based on what you need to continue fluent action” [Fowler 2005]. Many case studies have explored fluent interfaces (e.g. [Freeman and Pryce 2006; Hawick 2013; Dethlefs and Hawick 2017; Shrestha et al. 2021]) and this style of code is increasing in popularity [Nakamaru et al. 2020]. Encoding an API protocol can be quite involved, so another research direction is the creation of tools to generate code from a protocol definition [Levy 2016; Nakamaru et al. 2017; Gil and Roth 2019; Vuković et al. 2023]. Roth and Gil [2023] translates API protocols back to the functional world, showing a variety of encodings in Standard ML.

The probability monad we developed, which is specialized to sampling data, is only one of many possibilities. Sampling gives us an approximate representation of a distribution. Small discrete distributions can be represented exactly. Erwig and Kollmansberger [2006] show how this can be done, in addition to the sampling approach we used. Kidd [2007] shows how the exact and sampling approaches can be factored into monad transformer stacks. Scibior et al. [2015] uses probability monad as the underlying abstraction on which a variety of different statistical inference algorithms are defined. This is application of the idea of multiple interpretations that we have stressed throughout this book. Scibior et al. [2018] expands on this idea, breaking down inference algorithms into reusable components.

We introduced the probability monad in the context of property based testing [Claessen and Hughes 2000]. Randomly generating test data is not the only approach. Runciman et al. [2008] describes an elegant way of enumerating data. Also see Duregård et al. [2012] for an approach specialized to enumerating algebraic data types. More recently machine learning techniques are being explored. See, for example, Reddy et al. [2020] and Lemieux et al. [2023]. Goldstein et al. [2024] studies how property based testing is used in practice.

I mentioned that the probability monad can be used in generative art. Generative art is, broadly, art that is generated by some algorithmic process. This can include an element of randomness. While there are papers on generative art (e.g. [Boden and Edmonds 2009; Dorin et al. 2012]), and many other resources that discuss it, it’s much more fun to create some yourself. Figure 14 shows an example of generative art. The code is below, and it has many knobs that you can play with to create your own example. Just add the @main annotation to the cycloid method and you can run the code from the Scala CLI. Have fun!