New

Most programming languages have a “new” keyword, which is responsible for indicating to the compiler that memory needs to be allocated and initialized based on a given template (in object-oriented languages, a “class”). Even if languages lack that specific keyword, they have syntax for creating new structures with associated state and behavior. In this chapter, we refer to that syntax as “new”, even if the particular word is not used.

• Builder: A builder is a class, structure, or function whose sole purpose is the construction and configuration of other objects or structures.
• Business Object: A business object is an object which performs functions intimately related to the core purpose of the application.
• Data Object: A data object is a simple object whose only purpose is to store data. These are commonly either maps or arrays, but most languages include additional data structures in their standard library.

One of the easiest mistakes to correct in software is improperly combining the “building” (construction graph) logic with the “business” (application) logic. Separating those within the program reduces the MTC.

The way to accomplish this is simple: in any business object, only data objects may be constructed. All other objects must be created by builders. Or to put it another way: don’t use the “new” keyword (or anything analogous) in any business object. This concept is closely connected to the ideas in Chapter 3, “State.” Keeping the building logic separate from the business logic is a practical way to manage scope, as the building logic becomes responsible for the available scope in any given piece of code.

All program entry points operate as “builders,” and are responsible for the startup procedure of the entire application. For example, the following are the primary builder functions in Java and Python, respectively:

// This is the entrypoint builder function in Java
public static void main(String args[]) {}

# This is the entrypoint builder function in Python
def main():
    pass

if __name__ == "__main__":
    main()

Often, initial builder functions are quite long. They construct the entire application, starting with the most foundational objects and long-running application services, before creating the higher-level, more complex objects. As the initial builder function grows, it is useful to split these functions into smaller parts. When and where that is appropriate is more a matter of taste than pragmatism. Since the initial application wiring requires a lot of context and references, splitting it up will sometimes result in silly functions which take a very high number of arguments (have high “arity”).

The way the internals of the application are connected, the way the application is “wired”, can be imagined as a hierarchy of references, where high-level objects refer to lower ones, which in turn refer to even lower ones, all the way down to the foundational objects. This hierarchy can be modeled as a graph, where the objects are nodes and the dependencies are edges, and is often referred to as the application’s “construction graph”. The “construction graph” is the fundamental responsibility of the builder logic.

Once the initial application construction is complete, program execution is handed over to business objects. A business object is one that performs calculations, interacts with other systems, or is generally concerned with the actual goals of the application and users.

# This is a business object
class SlowGrep:
    def __init__(self, path: str):
        self.path = path

    def find(pattern):
        """
        Look for the pattern in all the files
        """

The separation of builder logic and business logic is incredibly powerful. It keeps the “doing” code obvious, simple, and maintainable. It ensures the “building” code is transparent and easy to modify.

Mixed Building and Business Logic

Let’s look at a bad example of object construction in Python, where the “building” and “doing” logic have not been separated, and discover how such code is modified over time:

class HttpClient:
    def __init__(self, url):
        self.url = url

    def fetch_data(self):
        resp = requests.get(self.url)
        return resp.text

class Parser:
    def __init__(self, url):
        self.http_client = HttpClient(url)

    def parse(self):
        data = self.http_client.fetch_data()
        parsed = {}
        for line in data.splitlines():
            key, _, value = line.partition("=")
            parsed[key] = value

The problem is that the Parser is building the HttpClient. On the surface, this doesn’t look too bad, but as soon as we add a few more requirements the problems become obvious.

Perhaps we need to optionally use a RetryHttpClient that handles retries:

class RetryHttpClient(HttpClient):
    def __init__(self, url, retries):
        super().__init__(url)
        self.retries = retries
        # Configure internal retry logic here

    def fetch_data(self, query):
        resp = requests.get(self.url, params=query)
        return resp.text

class Parser:
    def __init__(self, url, use_retries, retries=0):
        if use_retries:
            self.http_client = RetryHttpClient(url, retries)
        else:
            self.http_client = HttpClient(url)

    def parse(self, query):
        data = self.http_client.fetch_data(query)
        parsed = {}
        for line in data.splitlines():
            key, _, value = line.partition("=")
            parsed[key] = value

Well, this is going downhill fast. Now we have branching construction logic in the Parser. What happens if we need to use a more complex RetryHttpClient, like ComplexRetryHttpClient which takes a retry_strategy to determine how to do retries?

Let’s look at just the __init__ method, as this is getting quite long.

class Parser:
    def __init__(
        self, url, use_retries, retries=0, retry_strategy=None
    ):
        if use_retries:
            if retry_strategy is not None:
                self.http_client = ComplexRetryHttpClient(
                    url, retries, retry_strategy
                )
            else:
                self.http_client = RetryHttpClient(url, retries)
        else:
            self.http_client = HttpClient(url)

Wasn’t the point of the parser just to parse the data? Now it is convoluted! But it is easy to see how we got here: we made Parser responsible for builder logic. So as the construction graph complexity goes up, so does the complexity of the Parser.

An additional downside is that since the logic to construct the http client is inside the Parser, the quickest way to construct an http client is now to simply make a new Parser. This can lead to confusing situations like the following:

class RemoteData:
    def __init__(self, url, use_retries, retries=0, retry_strategy=None):
        p = Parser(url, use_retries, retries=retries, 
                   retry_strategy=retry_strategy)
        self.http_client = p.http_client

But that long parameter list is ugly, and this is Python, so we can hide it:

class RemoteData:
    def __init__(self, *args, **kwargs):
        p = Parser(*args, **kwargs)
        self.http_client = p.http_client

Now we have truly confused anyone trying to make sense of what this program is supposed to do. This is madness.

Separated Building and Business Logic

Let’s fix all of this, starting with the first example.

import requests

class HttpClient:
    def __init__(self, url):
        self.url = url

    def fetch_data(self):
        resp = requests.get(self.url)
        return resp.text

class Parser:
    # Note: Parser really shouldn't use an HttpClient at all.
    # It would be much better to separate the
    # concerns of fetching the data
    # entirely from the concerns of parsing the data
    def __init__(self, http_client):
        self.http_client = http_client

    def parse(self):
        data = self.http_client.fetch_data()
        parsed = {}
        for line in data.splitlines():
            key, _, value = line.partition("=")
            parsed[key] = value

def main(args):
    # Assume we first parse out the url from argv

    client = HttpClient(url)

    parser = Parser(client)

We have moved the building to a builder function main. Now let’s go back and add in all that additional retry code:

def main():
    # Assume we first parse out the parameters from argv

    if use_retries:
        if retry_strategy is not None:
            client = ComplexRetryHttpClient(url, retries, retry_strategy)
        else:
            client = RetryHttpClient(url, retries)
    else:
        client = HttpClient(url)

    parser = Parser(client)

Much better. The parser has not increased in complexity: instead, we have completely isolated the complexity within the builder function, where it can easily be understood, and updated or moved with little effort.

Finally, let’s see what the RemoteData example looks like in the corrected scenario:

class RemoteData:
    def __init__(self, http_client):
        self.http_client = http_client

def main(args):
    # Construct the http client and parser as before

    remote_data = RemoteData(http_client)

This is straightforward and easy to understand.

How The Code Should Look

The mixing of building and doing logic is one of the most pervasive and harmful programming mistakes, but also one of the simplest to fix.

When your application first boots up, it should read something like this:

function bootstrap() {
    let log = new Logger();

    let foundationA = new FoundationA(log);
    let foundationB = new FoundationB(log);

    let serviceA = new ServiceA(foundationA, foundationB);

    let serviceB = new ServiceB(serviceA, foundationA);
    let serviceC = new ServiceC(serviceA, foundationB);

    let application = new Application(log, serviceB, serviceC);
}

With this structure, it is trivial to see exactly what the chain of dependencies is. The sequencing of dependencies in your program is its construction graph. If additional functionality needs to be added, it can be easily accomplished by just wiring it into that construction graph.

Injecting Dependent Behaviors

Part of the benefit of separating “building” from “business” is how it allows us to configure behavior through composition (discussed in chapter 5, “Polymorphism”) and dependency injection. We can take advantage of this when we recognize that some business logic seems to do two unrelated tasks. We can then create two different constructs responsible for the two different tasks, and compose them together using the building logic.

Let us take another look at the SlowGrep Python example from the beginning of this chapter.

# This is a business object
class SlowGrep:
    def __init__(self, path: str):
        self.path = path

    def find(pattern):
        """
        Look for the pattern in all the files
        """

The find function takes a “path” argument, which indicates that the find function handles opening and searching all the files. However, the logic for matching should probably not be coupled to the logic for walking through the files, so let’s pull that out.

import os


class SlowGrep:
    def __init__(self, matcher):
        self.matcher = matcher

    def find(self, path):
        matches = []

        for root, dirs, files in os.walk(path):
            for name in files:
                self.search_file(os.path.join(path, name))

            for name in dirs:
                matches.append(*self.find(os.path.join(path, name)))
        return matches

    def search_file(self, file_path):
        with open(file_path) as f:
            return self.matcher.match(f.read())


class SimpleMatcher:
    def __init__(self, pattern):
        self.pattern = pattern

    def match(self, contents):
        return self.pattern in contents

Note that SimpleMatcher is not constructed by SlowGrep. Instead, SlowGrep expects it to be injected into the class. This is powerful, since we can now create other types of matchers. For example, let’s make one that uses regex:

import re


class RegexMatcher:
    def __init__(self, pattern):
        self.pattern = re.compile(pattern)

    def match(self, contents):
        return self.pattern.search(contents) is not None

This is more flexible. But from a construction graph perspective, there is still a problem: SlowGrep is creating new File objects, and SlowGrep appears to be a business object. We can extract the embedded notion of an iterator from SlowGrep, and put it in a dedicated object. Because we’re doing this in Python, we’ll take advantage of pythonic methods to enable iteration: __iter__, which must return the iterator to use, and __next__, which must return the next element in the sequence we are iterating.

import os


class FileContentsIterator:
    """
    FileIterator takes a string base path and
    iteratively returns the contents of all files,
    including subdirectories within that path
    """
    def __init__(self, path):
        self.path = path

    def __iter__(self):
        return self

    def __next__(self):
        return self._search_dir(self.path)

    def _search_dir(self, path):
        for root, dirs, files in os.walk(self.path):
            for name in files:
                yield from self._search_dir(os.path.join(path, name))

            for name in dirs:
                with open(os.path.join(path, name)) as f:
                    contents = f.read()
                    yield contents

class FileIteratorBuilder:
    def __call__(self, path):
        return FileContentsIterator(path)

class SlowGrep:
    def __init__(self, iterator: FileIterationBuilder, matcher):
        self.iterator = iterator
        self.matcher = matcher

    def find(self, path):
        for contents in self.iterator(path):
            return self.matcher.match(contents)

In the above example, FileContentsIterator acts as a short-lived object builder, responsible for creating File objects which do not persist for the entire execution of the application. These will be discussed in the following section.

Organizing code this way has tradeoffs. When used in simple systems, creating an abstraction like this increases MTC. However, because it decouples both the matching operation and the thing that finds us stuff to match, we could create an iterator that traverses a remote directory structure over SSH, or perhaps walks through website links, or reads SNMP values, or some other operation. Structuring code in this way involves creating a behavioral model. For more information, see “Choosing a Model” in Chapter 6, “If”.

A Note on Short-lived objects

You may be aware of the “Factory Pattern.” While most construction logic should take place when the application is starting up, at the “bootstrap” phase, this won’t always be sufficient. Occasionally shorter-lived objects (transient objects) will be necessary, based on external input and imbued with context (scope, discussed in Chapter 3, “State”) such that they can be passed around to long-lived service objects. Factories exist to create transient objects by combining service objects with runtime input.

The following is an example of how to create a factory (TaskBuilder), which can make transient objects (Task):

class Task:
    def __init__(self, log, api, id):
        self.log = log
        self.api = api
        self.id = id

    def cancel(self):
        self.log.debug(f"Cancelling task {self.id}")
        self.api.delete(self.id)

    async def results(self):
        r = await self.api.get(f"/results/{self.id}")
        self.log.debug(f"Results for task {self.id}\n" + str(r))


class TaskBuilder:
    def __init__(self, log, api):
        self.log = log
        self.api = api

    def build(self, id):
        return Task(self.log, self.api, id)


class TaskManager:
    def __init__(self, api, task_builder):
        self.api = api
        self.task_builder = task_builder

    async def run(self, spec):
        r = await self.api.post(f"/tasks", json=spec)
        return self.task_builder.build(r.id)

In this example, Task is an encapsulation of some executing value, and requires three inputs: log, api, and id. Two of these, log and api, are (probably) “long-lived” service objects, while id is only known at runtime. TaskBuilder acts as a way to combine the long-lived values with the temporary values to create the Task.

Instead of using TaskBuilder, it may seem reasonable to simply have TaskManager create the Task directly. This, however, would violate our rule, as TaskManager is a business object. Let’s examine further the negative impacts of that rule violation.

• Currently, Task only requires two service objects, but it may need more in the future. TaskBuilder allows those service objects to remain decoupled from everything that needs a Task.
• The log and api objects may actually need to be transient objects themselves, with specialized configurations related to Task, such as logging to an additional location, or sending additional headers. Making TaskManager responsible for the creation of additional objects that have nothing to do with it increases its coupling and unnecessarily complicates it.
• It would be difficult to test TaskManager, since it would return a static Task.
• If another Task type was introduced, either TaskManager would need to have a conditional added to figure out which one to create, or another TaskManager, with duplicate logic, would be introduced. We will cover this in more detail in Chapter 6, “If”.

Conclusion

Keeping the new operator separate from business logic feels odd at first, and it requires practice to get acquire the habit of immediately separating concerns. Often, going through the exercise of keeping builder logic and business logic apart seems unnecessary or a waste of time, or perhaps appears to add complexity. However, in your day-to-day programming you have likely noticed that as the codebase grows, somehow it also becomes more and more difficult to change, and more and more difficult to understand. Mixing these concerns of construction and business, failing to follow the new rule, is the culprit well over half the time.

If you encounter a codebase that is difficult to work with, and is in need of refactoring, separating construction and business logic is often the best place to start. Pull out new logic piece by piece, and put that logic in the bootstrap phase, as was done in example with the HTTPClient and Parser objects. The process of organizing and straightening “spaghetti code”, code whose logic is tangled and difficult to reason about, is the first step towards understandable software.