Minimize assumptions, maximize testability
Model the flow of data instead of the logic of the program? That’s crazy! How can you encapsulate anything that way?
My piece on flow-based programming set off a lot of conversations, notably at Slashdot and Reddit. Many of them were hostile, wondering how such a different model could possibly work. Not all, but a lot, were the right kind of productive skepticism.
The reason this makes sense to me isn’t my time programming. My time in markup makes it seem much more sensible. While there are ways to incorporate content by reference (entities) in markup, parsers typically flatten those references into a single tree that is “the document”. Markup processing certainly can combine document transformations with data picked up from some other aspect of the program, but clean transformations are pretty ordinary.
In my broader programming experience, clean transformations are rare. Most of the code that I’ve written is all about creating logical components (often mixed with data) that communicate with other components. Data flows in fragments, and comes and goes from databases, other processing, and whatever structures seem convenient. Testing pieces in isolation is difficult, and worse, not always meaningful. If tests only apply when a program or process is in one state but not another, gaps will grow and test suites will expand perpetually.
The path I see forward – whether in functional programming, flow-based programming, or other attempts at sanity – is making state explicit at every step. This means structuring flows of information so that pure transformations are separated from side effects. Transformations take all the data they need directly as input, and generate only resulting data as output. Other functions (or processes, or whatever name they have in your system) handle all the interactions that have side effects. “Side effects” includes a lot:
This summer, I’ve seen all kinds of programming approaches as I’ve bounced between the Web, XSLT, Erlang, and XML, with visits to many other environments. As I look through the cool new possibilities for interfaces, for scaling up and down, and for dealing with data, I keep seeing two basic patterns repeating: walking trees (of data or document structure), and handling events.
Software grows patient
The biggest change I’ve seen in the last few years of software development isn’t a new language, a new environment, or magical new algorithms. The biggest change is that programmers in many different arenas, working independently, have come to accept waiting.
Part of the joy of computers, a magic that grew and grew as computers and networks got faster and faster, was the confidence that making a change was immediate. Yes, it took time to get a message over a network or for a CPU to execute calls—but things happened.
The event loop has been around for a long time—computers have always had to wait for us slow humans. Transactions have provided a buffer against the possibility of simultaneous changes to the same data, and certainly slowed things down, but the time that buffering took was generally considered a cost, not a feature. Message queues have existed for a long time, but again, seemed bulletproof but expensive.
Over the last few years, these approaches have become more common and better understood. I suspect that there are two main drivers of this change:
As larger scale projects have become more common, the knowledge needed to use these approaches has become more widely available. The average project may be larger scale as well, but these techniques are appearing even in cases where I wouldn’t have thought them necessary. (Of course, I also like playing with Erlang in tiny single-user environments.)
I saw a great talk last week on IndexedDB. It wasn’t the data storage or the asynchronous API that grabbed my attention, but the conversation about promises and ways to make “wait for it” seem like a normal programming idiom. There are a lot of those conversations happening now, about many environments.
Should we make asynchronous seem normal with syntax sugar? Or should we flag it, call attention to it, and make sure programmers remember that their code is waiting?
Spin up Python-friendly services with 0 lines of code
Twisted is a framework for writing, testing, and deploying event-driven clients and servers in Python. In my previous Twisted blog post, we explored an architectural overview of Twisted and examples of simple TCP, UDP, SSL, and HTTP echo servers.
While Twisted makes it easy to build servers in just a few lines of Python, you can actually use Twisted to spin up servers with 0 lines of code!
We can accomplish this with
twistd (pronounced twist-dee), a command line utility that ships with Twisted for deploying your Twisted applications. In addition to providing a standardized deployment interface for common production features like daemonization, logging, and authentication,
twistd can use Twisted’s plugin architecture to run flexible servers for a variety of protocols. Here are some examples:
twistd web --port 8000 --path .
Run an HTTP server on port 8000, serving both static and dynamic content out of
the current working directory. Visit
http://localhost:8000 to see the directory listing: