"asynchronous" entries
Variations in event-driven architecture
Find out if mediator or broker topology is right for you.
Editor’s note: this is an advance excerpt from Chapter 2 of the forthcoming Software Architecture Patterns by Mark Richards. This report looks at the patterns that define the basic characteristics and behavior of highly scalable and highly agile applications, and will be made available to download in advance of our Software Architecture Conference happening March 16-19 in Boston.
Available now!
The event-driven architecture pattern is a popular distributed asynchronous architecture pattern used to produce highly scalable applications. It is also highly adaptable and can be used for both small and large, complex applications. The pattern is made up of highly decoupled, single-purpose event processing components that asynchronously receive and process events.
The event-driven architecture pattern consists of two main topologies, the mediator and the broker. The mediator topology is commonly used when you need to orchestrate multiple steps within an event through a central mediator, whereas the broker topology is used when you want to chain events together without the use of a central mediator. Because the architecture characteristics and implementation strategies differ between these two topologies, it is important to understand each one to know which is best suited for your particular situation.
Toward Explicit State
Minimize assumptions, maximize testability
Model the flow of data instead of the logic of the program? That’s crazy! How can you encapsulate anything that way?
My piece on flow-based programming set off a lot of conversations, notably at Slashdot and Reddit. Many of them were hostile, wondering how such a different model could possibly work. Not all, but a lot, were the right kind of productive skepticism.
The reason this makes sense to me isn’t my time programming. My time in markup makes it seem much more sensible. While there are ways to incorporate content by reference (entities) in markup, parsers typically flatten those references into a single tree that is “the document”. Markup processing certainly can combine document transformations with data picked up from some other aspect of the program, but clean transformations are pretty ordinary.
In my broader programming experience, clean transformations are rare. Most of the code that I’ve written is all about creating logical components (often mixed with data) that communicate with other components. Data flows in fragments, and comes and goes from databases, other processing, and whatever structures seem convenient. Testing pieces in isolation is difficult, and worse, not always meaningful. If tests only apply when a program or process is in one state but not another, gaps will grow and test suites will expand perpetually.
The path I see forward – whether in functional programming, flow-based programming, or other attempts at sanity – is making state explicit at every step. This means structuring flows of information so that pure transformations are separated from side effects. Transformations take all the data they need directly as input, and generate only resulting data as output. Other functions (or processes, or whatever name they have in your system) handle all the interactions that have side effects. “Side effects” includes a lot:
Walking Trees and Handling Events
The core of web programming, in JavaScript and beyond
This summer, I’ve seen all kinds of programming approaches as I’ve bounced between the Web, XSLT, Erlang, and XML, with visits to many other environments. As I look through the cool new possibilities for interfaces, for scaling up and down, and for dealing with data, I keep seeing two basic patterns repeating: walking trees (of data or document structure), and handling events.
Walking trees can be annoying, to put it mildy. The Document Object Model (DOM) is famously a headache for JavaScript (and other) developers. There are obvious opportunities for advanced developers to focus on graphs and other more flexible data structures as well. Trees are not necessarily the most efficient way to store information, especially when their content changes regularly.
Learning to Wait: Asynchronous Becomes Normal
Software grows patient
The biggest change I’ve seen in the last few years of software development isn’t a new language, a new environment, or magical new algorithms. The biggest change is that programmers in many different arenas, working independently, have come to accept waiting.
Part of the joy of computers, a magic that grew and grew as computers and networks got faster and faster, was the confidence that making a change was immediate. Yes, it took time to get a message over a network or for a CPU to execute calls—but things happened.
The event loop has been around for a long time—computers have always had to wait for us slow humans. Transactions have provided a buffer against the possibility of simultaneous changes to the same data, and certainly slowed things down, but the time that buffering took was generally considered a cost, not a feature. Message queues have existed for a long time, but again, seemed bulletproof but expensive.
Over the last few years, these approaches have become more common and better understood. I suspect that there are two main drivers of this change:
-
As larger scale projects have become more common, the knowledge needed to use these approaches has become more widely available. The average project may be larger scale as well, but these techniques are appearing even in cases where I wouldn’t have thought them necessary. (Of course, I also like playing with Erlang in tiny single-user environments.)
-
The Web always has latency, and JavaScript practice has evolved to support that. Asynchronous JavaScript may not be everyone’s dream work, but the pattern has evolved from UI events to Ajax to promises, deferreds, and much more. Node is built on these foundations. It’s not just JavaScript, though—I just found Asynchronous Processing with PHP on App Engine.
I saw a great talk last week on IndexedDB. It wasn’t the data storage or the asynchronous API that grabbed my attention, but the conversation about promises and ways to make “wait for it” seem like a normal programming idiom. There are a lot of those conversations happening now, about many environments.
Should we make asynchronous seem normal with syntax sugar? Or should we flag it, call attention to it, and make sure programmers remember that their code is waiting?
Easily invoke common protocols with Twisted
Spin up Python-friendly services with 0 lines of code
Twisted is a framework for writing, testing, and deploying event-driven clients and servers in Python. In my previous Twisted blog post, we explored an architectural overview of Twisted and examples of simple TCP, UDP, SSL, and HTTP echo servers.
While Twisted makes it easy to build servers in just a few lines of Python, you can actually use Twisted to spin up servers with 0 lines of code!
We can accomplish this with twistd (pronounced twist-dee), a command line utility that ships with Twisted for deploying your Twisted applications. In addition to providing a standardized deployment interface for common production features like daemonization, logging, and authentication, twistd can use Twisted’s plugin architecture to run flexible servers for a variety of protocols. Here are some examples:
twistd web --port 8000 --path .
Run an HTTP server on port 8000, serving both static and dynamic content out of
the current working directory. Visit http://localhost:8000 to see the directory listing: