I’ll be honest, there’s little that sparks my morning quite like a robust programming language debate. It’s invigorating to witness someone dissect a so-called “blub” language used by the masses, those who muddle through their coding days, only momentarily surfacing for StackOverflow guidance.
(Meanwhile, we, the enlightened few, naturally favor languages of superior design. Tools meticulously crafted for the refined hands of expert developers like ourselves.)
Of course, as the author of such a critique, I tread on thin ice. The language I choose to lampoon might just be one you hold dear! Unintentionally, I could be inviting the digital mob into my blog, pitchforks and torches ablaze, all incensed by my audacious pamphlet!
To shield myself from the ensuing flames and to sidestep any potential offense to your possibly delicate sensibilities, I’ve opted to rant about a language of my own invention. A mere strawman, purpose-built for immolation.
I know, it might seem futile. But trust me, by the end of this, we’ll clearly see whose faces (or corporate logos!) are painted on this straw effigy.
Introducing a Novel Language
Devising an entirely new (and deliberately flawed) language solely for a blog post is a considerable undertaking. So, let’s assume it bears a strong resemblance to one we’re both already familiar with. We’ll say its syntax echoes JavaScript – curly braces, semicolons, if, while, and so on. The lingua franca of the programming domain.
JavaScript is chosen not as the target of this critique, but simply because it’s the language most likely to be understood by you, the statistically average reader. Behold:
function thisIsAFunction() { return "It's awesome"; }Being a modern (and intentionally subpar) language, our strawman also embraces first-class functions. This allows for constructs like:
// Return a list containing all of the elements in collection
// that match predicate.
function filter(collection, predicate) {
var result = [];
for (var i = 0; i < collection.length; i++) {
if (predicate(collection[i]))
result.push(collection[i]);
}
return result;
}This exemplifies a higher-order function, a concept that’s both sophisticated and exceptionally useful. You’re likely accustomed to them for collection manipulation, but once you grasp their essence, you’ll find yourself using them extensively.
Perhaps in your testing framework:
describe("An apple", function() {
it("ain't no orange", function() {
expect("Apple").not.toBe("Orange");
});
});Or when parsing data:
tokens.match(Token.LEFT_BRACKET, function(token) {
// Parse a list literal...
tokens.consume(Token.RIGHT_BRACKET);
});You’re set to create impressive, reusable libraries and applications, passing functions around, invoking them, and returning them. A true functapalooza.
What Color Is Your Function, Really?
But hold on. Here’s where our invented language takes a peculiar turn. It introduces one rather odd feature:
1. Every function is defined by its color.
Each function—be it an anonymous callback or a formally named one—is designated as either red or blue. Instead of a single function keyword, we now have two:
blue_function doSomethingAzure() {
// This is a blue function...
}
red_function doSomethingCarnelian() {
// This is a red function...
}Colorless functions are simply not an option in this language. If you intend to define a function, color choice is mandatory. These are the rules, and there are a few more to consider:
2. Function invocation syntax is color-dependent.
Imagine distinct syntaxes for “blue call” and “red call.” Something along the lines of:
doSomethingAzure()blue;
doSomethingCarnelian()red;Function calls must adhere to the color of the function being invoked. Mismatching the call syntax with the function’s color—attempting a red call on a blue function, or vice versa—leads to undesirable outcomes. Picture a long-repressed childhood nightmare, like a clown with serpentine arms lurking under your bed, suddenly materializing from your monitor to extract your vitreous humor.
An irritating rule, wouldn’t you agree? Oh, and there’s one more:
3. Red functions can only be invoked from within other red functions.
Invoking a blue function from within a red one is perfectly acceptable:
red_function doSomethingCarnelian() {
doSomethingAzure()blue;
}However, the reverse is prohibited. Attempting this:
blue_function doSomethingAzure() {
doSomethingCarnelian()red;
}Will likely result in a visit from Spidermouth the Night Clown.
This complicates the use of higher-order functions like our filter() example. We must assign a color to filter() itself, which then dictates the colors of functions permissible as arguments. The immediate solution is to make filter() red. This way, it can accept and invoke both red and blue functions. But this leads us to another thorny aspect of this language:
4. Red function calls are deliberately cumbersome.
The precise definition of “cumbersome” is intentionally vague for now, but envision programmers encountering annoying obstacles each time they invoke a red function. Perhaps it involves excessive verbosity, or restrictions within certain statement types. Maybe red functions are only callable on prime-numbered lines.
The crucial point is that opting for a red function will likely incur the ire of anyone using your API, possibly leading to coffee tampering or worse.
The seemingly obvious workaround is to shun red functions entirely. Make every function blue, and normalcy is restored. All functions share the same color, which is akin to having no color distinction at all, making our language marginally less absurd.
Unfortunately, the language designers—and let’s be honest, aren’t all programming language designers a tad sadistic?—have one final trick up their sleeve:
5. Essential core library functions are red.
Certain platform-intrinsic functions, indispensable and beyond our capacity to reimplement, are exclusively available in red. At this juncture, one might reasonably conclude that the language is actively hostile.
Alt text: A cartoon character with a confused expression, scratching their head, representing the bewilderment of developers facing the complexities of colored functions.
Is Functional Programming the Culprit?
One might speculate that higher-order functions are the root of this predicament. If we were to abandon functional programming’s “frippery” and stick to conventional, first-order functions as intended, we might sidestep this heartache.
If a function exclusively calls blue functions, make it blue. Otherwise, make it red. As long as we avoid functions that accept other functions, we can avoid the complexities of “polymorphism over function color” (“polychromatic”?) and similar absurdities.
However, higher-order functions are just one manifestation of this issue. The problem is pervasive whenever we aim to modularize our programs into reusable functions.
Consider a segment of code that, say, implements Dijkstra’s algorithm on a graph representing social network crushes (I spent too long pondering the representation of such a result—transitive undesirability?).
Later, you need this same code elsewhere. Naturally, you refactor it into a separate function, intending to use it in both the original context and your new code. But What Color should it be? Blue is preferable, but what if it relies on those red-only core library functions?
And what if the new context where you want to use it is blue? Then you’re compelled to make it red. Which might then necessitate turning the functions that call it red. It becomes a color-management nightmare. Color considerations become a constant, unwelcome presence throughout development.
A Colorful Allegory Unveiled
Of course, the notion of “color” here is metaphorical. It’s an allegory, a literary device. “The Sneetches” isn’t really about stars on bellies; it’s about racial prejudice. By now, you might suspect what “color” truly represents. If not, the revelation is at hand:
Red functions are asynchronous functions.
If you’re coding in JavaScript on Node.js, each time you define a function that “returns” a value via a callback, you’ve just created a red function. Revisit the rules and observe the parallels:
- Synchronous functions return values; asynchronous functions do not, instead invoking callbacks.
- Synchronous functions deliver results as return values; asynchronous functions deliver them by invoking a callback you provide.
- Asynchronous functions cannot be called from synchronous ones because the result isn’t immediately available.
- Asynchronous functions, due to callbacks, don’t compose cleanly in expressions, have distinct error-handling mechanisms, and are incompatible with
try/catchand many control flow structures. - Node’s core principle is asynchronous operations throughout its libraries (though they’ve somewhat backtracked, introducing
___Sync()variants for many functions).
“Callback hell” is precisely the frustration of dealing with red functions. The proliferation of 15,118 async libraries (as of 2024) on npm reflects the community’s struggle to manage language-imposed asynchronicity at the library level.
Alt text: A humorous cartoon depicting “callback hell” as a chaotic, fiery landscape with tormented developers reaching out for help, highlighting the difficulties of managing nested callbacks in asynchronous JavaScript.
The Promise of a Brighter Future?
The Node.js community has long recognized the challenges of callbacks and sought solutions. Promises, also known as “futures,” have garnered significant attention.
Promises are essentially enhanced wrappers around callbacks and error handlers. If you consider passing a callback and errorback as a concept, a promise embodies that concept as a first-class object representing an asynchronous operation.
Despite the sophisticated language surrounding promises, they are, in essence, a partial remedy. Promises do marginally simplify asynchronous code and improve composition, slightly mitigating rule #4.
However, the improvement is incremental. It’s akin to choosing between a punch to the gut and a punch to a more sensitive area. Technically less severe, but hardly a cause for celebration.
Promises still don’t integrate with exception handling or standard control flow. Synchronous code still cannot directly invoke a function that returns a promise. (While technically possible, such practice is frowned upon and likely to incur the wrath of future maintainers.)
The fundamental division between synchronous and asynchronous realms persists, along with its associated miseries. Even languages featuring promises or futures bear a striking resemblance to our strawman language in this aspect.
(Yes, this includes Dart, my language of work. Hence my enthusiasm for the team’s exploration of alternative concurrency models.)
Awaiting a True Solution
C# and its programmers might feel smug, given the language’s continuous feature enhancements. C# offers the await keyword for invoking asynchronous functions.
await enables asynchronous calls with the same ease as synchronous ones, requiring only a small keyword addition. await calls can be nested, used in exception handling, and integrated into control flow structures.
Async-await is indeed a step forward, which is why Dart is adopting it. It simplifies writing asynchronous code. But, and there’s always a “but,” the fundamental duality remains. Async functions are easier to write, but they remain asynchronous.
The two “colors” persist. Async-await addresses rule #4, making red functions less cumbersome than blue ones. However, the other rules endure:
- Synchronous functions return values; asynchronous ones return
Task(orFuturein Dart) wrappers. - Synchronous functions are directly called; asynchronous ones require
await. - Invoking an asynchronous function yields a wrapper object when the actual value is needed. Unwrapping requires making the calling function async and using
await. - Rule #4 is largely mitigated by
await. - C#’s core libraries predating async avoided this issue initially.
Async-await represents improvement, preferable to raw callbacks or promises. But it’s self-deception to believe it eliminates all issues. Higher-order functions or code reuse quickly reveal that “color” still permeates the codebase.
Languages Beyond Color?
JavaScript, Dart, C#, and Python share this “color” problem. CoffeeScript and most JS-compiling languages inherit it. Even ClojureScript, despite efforts with core.async, likely faces similar challenges.
Which languages sidestep this issue? Java, surprisingly. Rarely does one commend Java for getting something right, yet here we are. Though, to be fair, Java is now moving towards futures and asynchronous IO, seemingly joining the trend.
C# can also technically avoid “color.” Before async-await and Task, synchronous API calls were the norm. Go, Lua, and Ruby also avoid this problem.
Their common thread?
Threads. More precisely, multiple independent call stacks that can be switched between. These don’t necessarily need to be OS threads. Goroutines in Go, coroutines in Lua, and fibers in Ruby suffice.
(This explains C#’s caveat. Thread usage in C# can bypass the async complexities.)
Reflecting on Operations Past
The core challenge is resuming operations post-completion. A call stack is built, then an IO operation is invoked. For efficiency, OS asynchronous APIs are used. Waiting is not an option, necessitating a return to the language’s event loop for the OS to process.
Upon operation completion, execution must resume. Languages typically track state via the call stack, recording active functions and instruction pointers.
However, asynchronous IO requires unwinding the C call stack—a Catch-22. Fast IO is achievable, but result handling becomes problematic. Languages with core asynchronous IO—or browser event loops for JS—address this differently.
Node.js, with its callback-driven approach, preserves call frames in closures. In:
function makeSundae(callback) {
scoopIceCream(function (iceCream) {
warmUpCaramel(function (caramel) {
callback(pourOnIceCream(iceCream, caramel));
});
});
}Each function expression closes over its context. Parameters like iceCream and caramel are moved from the call stack to the heap. After the outer function returns and the stack is discarded, data persists on the heap.
However, each step must be manually reified. This transformation is known as continuation-passing style (CPS), a 70s language hacker invention for compiler internals—an unusual code representation facilitating compiler optimizations.
CPS was never intended for direct programmer use. Yet, Node.js emerged, and suddenly, we’re all compiler backends. Where did we deviate?
Promises and futures don’t fundamentally alter this. They merely shift function literal creation to .then() calls instead of direct asynchronous function arguments.
Awaiting Generated Solutions
Async-await offers genuine aid. Compiler analysis reveals CPS transformation upon encountering await. This is why await is needed in C#: it signals the compiler to split the function. Code after await is hoisted into a compiler-generated function.
Async-await requires no new .NET runtime support. It compiles to chained closures, already supported. (Intriguingly, closures themselves require no runtime support, compiling to anonymous classes. In C#, closures are indeed poor man’s objects.)
Generators, via yield, offer a similar approach.
(Generators and async-await are arguably isomorphic. I possess code implementing a generator-style game loop solely with async-await.)
Returning to the point: callbacks, promises, async-await, and generators all ultimately smear asynchronous functions into heap-resident closures.
The outermost closure is passed to the runtime. The event loop or IO operation invokes it upon completion, resuming execution. However, everything above it must also return, necessitating call stack unwinding.
This underlies the “red functions can only call red functions” rule. The entire call stack, up to main() or the event handler, must be closurized.
Reified Call Stacks
Threads (green or OS-level) eliminate this need. Threads can be suspended, returning directly to the OS or event loop without unwinding call stacks.
Go exemplifies this beautifully. IO operations park goroutines, resuming others.
Go’s standard library IO appears synchronous—operations execute and return results. But unlike synchronous JavaScript, other Go code executes concurrently during pending operations. Go eliminates the synchronous-asynchronous distinction.
Concurrency in Go is a program modeling choice, not a function-level characteristic. The five “color” rules vanish entirely.
So, next time a new language boasts asynchronous APIs as a concurrency strength, remember the “color” problem. It signals a return to red and blue functions.
