Why Putting JSON in a Database Column is a Bad Idea. Or, Why Would You Ever Put JSON in a Database Column?

When I was a beginner at databases, I was tempted to put JSON in a string column. It’s a weirdly common thing for beginners to relational databases to want to do. A lot of tutorials aimed at beginners suggest doing it for some reason. When people show up on Stack Overflow wanting to know why their code isn’t working, and that code happens to be putting JSON in a relational database column, sometimes someone will show up and tell them to quit doing that, but it’s only like a 50/50 shot, whereas pretty much every other questionable beginner practice will attract hordes of smartasses, some of them barely more than beginners themselves, who will strongly and sometimes stridently censure the beginner for contemplating it. I attribute the popularity of Mongo and other NoSQL databases in part to this instinct: Mongo is basically a database with nothing but string columns that have JSON in them, and some search functionality for JSON in those string columns.

If you’re sometimes tempted to put JSON in a string column, I hope I can explain today why it’s a bad idea and you shouldn’t do it, in a way that will make sense. On the other hand, maybe you’re one of those relational database savants who understood third normal form right away, and you don’t understand why anyone would ever want to put JSON in a string column instead of having a real schema. If so, I hope I can explain to you why someone might want to put JSON in a string column.

Strong Static Typing, Relational Data Modeling, and Escape Hatches

In “Is Weak Typing Strong Enough?”, Steve Yegge describes (buried somewhere in the point he was actually making about programming languages) how teams that he worked with at Amazon were too boxed in by the strong statically typed strict schema the relational data models imposed on them, and resorted to tactics like an untyped name / value system and passing an XML parameter as a string to a CORBA interface. (CORBA was a standard for an early style of distributed services that let you treat objects running on remote servers as if they were in the same address space and could call each others’ methods. It was also language agnostic, so you could run Java on one server and APL on another server and they could use their CORBA implementations to call each others methods. At least, that’s what Wikipedia said; CORBA predates my programming experience by some years.)

The point here is that relational database models are a sort of strong static typing. No matter how much you love strong static typing, sometimes it’s not flexible enough, and you need escape hatches. In 2005 when Steve Yegge wrote his piece, it was XML strings through CORBA. On a programming language level, it’s things like downcasting from Object in Java or using the dynamic keyword in C#. Storing JSON in a string column is another one of these escape hatches. What you get is flexibility. Let’s keep this in mind as we go into our big example, which I hope will demonstrate both why you shouldn’t put JSON in string columns and why you might want to sometimes.

Hugs and Warm Fuzzies: An Example

This example is loosely based on a real system I worked on that had JSON in string columns.

Let’s say we’re working on an MMORPG called Crayon Art Online. Unlike most MMORPGs, which are all about killing things, Crayon Art Online is all about love and friendship and positivity.

Every few hours, one of our servers needs to kick off a job that will read a bunch of player actions from the game’s main database and calculate metrics on them, storing them in a separate metrics database. Since the game is all about doing friendly, happy things, the actions will be things like hugs_given, crayon_drawings_gifted, lunches_bought_for_others, pep_talks, and positive_affirmations. There will be some actions that have extra associated metrics, like hug_warmth and affirmation_positivity and pep_talk_peppiness. There will be some conditions where we just check for a Boolean answer, like sculpting_class_taken?, which just has a true or false answer. We need to calculate all these metrics for all the players across different periods of time: the past 30 days, the past 90 days, the past year, and the entire period the player has existed. Then we need to store them in a database so another job that runs later can read them and distribute prizes to the players that reach certain goals, and messages of encouragement and hope to the others.

Let’s go into some aspects of our data that we’ll have to model. We have a ton of different actions–let’s say there are about 25 total. It’s not a fixed set; there will be new metrics if the developers add new actions to the game, and they might also remove actions, so we would lose the metrics associated with those. That means making a separate table for each metric is doable, but a huge pain. Every time we add a new metric, we have to add a new table. Reading all the metrics for a player means querying 25 tables. A simple manual check for data consistency between the main game’s database and ours requires us to write a select command that includes 25 tables. It seems there must be an easier way.

However, one table also clearly won’t work unless we store all the values as strings. We need a value column to be able to store integers (hugs_given), floats (average_positivity), Booleans (is_wearing_floppy_hat?), and strings (floppy_hat_color). So another option would be one table per value type: int_valued_metrics, real_valued_metrics, boolean_valued_metrics, and string_valued_metrics. But that’s pretty tacky. Your data model is supposed to model the data, like it says in the name. The data itself doesn’t naturally divide into these different types; they’re all just metrics, with the same fundamental features: a name, a time period, a user, and a value. Our database engine needing to store the value as an integer or float or whatever is an implementation detail, not an actual part of the domain, and it feels unnatural to spread the data model for the single metric concept across four tables (and counting, if you add types later—maybe at some point you have tuple-valued metrics, or BLOB-valued). This is better than 25 tables, and it’s better than one table, but it’s still not good.

At this point, things look pretty dire. Relational data modeling has failed us. But wait, you realize. Every programming language nowadays, well except for Java anyway, has a powerful JSON parsing engine built in that can turn a string of JSON into a native map, with type conversions and everything! So, why don’t we just store metrics as a string of JSON per user?

Strings of JSON have lots of attractive qualities. They’re easy to understand; JSON is dead simple, by construction, whereas SQL and relational data models are complicated and hard to understand. Strings of JSON are easy to parse into whatever kind of map or hash thingy your language supports (unless you’re using Java or Scala), and once you’ve done that, you know exactly what you can do with them. Reading from a relational database usually requires some annoying library that binds the data to native data types, or else you can make raw SQL calls, get handed back the most useless, primitive thing your language supports, and do the conversion yourself. JSON is dynamically typed, so you don’t have to worry about finding the correct data type; just store the string, and when you read it back the parser will figure out what types everything should be. JSON is extremely flexible, letting you store anything from a simple glob of key-value pairs to a bramble of arbitrarily nested lists and maps, whereas with relational data models you’re stuck with two dimensional tables.

So, let’s just forget all this relational modeling stuff and make a table that maps a user id to a string of JSON that looks like this:

{
    “hugs_given”: {“30_days”: 5, “90_days”: 10, “1_year”: 85, “all_time”: 345},
    “has_taken_respect_seminar?”: true,
    // ...
}

Now it’s easy to look up a user’s stats. Just select the_json_string from metrics where user_id = 394. Now you have a JSON string with all the metrics. Now, whenever you add a new metric, you just have your code calculate it and stick it in the JSON object before writing it back as a string. No schema changes, no data model cabal to get through, just code sticking things in JSON. If you decide integer-valued stats should also support a count for the last 42 days, just calculate it and stick it in everyone’s JSON.

But here’s a problem: suppose you want to find all the users with more than 26 hugs given in the last 30 days. How do you query this data? SQL can’t read JSON, so you can’t do select * from metrics where the_json_string.hugs_given.30_days > 26. That JSON type is opaque to the relational database. (Unless you’re using PostgreSQL. We’ll pretend you’re not. The real project I worked on used MySQL, but I believe Oracle and MS SQL Server also lack a JSON type.) So how do you write this query? You can write your own terrible ad hoc JSON parser using regexes and LIKE right there in your database console, or you can write a script in some language like Python or Ruby that reads in the data, parses the JSON string, and does the check for you. Writing this script can also get surprisingly tricky. If the entire table doesn’t fit in memory, you can’t just read all the rows and loop over them. The simple approach would be to get the maximum id of the table and then select each row by id, skipping over misses, until you’re done. But this incurs a disk read on each iteration of the loop, plus possibly a network call if you’re not allowed to run the script right on the database server, so it’ll be agonizingly slow. So you’ll probably read in batches. Now you have to mess with the logic around batch sizes and offsets. It probably won’t take more than an hour or so to write this script. Still, a simple select is something you can dash off about as fast as you can type it in. And even if you write the script generically enough that you now have a general purpose script for querying based on values in the JSON string, you still don’t have all the tools available in one place. If you’re working in the database console and suddenly realize you need to query some data in the JSON string to proceed, you have to stop, go to another window, and run the script. There’s extra friction if you need to feed that data into an SQL query at the database console to move on. There’s this gaping hole where you can’t use SQL to find data that’s stored in your SQL database.

Putting data in strings also kills the validation benefits that a statically typed data model gives you. If someone accidentally writes {"hugs_given": "fish"} to the database, the database doesn’t know that that’s not how you measure hugs. If someone accidentally writes {"hugs_given": {"30_day": 5, "30_day": 10} to the database, the database doesn’t know or care that you repeated a key.

How to Proceed?

So now we know both the strengths and weaknesses of storing JSON in string columns. It’s flexible, easy to change, easy to write, easy to understand, easy to work with in code. On the other hand, it’s meaningless to the SQL database (unless you’re fortunate enough to be working in Postgres). It’s just this string of characters that might as well be the complete text of the classic novel Lady Chatterley’s Lover by DH Lawrence. You can’t query it from SQL in any meaningful way. You can’t put indexes on it. (Well, you could put an index, but it would be massive and not very helpful.) You lose the data validation benefits that a relational schema gives you. What do we do?

It’s possible that you feel you can live with the limitations of JSON strings in database columns. Maybe you can. I have a feeling they’ll eventually get in your way, though. They certainly got in mine.

If you have the option of switching data stores, you could just use Postgres, or some JSON-oriented document store like MongoDB, although I have reservations about that in general. In Crayon Art Online, as in the real application that inspired it, metrics are calculated by a separate process and stored in a different database than the main application, so even if your main application is on MySQL, it might be doable to switch your metrics application to Postgres. (It wasn’t for me, for a variety of boring non-technical reasons.)

But suppose you’re convinced that storing an opaque string of JSON won’t work for you, and you don’t have the option of switching data stores. You’re left with trying to find a better data model to fit this into a relational schema. In this particular case, where the nesting isn’t that deep, it’s not actually that hard to come up with an acceptable solution that’s better than JSON string in a column.

To start with, I’d represent each individual calculated metric as a row with a user id, type tag (hugs_given, positivity_quotient, etc.), a date range tag (30_day, 90_day, etc.), a value, and a created_at date to record when we calculated it.

In my experience, rolling date ranges are a lot simpler than actual dates, so even though it might be tempting to get rid of the string keys for 30_day, 60_day etc. and come up with some way of storing the date range based on actual dates, it could also complicate things a lot. If you need to know the actual date range this covers, you can count backwards from the created_at. If you expect to need this a lot, it might make sense to store the date keys as integers that represent the number of days covered by this metric, so we’d have a column days_covered, and then we can find the start date of this period by subtracting that from the created_at.

The main hurdle with representing this in a schema is the fact that the value type can be an integer, real number, Boolean, string, or possibly other things. Making separate tables for each value type is tacky in my opinion, but you could go this route if you want. Then you could use some kind of wrapper at the data access layer to hide the four classes from your code and just present one consistent interface for them. An even more tacky variant would be to make your value column store a foreign key into another table, and make four value tables that hold your values. So you’d have the five tables metric, int_value, boolean_value, real_value, and string_value, and metric would have a type column that tells it which table to look for its values in, and a value_id column that points into one of the value tables. Not only does this have the same disadvantages as the four metrics tables, it’s also weird and complicated to understand. Another option: store the value, and only the value, as a string, and implement some kind of conversion to the correct type at the data access layer. You can still run queries on the numeric values by casting the string to a number, so you’re not as out of luck as you were with opaque JSON strings. None of these are great solutions. This was the original hurdle that led us to putting JSON in a text column, and I don’t know of any really good way out of it. But I do think there are better ways than JSON in text columns.

This is not a perfect, beautiful schema. It’s not a work of art. It’s not going to get written up in textbooks or research papers as amazing. But it is a real relational database schema which will work better for you in the long run than shoving a string of JSON into a column. (Or a string of CSV, or XML, or YAML, or any other kind of structured text format that isn’t supported natively by the database type system.) You can do queries on it, you can analyze it, you can put indexes on it, you can implement certain data consistency standards on it (e.g. a unique key on type, days_covered, and created_at that prevents you from accidentally inserting the same start for the same date range more than once). There are also opportunities to make things better as you discover problems. With text in a column, there’s not a lot else you can do with it when you discover problems.

This schema is also more flexible and manageable than separate tables for each metric. If you need to add a new stat, just start calculating it and add a new tag for it. If you want to support a new period of time, just make a tag for it and start storing it. No need for schema changes.

The Takeaway

If you came into this as an expert who loves relational databases and strongly typed schemas, I hope you can understand a little more the kind of problems that seem too hard to fit into the strictures of such a schema, driving beginners to want to put JSON in string columns.

If you came into this as someone who’s occasionally—or more than occasionally—thought the answer to all your problems was shoving JSON in a text column, I hope you now understand why that’s not really the answer to your problems. Or you learned that you need to switch to another data store that supports shoving JSON into text columns better, and now every company you work at, you’ll lobby to switch to Postgres or Mongo. Someday that might even lead you to a job as a MongoDB ambassador who gets to travel around to conferences telling everyone why shoving JSON in text columns is terrible, and they should instead shove JSON in MongoDB to solve all their problems. Just think; I could be starting careers here today.

Advertisements

Foxes and Hedgehogs

When I first started programming, which wasn’t actually that long ago, but it feels like ages because of the constant churn, languages were like drunks in bars: they all had a philosophy. Some of them made a lot of sense, like Ruby’s “Principle of Least Surprise”. Some of them were incoherent, like Perl’s “TIMTOWTDI” and “GALFKGAFCVHAJRTOAIADGDLFJKC”. Many were encapsulated in nice little slogans that people could yell at each other in forums. Others were implicit, like Java’s philosophy, “Make Insanely Complicated Libraries So Companies Have To Hire Exorbitantly Priced Consultants To Teach Their Programmers To Accomplish Basic Tasks In Java”. But whether the philosophy made sense or didn’t, whether the philosophy was implicit or explicit, the languages all had one.

The philosophies served a lot of social purposes, but in most cases they did begin as a motto or design principle that the language creators kept in mind when designing the language. Recently, the languages I’ve seen becoming popular, like Kotlin and Swift, don’t seem to have philosophies. They have purposes: Swift’s purpose is freeing iOS and macOS programmers from having to manually decrement reference counts, and Kotlin’s purpose is to integrate with Jetbrains IDEs. But they don’t seem to have language design philosophies the way older languages do.

A lack of a language design philosophy is sort of a language design philosophy. It means the creators don’t have an overriding vision for the language. But without an overriding vision, how do they decide which features get in and which don’t? I don’t know a ton about either Kotlin or Swift, but one thing always stuck out to me in tutorials on them: the tutorials read a lot like lists of features from other languages. “Swift has for-each loops. Swift has lambdas. Swift has classes. Swift has limited type inference.” They would explain how the particular implementation worked, but it was usually substantially similar to other well-known languages. Kotlin and Swift are not random assemblages of features the way languages like C++ and Perl 5 and PHP are. They have an aesthetic, which I’d call roughly “functional-lite”. They’ve settled on a Python-esque profile, largely imperative with object-oriented and functional features worked in around the edges, and added some Java-style static typing. I divined that they choose features based on what they think working programmers will find useful.

Of course, no popular language aimed at working programmers purposely adds features that are highly likely to be useless. But some languages are built according to a strong, overriding vision of what set of abstractions will ultimately prove useful to working programmers. Others are more pragmatic; they add whatever features have been proven, through long use in other languages or through long experience without the feature, to be practical and useful for working programmers. Older languages made this into a philosophy itself: Perl’s “There Is More Than One Way To Do It” was expressing a belief that there should be multiple syntaxes for the same abstraction, presumably because each one would feel natural in different situations. (I never worked with Perl, but the concept does make sense—the same abstraction can manifest concretely in different ways, each of which feel appropriate in different situations.) This distinction is sort of a meta philosophy on top of the various philosophies (explicit and implicit) of different languages. Swift and Kotlin are in the latter camp, but since they’re backed by large corporations and have special niches where their dominance is uncontested, they didn’t need to create marketing slogans that explicitly state it the way Perl did. They didn’t need to differentiate themselves from other languages with marketing; they differentiated themselves by being the language of Apple / Jetbrains and by being the language of iOS development / extreme IDE integration.

Once I realized this divide existed, it reminded me of foxes vs. hedgehogs. The idea comes from a snippet of Ancient Greek poetry attributed to Archilochus: “The fox knows many things, but the hedgehog knows one big thing.” To wit, the fox stays alive with many tricks to keep ahead of its enemies, whereas the hedgehog has one big trick, stabbing its enemies on its spines. (We’re excluding the possibility of a hedgehog that’s the fastest thing alive.) I first heard about it in politics. Some politicians dabble in many issues; they might have a few flagship issues, but they’re conversant with almost all of them, and they’re quick to educate themselves about new issues. Other politicians have one big issue that they care about more than anything, and are less concerned with other issues. Isaiah Berlin applied it to writers in his essay The Fox and the Hedgehog: some writers, like Plato, Dante, and Nietzsche have one big idea that colors all their work, while others, like Shakespeare, Joyce, and Goethe, write about a wide range of ideas and experiences. It’s the same with programming languages. Some languages are fox languages: they might have a few flagship features, but they’ll include various features from all over the place, and while they’re not necessarily garbage heaps of trendy features, they’re more open to borrowing. Some languages are hedgehog languages: they’re built around a singular vision. The vision might be towards a stance on a big philosophical issue. It might be a vision of a language built around a big abstraction. It might be a vision of a language perfectly suited for a single restricted domain. For a hedgehog language, everything that goes into the language serves its one singular vision.

Why is this distinction useful? Because new languages (and libraries, frameworks, platforms, etc., which can be judged much like programming languages) appear constantly, and they’ll all tell you that they can solve all your problems. Evaluating whether the tool’s meta-philosophy matches your goals is a good first step to finding out if that’s true. It’s not about one or the other being bad or wrong. Both have their uses.

If you think hedgehog languages are useless, think about this. Nobody likes writing multithreaded code. Well, okay, some people like writing it, but nobody likes debugging the insane synchronization issues that crop up later. Nobody likes spending a bunch of time crafting the perfect multithreaded architecture only to discover it has a fundamental flaw that makes all your code actually just really expensive sequential code. So a bunch of hedgehogs who had a vision of a better way to do this concurrency thing went off and created abstractions like software transactional memory, actors, communicating sequential processes, single-threaded event-driven programming, and others, and then a bunch more hedgehogs made languages like Clojure, Erlang, Scala, Go, and Node that baked in support for them as part of their visions. Another big victory for hedgehogs: OOP. Hedgehog languages like Smalltalk and Java went all in on OOP and convinced a generation of programmers that writing libraries in C was painful and bad. Whether you think OOP was a good idea or not (I’m starting to have my doubts nowadays), it was a huge propaganda victory for hedgehogs.

I hope it takes less convincing to see that fox languages have their uses. Their benefits are more obvious: programmers who’ve used the same features in other languages can jump right in and understand them without learning new abstractions. Knowledge about how to solve problems in other languages carries over without any trouble. The thing is, these seem to be cyclic. For a while, everyone just uses the fox languages and appreciates all their homey conformity and routine. Want to write a website? Here’s a fox language. Want to write a mobile app? Here’s another fox language that’s basically the same, except maybe you can leave parentheses off no-arg function calls. Want to program an Arduino? Here’s yet another fox language that’s basically the same as the first two, except maybe it has an easier dictionary syntax. Then some new problem appears. Reusable software. Distributed computation. Big data. Machine learning. Whatever. The old fox languages are missing something that makes this new problem easy to solve. So a round of hedgehog languages appears, built around a single grand vision of how to solve this new problem. Sometimes people adopt one of these hedgehog languages at large scale. Sometimes it survives multiple cycles by turning into a fox language. Other times, the new hedgehog languages are too inaccessible or don’t have enough marketing or whatever, but after a while a new round of fox languages appears that brings their features to the masses.

This is where I see Swift and Kotlin fitting in. Everyone was aware that Java 6 was making things harder. Everyone knew Java 6 didn’t have the right abstractions for modern, quick-churn web and mobile code. A round of hedgehog languages came out to address this—-Scala, Clojure, Go. They’ve gotten some adoption. Go is foxy enough, and backed by a huge enough company, that it might live to become a fox language. But the barrier to entry was high enough, and the lack of marketing profound enough, that most of these languages haven’t caught on in a big way. Not like Java, or Python, or PHP have. So now we’re having a round of fox languages—-Swift, Kotlin, Javascript after ES2016—-that consolidate features from all of them. We’re having languages like Java go foxy and add features from these languages to new versions.

So at this point, you might be saying “That’s great, but I’m a programmer who writes boring business apps. How do I figure out what I should be using?” I’m glad you asked! The distinction between fox languages and hedgehog languages is most important of all for you. Boring business apps have a particular set of constraints that are different from most other kinds of program. The code will be changed in a million places in a million small ways over its lifetime. A wide array of programmers should be able to understand the code, and legibility, documentation, and easy modification are the most important characteristics. There is almost never a good reason to use hedgehog languages for boring business applications. Hedgehog languages are based on a grand vision of how computation should be expressed. Boring business applications are not about grand visions of how computation should be expressed. Boring business applications are best expressed in the abstraction of structured programming, which is now such a fundamental part of programming languages that it doesn’t even matter whether a language is for foxes or hedgehogs, it does support structured programming. So, if you write boring business applications and you’re evaluating a language (or framework, database, etc.), you should look for signs of foxiness. Are the features widely supported in other, similar, existing tools? Do the creators stress practicality, usability, reliability? Are the overarching abstractions familiar, or at least widely available in other foxy tools? If you’re seeing a bunch of unfamiliar features, or familiar features mixed in unfamiliar ways; if the creators are talking about vision, about changing the way things are and blowing up the status quo and revolutions and paradigm shifts; or if the tool is based around abstractions that you’ve never seen before or only exist in very obscure tools, then you’ve got a hedgehog tool. It’s much harder to evaluate whether a hedgehog language is suited to what you need. Hedgehogs have to be carefully matched to a use case and a set of problems, because hedgehogs are usually created in response to a specific use case and set of problems.

I’ll end by listing off a few languages and evaluating whether I consider them fox or hedgehog, as they currently stand.

C: At the time it was made, very much a fox language. Now it’s something of a hedgehog language, in that its one big use case is programming in low-level, resource constrained, or high performance contexts. You probably wouldn’t choose to write your startup’s killer new social network in C.

C++: Definitely a fox language. C++ keeps on adopting new features like lambdas that proved popular in other languages. It’s mostly seen nowadays in similar contexts as C, but unlike C, it keeps piling on new features and new abstractions in response to programmer demand.

Java: Java started as something of a hedgehog language. It’s become less of one over time, but still looks spiny enough to me. It was object oriented programming for the masses, C++ without the rough edges, Smalltalk without the flexibility, and its adherence to object orientation still seems a bit militant. It’s gotten new features like generics and lambdas, but it’s always added them by putting a thin layer of syntax on top of an existing part of the language: generics just tell to compiler to automatically insert a cast for runtime, and lambdas are just syntactic sugar for anonymous classes implementing interfaces with one method.

Python: Python is a fox language with a few hedgehoggy principles, like “There should be one and only one obvious way to do it”, which leads to enforced whitespace conventions. Over time, it added features like classes, map and reduce, list comprehensions, iterators, generators, and async / await in response to other languages. It also made it relatively easy to call code written in C, which is the foxiest trick of all.

Python is also a good case study on communities vs. languages themselves. The Python language is a fox, but the community behaves like a bunch of hedgehogs. Python seems to usually resist adding any feature until people have been asking for it for three or four releases, and then someone will make a PEP and it’ll get added.

Ruby and Perl: These two are a lot like Python, but with even fewer overt hedgehog tendencies. They both borrowed things from bash, which is a very foxy language in that you only use when you have no other way to accomplish something.

Scala and Clojure: These two are the opposite of Python, Ruby, and Perl, which were foxy with a few hedghoggy traits. Scala and Clojure are hardcore hedgehogs with a few foxy traits, the biggest one being that they run on the JVM and can interoperate with Java code. Scala has a grand vision of mixing Java-style OOP with Haskell-style functional programming. Clojure has a majestic ambition to be a Lisp dialect for the modern era, mixing a solid concurrency story and good support for hash maps with the functional features of Haskell and the homoiconic power that Lisp is known for. Both of them have high learning curves due to lots of unfamiliar abstractions and unusual features, like Scala’s implicits and Clojure’s menagerie of types for managing state under shared memory concurrency.

Lisp in general is a hedgehog language. It has a grand vision of all programming as lists of lists that directly represent an abstract syntax tree. Common Lisp is a Lisp that tried to be foxy with stuff like rplaca and the progn macro. Scheme is a Lisp that skewed further towards the hedgehog. Clojure is somewhere in the middle; it runs on the JVM and adds features like hash maps, sets, and regex literals that are foxy compared to Scheme, but also encourages pure functional programming and immutability to an uncomfortable hedgehoggy extent. Its concurrency stuff is pure hedgehog—yes, even core.async. We’ll get into this in the next section.

Go: I’ve only used Go a little bit, but to my eyes it’s a language that lures you in by appearing to be a fox, but then when you go to pet it, surprise! You’ve got hedgehog spines stuck in your hands! It’s a hedgehog that knows how to play tricks like a fox.

There are two aspects to Go’s inner hedgehog. One is its concurrency stuff, with goroutines. Yes, I know it’s based on communicating sequential processes, from the 70’s. The thing is that there are at least two whole generations of programmers who’ve grown up never hearing about communicating sequential processes and never using a language that supported it. There’s the 1990s era of big enterprise Java developers, and there’s the current era of hipster startup Node hackers, plus a few intervening generations, all united in their never using a language that supported CSP as a first class abstraction. Since Clojure’s core.async is a ripoff of goroutines (as admitted by Rich Hickey himself), it’s also a hedgehog trait.

The other aspect is Go’s slavish adherence to C’s precedent. It has pointers. It has printf and scanf. It doesn’t have generics, and has repeatedly refused requests to add them, and justified this decision by appeals to Go’s design philosophy of simplicity. Whenever a widely requested feature is refused because of a design philosophy, that’s a sure sign of a hedgehog. I’m not saying whether that’s right or wrong, but it is a fact. C itself was foxy, but Go’s insistence on copying it beyond reason is total hedgehog.

My Biases

To disclose my own biases, I actually like hedgehogs, except when they go wrong, which they can, terribly, much more easily than foxes. Go’s hedgehog nature was what induced me to try it over something more foxy like Node or Kotlin. Clojure’s hedgehog nature induced me to try it over something more foxy like Scala. (Yes, Scala is also a hedgehog, but it looked deceptively like a fox from far away.)

Hedgehog languages are usually the ones that win over the people, but don’t get any traction in industry, and forums are filled with programmers wondering why no one can see how great this language is and what a killer secret weapon it would be for some team.

Even though I like hedgehogs, when the chips are down and the title is on the line, I always go back to foxes. If I have to program something and I have no idea what I’m doing, it’s back to Python. I don’t reach for Clojure or Go, because with hedgehog languages, you never know when you’ll run into some problem that, due to its very nature, doesn’t translate well into that language’s idioms and abstractions. If it turns out that a bunch of hideous index manipulation is the only way to solve the problem, I’d rather take my chances with Python than write weird Clojure code full of atoms and swap! and reset!.

Java is a hedgehog language. And a lot of pain in modern software has happened because people used Java somewhere where its insistence on object-oriented programming, combined with the militant adherence to object-oriented design principles that its programmers have developed, didn’t do a good job representing the problem domain. I work on a Java codebase at my job, and every day, I run into things that Java makes harder by being such a freaking OOP hedgehog. Testing, for example. You can write testable code in Java, but features like private access and lack of standalone functions are hostile to it, not to mention all the ways Java developers’ culture has evolved to make it harder to write testable code. This has led to tons of cultural conventions and weird libraries to make writing testable code possible, but still not easy. If the chips were down and the title was on the line, I would never turn to Java. It’s fairly likely if I pick Clojure that I’ll have to do some sequence manipulation that’s hard to translate into point-free applications of map, reduce, and filter, but it’s just as likely if I pick Java that I’ll end up wanting arbitrarily nested maps, or pairs, and have to run off and write new classes to represent them. And then I might want a 3-tuple. Guess what? Another new class!

On the other hand, when you have to explore a gigantic monolithic codebase written by 50 programmers over five years, Java’s not so bad. It forced the other programmers to explicitly write out everything, explicitly represent everything with classes and statically typed variables, and even tag which methods you need to worry about seeing again outside of that class. It’s all about knowing when to pick the right tool. Evaluating foxiness vs. hedgehoggosity is one way to get a feel for a tool. Then you can decide whether that particular hedgehog’s spines will solve your problem, or if you need the cunning of a fox to get through.

Web Programming: A Prophecy

Over the course of the 20th and 21st Centuries, a lot of jobs that used to require human labor have been turned over to robots. Factory jobs are going to robots. Mining jobs are going to robots. Fighter jet and spy plane pilot jobs are going to robots. Retail jobs are going to robots, with self checkouts and restaurants like Eatsa.

I used to believe, smugly, that programming was too complex and difficult to ever be taken over by robots. When I read an article by some business commentator that claimed programming jobs were going to be this generation’s equivalent of the high-wage factory jobs of the 1950s and 1960s, I scoffed. Until I actually did some of what passes for programming in industry.

It’s pretty much all been lamented before: modern programming in a corporate environment is mostly writing little bits of glue code to combine a bunch of libraries written by some distant genius. Mike Taylor’s version was the first version of this lament that I saw, back when I was still in school and under the impression that industry programming was at least a little bit like personal or school programming. But I’m not here to lament. I’m here to make a prophecy.

In my current job, I am a full-stack web developer. I write the server-side code. I write the client-side code. I write the HTML and CSS. I create the database schema and write the SQL scripts to update the database. Is it complex and difficult? Kind of. But it’s complex and difficult because I have to have passing familiarity with Java, Javascript, HTML, CSS, and SQL, and work through all the amazing ways they don’t fit together well. My part in the whole mess is to drop in fairly simple, monotonous bits of boilerplate code to grab things from the database and insert them into an HTML document. Then I manipulate another mess of frameworks and libraries to show it all nice and pretty and make dynamic behavior on the page. Between the server-side framework, the Javascript framework, Bootstrap, and various other libraries, about 90% of the work is done by magical frameworks that just use little spludges of repetitive code I write to get that last 10% of the way towards becoming a web app.

I find it extremely hard to believe that there’s no way to automate that last 10%. There has to be a tool that makes it possible for someone to drop in those repetitive little globs of boilerplate just the way they like, and get a web application out of it.

As a full-stack web developer, I also offer some advice on design, UX, and business needs. But I’m not a designer, a UX pundit, or an expert on the business needs. My degree is in computer science. My expertise is programming. But there’s damn little programming in the code I write, and nothing I’d call computer science. The main hurdle is the delicate little interconnections between these rickety, rigid frameworks, the brittle wooden bridges I have to cross by writing repetitive boilerplate. But, per hypothesis, there has to be a way to create a tool that automates that repetitive boilerplate. Without that, there’s no programming left in creating the sort of custom line of business app that I work on. You could get together a designer, a UX pundit, and an expert on the needs of the business, and have them use this tool to create whatever web app they want. There are actually already tools that are close to this. Squarespace. Drupal and Joomla. WordPress is a little closer to full-stack web development, but I see job ads for “Wordpress programmers” rather than “PHP programmers” or “Front end programmers”, which suggests to me that WordPress is a specialized tool. As I was writing this I discovered WebML, which already sounds pretty close to what I was envisioning.

Once I realized that, I saw how right Mr. or Ms. Business Commentator whose name I’ve forgotten was in his or her comparison of software developer jobs to factory jobs. Right now, tools like Squarespace and Joomla still have certain limits, and there’s cachet in having a custom web app, so everyone and their dog needs a full-stack web developer to make one for them. Trucking companies need web apps; banks need web apps; government agencies need web apps; churches need web apps; even the Amish probably have a full-stack web developer on their payroll to make an Amish web app for them. (Job perks include hot apple Brown Betty in the kitchen, bonuses paid in handmade furniture, and a company carriage for the commute.) But once this magical tool whose future existence I’ve postulated comes to pass, none of these people will need full-stack web developers. Except the Amish; they need someone who’s okay being in a building with electricity every day. But everyone else will just have business experts make the web app. This might be the product manager, or the head of the department that’s going to use the web app, or the lady down the hall who knows Excel and VBA really well and consequently always gets stuck doing everyone’s spreadsheets. But there’s no reason to bring a programmer into the company and pay them a bunch of money and have them learn the business requirements when you can just have someone who already knows the business requirements use the tool to build the web app.

This doesn’t mean that all programming jobs will suddenly disappear. There will still be programmers. You can’t have a business person build something with a prepackaged tool when you don’t have any business requirements and no one’s ever built anything like it before. Google-scale stuff isn’t going anywhere. Google-level programmers aren’t going anywhere. But people like this guy are going to lose their jobs, or they’ll be slotted in at a lower pay grade to build web apps with the new tools. Even people who aren’t like that guy, people who don’t obstinately hold it as a personal tenet that no one should know more advanced technical subjects like Big O and data structures and we should all just concentrate on understanding business requirements and spelling Dijkstra’s name right, will lose their jobs in droves. They’ll lose their jobs because they’ll have spent careers working on simple little full-stack web apps built on existing frameworks instead of on Google-scale stuff, and suddenly businesses won’t need that anymore, or at least they’ll be paying a lot less for it.

So how long do we, the generic full-stack web developers working on dinky little web apps, have before the apocalypse? My guess is no more than ten years. The tools are already getting there, and they keep advancing. With single-page apps and faster connections, users are already being trained to expect more lag in web apps, so the kind of performance tuning that an experienced developer does will become less relevant. And businesses would love to be able to fire a bunch of expensive programmers and squeeze a little more work out of the business people. They’d love to be able to stop stocking soda and Terra chips for us arrogant, overpriced keyboard jockeys.

A few questions

At this point, you might be thinking my prophecy is wrong and the era of anyone who knows Javascript and one server-side framework being able to find jobs anywhere at relatively high salaries will last forever. And you might be right. The odds are against it, but you might be right.

However, I’m going to try and preemptively address a few questions I can see people having about this prophecy.

Won’t programmers just switch to programming the tools?

No. Well, yes, some of them will. But there aren’t nearly enough jobs working on the tools for everyone who now works as a generic full-stack web dev to get one. The world only needs two or three competing tools in each space, but even if a dozen more manage to scrape by, the scale won’t match the number of people currently working on dinky custom web apps alone for non-tech companies. Think about it: have you ever written a framework? And if you have, did anyone use it? Even though it sometimes feels like there are new frameworks for Javascript coming out on a daily basis, only about three or four actually get widely used, and most framework creators don’t get paid to work on them. The ones that do get used, the ones whose creators do get paid to work on them, are the ones like Angular and React that are backed by large tech companies.

Writing the tools will be the kind of Google-scale programming that will hang on after the Twilight of the Generic Full-Stack Web Devs. But for every former generic full-stack web dev who switches to working on tools, there will be at least ten who get shut out because of significantly lower demand for programmers, and probably another five who switch to using the new tools to do their own jobs at a significant pay cut just to stay employed. Companies in all industries needing to have their own custom web app and hiring a generic full-stack web dev to glue together some frameworks for them has created a supply of generic full-stack web devs that won’t be nearly as much in demand once business people can make the dinky custom business app.

Isn’t programming too complex and difficult to automate like you describe?

I hope if you’ve done the kind of generic full-stack web development I’m talking about here, you know that’s not true. The frameworks already do most of the work. Disorganization and the dismal state of web standards is the only thing holding us back from a single isomorphic client/server framework that lets us build monolithic web apps in a single language the way we used to build desktop apps. And with Node.js, client-side MVC frameworks, and Bootstrap, we keep getting closer to that one framework to find them all and in the DOM bind them.

There are other kinds of programming that I don’t foresee being automated so easily: hardware programming, big data, machine learning, cryptography, and of course the tools such as compilers, IDEs, frameworks, servers, and databases. Programming those things is complex and difficult, and programmers working on those things will continue to have lucrative careers. This prophecy will largely affect people not working in the tech industry, but working on web apps (and possibly mobile apps) in other industries.

Didn’t this already fail?

There have been various attempts to create tools that business people could use to make custom applications, e.g. CASE tools. Many failed. But one, COBOL, turned out to be one of the most successful programming languages of all time.

It’s possible that efforts to create the super tool I’m prophesying here will also fail. But I think modern day has one big thing that makes such a tool more likely to thrive: Google. Older tools were expensive proprietary things, and you had to read a gigantic manual to figure out how to use them. But nowadays, if you need help, you can Google for answers, or you can ask a question on a Stack Exchange site. And there will probably be blogs and tutorials and help guides for the tools, and maybe even public online documentation by the manufacturers, the way Oracle does with Java.

If you’re a busy business expert trying to make a web app, this makes a huge difference: instead of having to spend hours reading through a giant manual and days figuring out the best way to use what your tool gives you, you can just Google for an answer and copy it verbatim, like programmers do.

I’m a generic full-stack web developer and I believe in your prophecy. What should I do?

As a prophet of doom, I sadly don’t have any constructive suggestions for you, other than to consider where your career might go after this doom comes to pass.

If you enjoy the technical side of things, you can try to become an expert in one of those other areas of programming that won’t be wiped out in this apocalypse. If you find cryptography or machine learning too imposing, go for stuff like dev ops and database adminstration. Or if you’re more of a front end person and you like design or UX, you could head in that direction.

Alternately, if you like business requirements and management, like this guy, go full speed in that direction. I don’t, so I won’t be joining you.

What are you planning to do once the prophecy comes to pass?

Once my career in generic full-stack web development dries up, I’m going to take a few months to backpack around Europe and find myself. In a small Italian village on the Amalfi coast, I will rescue a stunningly beautiful woman from three drunken Sigma Chi boys on Spring Break from Penn State. The only daughter of the owner of a struggling trattoria, she will have the heart of poet and the sensitive wit of a Regina Spektor lyric. Our love will blossom like the last flowers of June, and after months of sexual tension and exquisite trattoria food, we will finally consummate our relationship. But the next morning, I’ll receive word that my beloved Albanian bulldog, Bullard, has died. Stricken with grief, I will weep tears of manly remorse, and rue that I was not at Bullard’s side in his final hours. I will return to the United States to further weep, and to visit his grave under the shade of a poplar tree in my parents’ backyard. But in the ashes of this tragedy a fire will kindle within me, and I’ll start a popular YouTube channel in which I talk about the wonders of Italy and Albanian bulldogs while wearing an oilskin duster in burnt sienna, a color that brings out my eyes. As my channel gains subscribers, I’ll start a companion blog and podcast and soon become a widely recognized blogger and YouTube celebrity.

So you see, I’ve got this all figured out. The end of generic full-stack web development will ultimately be a good thing for me. Maybe it can be a good thing for you too.

Introducing MonkeyType.js!

When I needed to choose a Javascript framework for a work project, I put a lot of thought into it. I tried Angular, Angular 2, Ember, React, Redux, Backbone, Dojo, Vue, Mithril, Marionette, CanJS, Knockout, Polymer, and js_of_ocaml. None of them worked. They all just felt wrong.

What I eventually realized is that these frameworks felt wrong because they don’t use the right abstraction. Modeling web apps as MVC, MVVM, MVP, MVT, MVASDFASDHSFGSDFGHQRTQRTAFGLKJNNGJSKL, these are all the wrong abstractions.

That’s why I decided to create my own Javascript framework. Instead of using broken models of abstraction like Model-View-Controller or Model-View-View Model, it uses MOT: Monkeys On Typewriters. And I called its name MonkeyType.js.

I know what you’re thinking: what does a stupid front-end engineer who took on $157,000 in student loans to follow his passion and get a degree in Lithuanian literature with a minor in Cloud Shape Interpretation Studies, laid around on his mom’s couch for five years, got an online Master’s degree in Motivational Speaking, worked as a tutor for left-handed children, decided to enroll in divinity school, dropped out after accruing another $34,000 in debt, lucked into a plum job at a grocery store making $15 an hour where his only real duties were printing signs for the manager to stick in the break room, learned HTML, CSS, and Javascript after the manager asked him to make a website for her side business selling purebred blue fawn Azawakhs (also known as the Tuareg Sloughi according to the AKC), found out you could make money with this Javascript stuff, and finally moved out of his mom’s house, know about designing a framework? Way ahead of you! MonkeyType.js is based firmly on MATH! It uses the celebrated Infinite Monkey Theorem, described by famed French measure theorist Félix Édouard Justin Émile Borel in 1913, to created a model of UI firmly based on MATH!, with grounds in MATH!, concretely resting on a foundation of MATH! to create a uniquely appropriate abstraction coming, like all the greatest and most appropriate abstractions, from MATH!

The Infinite Monkey Theorem states that given infinite monkeys banging the keys on infinite typewriters for an infinite length of time, the probablity that Hamlet will appear somewhere on one of the monkey’s typewriters is equal to 1, i.e. one of the monkeys almost surely produces Hamlet. MonkeyType.js takes this theorem to heart and uses it to create an abstraction for UI programming that’s so wrong, it couldn’t be more right. The best part about a client-side framework based on the Infinite Monkey Theorem is that the Infinite Monkey Theorem is MATH!, and uses terms like almost surely in confusing ways that only make sense if you know MATH!, and are otherwise just confusing and misleading! You have to be really smart to know MATH!, so you also have to be really smart to understand MonkeyType.js. No dummies who only know how to make solid, functional user interfaces that are intuitive and aesthetically pleasing; only the best front-end engineers need apply to use MonkeyType.js, because with MonkeyType.js, you have to be able to do all of those things, and also know MATH!, and not dumb kiddy stuff like fractions, but REALLY HARD MATH! that famous French guys invented. Everyone knows French stuff is really cool and hard to understand; the other night I watched Amelie, and I had no idea what was going on, but it was French, so I can just go up to people and say “Hey, have you seen Amelie? It’s French” and they know how smart and cool I am to be able to understand French stuff. And then they’re even more impressed when they find out I’m the inventor of MonkeyType.js and I understand MATH!

A typical MonkeyType.js program has two parts: the monkeys, and the typewriters. The monkeys represent the data. They’re sort of like the model in other frameworks, except with more MATH!, and more French. The reason monkeys are a better abstraction for data than a model is that models are stupid and for babies. When I was a stupid baby I got this model of an airplane for my birthday. It was wood. I kept trying to build it and getting splinters in my fingers that really freaking hurt. Finally I got a splinter under my fingernail, and it hurt so bad that I got mad and threw that dumb airplane model across the room and it broke, and my mom had to come in and hug me. I wouldn’t stop crying, so finally my mom had to take me to the zoo to make me quiet down. I didn’t stop crying until I saw the monkeys. I was only fifteen years old, but I kept that love of monkeys my whole life, and I never cried again, until I got home and stepped on a broken piece of the airplane model and got a splinter in my foot.

The typewriters represent the appearance, sort of like the view in other frameworks, except with more MATH! Typewriters are way better than views. I hope I don’t have to explain this one; views remind me too much of 1985’s A View to a Kill, and then I spend all day mumbling “Dance into the fire” under my breath and my smarmy co-workers who pretend to be my friends but secretly laugh at me because they all hacked into the Pentagon’s computers when they were ten years old and I didn’t learn to program until I was forty-seven start talking behind my back.

The monkeys pound on the typewriters to create the website. Unlike other dumb frameworks that have something like a controller standing between the model and view, MonkeyType.js lets the monkeys pound directly on the typewriters. Here’s where the really cool part comes in. Obviously we can’t have infinite monkeys and infinite typewriters on a modern computer with finite memory, so MonkeyType.js uses a clever optimization called a monkey barrel. Instead of infinite monkeys, the monkey barrel contains about twelve or thirteen monkeys that it keeps recycling. Every time you ask for a monkey, the monkey barrel gives you a monkey. It might be the same monkey you got last time. It might be a different one. Since one monkey’s as good as another monkey, with this approach, you effectively have infinite monkeys. The typewriters are stored internally in a circular buffer, which gives the impression that there are infinite typewriters, but it’s actually just, like, twelve or thirteen typewriters on a loop. And to create the impression of infinite time, MonkeyType.js is coded in an excruciatingly dumb way so that all the pages hang for at least a minute before doing anything, and when they do do something, they just load a spinner that swirls for another minute or two before the actual page you were looking at finally loads. This impression of infinite time is increased even further in production enviroments, where hundreds of trackers and advertising services also get to load before your content. This makes MonkeyType.js the perfect framework for rich media sites like Buzzfeed, Huffpost, and ZergNet, where every page is so obviously loading five hundred ads before it even thinks about delivering the content you asked for, because the advertising is actually the content and the content was actually just a bait-and-switch to get you to look at ads. But who cares, because we deliver ads with a firm basis in MATH!

In conclusion, where other JS frameworks give you broken abstractions based around dumb ideas like being easy to learn, easy to write, or easy to use to create great UIs, MonkeyType.js gives you abstractions based on MATH! and awesome stuff like monkeys and typewriters. Who cares if it causes memory leaks in your browser? It’s MATH! I hope you’ll consider MonkeyType.js for your next project, and finally learn what a real abstraction looks like.

Three Haiku on Clojure

These are three random topics about Clojure that are sort of connected, in that all three address Clojure’s incredible flexibility and genericity from different angles. They were born from a few recent small projects that I started.

My favorite facility for polymorphism in Clojure

Object-oriented programming, at least as presented in C++ and Java, lets you do three things much more easily than languages like C:

  1. Encapsulation. You can hide data from certain parts of the code.
  2. Inheritance. You can create hierarchies of related objects which share some behaviors and modify other behaviors to suit themselves.
  3. Polymorphism. You can implement a function which behaves differently depending on the type of its arguments.

I’m being deliberately vague. Even though these three principles were all over the multiple choice section of the exam on Java when I was in school, they actually transcend object-oriented programming. They’re things which are helpful in all languages. Languages differ in how they support these principles, though.

Encapsulation lets us limit the number of things we have to worry about by only allowing certain parts of the code to modify data. If we have a global variable and we discover in debugging that its value is wrong, the piece of code that made its value wrong could be anywhere. If we have a private member of a class, the piece of code that made its value wrong can only be in that class.

Clojure doesn’t really support encapsulation. Like JavaScript and C, data in Clojure is local or global. You can do weird contortions with closures to make things private, like people do in JavaScript. I wouldn’t, but you can. Since Clojure data is almost all immutable, it’s not nearly as necessary to have encapsulation.

Clojure doesn’t really support inheritance either. But how much of the inheritance we see in Java is about avoiding code duplication, and how much is about the type system? Clojure is dynamically typed, so it doesn’t need inheritance for a type system. And Clojure has macros, the ultimate way of reducing code duplication, so it doesn’t really need inheritance for that reason either. You can build your own inheritance using derive and the polymorphism tools, but the language doesn’t support it directly, not really.

Clojure supports all kinds of sophisticated mechanisms for polymorphism. Records, types, protocols, multimethods. Probably some more that I’m forgetting or that only exist in the upcoming 1.8.1 alpha pre-release snapshot. All of them are useful and interesting, but my favorite example of Clojure polymorphism is probably tree-seq.

Why do I love tree-seq so much? Because this is the only polymorphic function in any language I can think of that both does not care, in any way, in the slightest, not even a little, how you represent the data it works on, and is also so simple that you can make a call to it in one line with practically no boilerplate:

(tree-seq #(not (nil? %))
  #(get % 1)
      [:root
          [[:a [[:b nil] [:c nil]]]
              [:d [[:e
                    [:f nil]]]
                  [:g nil]]]])

It works on this horror of nested vectors, producing a lazy sequence of the child vectors of each node. Even though this thing is the most ghetto way I can imagine to represent a binary tree, tree-seq doesn’t care and still processes it.

It also works on slightly more sophisticated structures:

(defrecord Node [name children])
(def t
    (->Node :root
      [(->Node :a
               [(->Node :b nil) (->Node :c nil)])
       (->Node :d [(->Node :e
                           [(->Node :f nil)])
                   (->Node :g nil)])]))
(tree-seq #(not (nil? %)) :children t)

tree-seq is so cool, you can even do weird things like storing the tree’s structural information in one place and its data in a completely different place:

(def children {:root [:a :d], :a [:b :c], :d [:e :g], :e [:f]})
(tree-seq #(not (nil? (get children % nil)))
      #(get children % nil)
      [:root :a :b :c :d :e :f :g])

Sure, this example is weird and pointless. But it works. tree-seq does not give a hot toddy how your tree is represented, as long as there’s some way to tell which nodes are leaf nodes and some way to get the children of non-leaf nodes.

There’s something refreshing about that freedom. It’s nice to think “For the rest of my code, it would be really nice to represent a tree as XXX” and know that Clojure’s built-in functions can work with that representation. No need to create abstract anonymous protected thread-non-mutable generic reified construction apparati for your new tree representation, or write five hundred lines of XML to register it with your dependency injection framework.

You Can Only Luminus Once Per Project

When I wrote Odyssey Through Three Web Frameworks, I was under the mistaken impression that Luminus was a web framework like Rails. It’s not, and you will be disappointed if you expect it to be.

Facts about Luminus:

  1. It’s implemented as a Leiningen plugin, and in the Java world it would probably be part of Eclipse.
  2. It generates a bunch of useful namespaces for you, and includes a bunch of useful libraries for you, and it does give you some stub functions, but it has nothing as extensive as Rails’s ActiveRecord or Django’s admin interface.
  3. If you don’t like the libraries that Luminus includes for you, it’s easy to pick a different one. Luminus uses Selmer, which is like a Clojure implementation of Django’s template language. I use Enlive. Luminus was okay with that, and we were still able to work together.
  4. Luminus still does not do security.
  5. Luminus is Homu Homu’s favorite Clojure technology, even if she can only use it once a day.

In short, Luminus is still just as do-it-yourself as Clojure always was, while helping out with boilerplate and giving just a bit more guidance than you would get without it.

The thought-polluting effects of Clojure

Recently, I wrote some Python code. It’s been a while, and it was surprisingly good to be back, even though I was definitely not loving on LXML.

A few years ago, whenever I started a new program, I would write a class. I would think about how to carve things up into classes. A few times, I wrote C programs, and I felt lost and confused because I couldn’t put things in classes.

I did use standalone functions in C++ and Python, but only when the function was completely unrelated to the rest of the program. If it was some random utility, I would write it as a standalone function. And if I was overloading an operator that C++ requires to be overloaded as a standalone function, I would write it as a standalone function, after first trying to write it as a method, staring at the bizarre compiler errors for five minutes, and then remembering that you can only overload certain operators as standalone functions. But I usually wanted everything in a class, even when I wasn’t really using the facilities of classes for anything. Both versions of PySounds, for example, stick all the logic in a class, then have a main module that instantiates an object of that class. In PySounds2, the main module does other UI-related things as well, but in PySounds1, it was almost literally just the code if __name__ == "__main__": PySounds().runStuff(lexfile, scfile).

Then suddenly I ran into Clojure. In Clojure, there are no classes. You write everything as standalone functions and stick it in a namespace. I wouldn’t say I felt lost and confused over this. Unlike C, Clojure has namespaces, so I didn’t feel like I was making program soup. But it did take some getting used to. I proceeded to get used to it.

Now, when I go to write some Python, I don’t jump right to classes. I go as far as I can with standalone functions and modules. If I write some code that feels twisted because of the data access patterns, I’ll consider whether it seems classy or not. If it does, I might write a class. Or I might not; I might just add some keyword arguments to a function. I might just stick two pieces of data in a tuple and pass that in or out. I organize Python like it’s Clojure, with modules as namespaces.

This is still pretty idiomatic in Python, but a while back, when I unconsciously applied the same pattern to Java, things got ugly. Java wants, needs, can’t live without, everything in classes. I wanted things not to be in classes. I wrote a class with a bunch of static methods. When I put it up on Code Review.SE, someone complained that it wasn’t object-oriented. And that was just, because it wasn’t, because I didn’t want it to be and Java did and we fought and ended up at a lousy compromise, as usually happens when you disagree with Java.

I’m not sure if there’s really any significance to this change in approach. I’ve felt for at least the past couple years that object-oriented programming was a lot more heavy machinery than I needed for most of the programs I work on. Especially in Java, when you start getting into generics and things get crazy-complicated.

I kind of appreciate Martin Odersky’s approach to Scala, where he knowingly cordoned off all the crazy type rules and made them library implementors’ problem, both at the language level and culturally in the Scala community. Actually, I appreciate a lot of things Odersky has said and done, which is why I have kind of a soft spot for Scala even though the language, on first inspection, doesn’t really match my aesthetic taste. I would like to spend some more time with Scala someday. That might revive my interest in object-oriented programming a bit. Right now it’s at an all-time low.

On the other hand, I recently went over some posts on Code Review.SE where the posters seemed to have totally ignored Clojure’s rudimentary object-oriented features, either knowingly or from ignorance, even though those features were perfect for their problem domain. (Both games, as I remember.) I realized that I really don’t know these features that well, so I got started on a simulation program where I would use them. I haven’t gotten far yet, but I’ve been quite pleased with records and protocols so far. They seem to be simple, flexible, and performant, while also being more sophisticated than regular maps or C structs.

Conclusion

I guess the only takeaway here is that Clojure has had a positive effect in code I write in other languages, by pushing me to make things more polymorphic and generic, and by spurring me to resist needless complexity. Paradoxically, by refusing to be object-oriented, Clojure has brought me greater understanding of object-oriented programming, both its upsides and its downsides. Now, instead of mindlessly classifying everything, I see how far I can go without classes, and when some particular code seems like it could really use a class, I use one. Rather than heavyweight classes with fifty member variables and hundreds of methods, I prefer lightweight classes with a single obvious purpose.

It’s tempting when you first learn about OOP to use it all the time, for everything, especially when your professors and your languages are expecting it. But it’s good to find out how things would go if you didn’t have OOP, whether you’re using C, JavaScript, or Clojure.

I like Python! No, wait, Ruby! No, but Python…

I hate Ruby.

There I was, just buzzing along, using Python, satisfied that I knew one of the coolest languages around, and definitely the coolest language you could actually get a job with.

Then Ruby came along.

It was English-like. It was practical. It was fun. It was possible to get a job with it. It was even Japanese.

Almost everything about Ruby’s syntax, language features, and library just made sense to me almost immediately. I don’t mean I immediately understood the entire language—blocks took me quite a while to understand, and metaprogramming can be pretty confusing for almost anyone, unless you’re Dan Friedman. I mean that I almost immediately saw, on some intuitive level, why Ruby had been designed the way it was. My mind easily fell into thinking the way Ruby wanted. Even when I barely knew the language, I could figure out just where to bust out a line-ending if or while. I immediately loved string interpolation.

By contrast, there are plenty of things in the Python language and libraries that, to this day, do not make sense to me. I learn them by memorization and use them. Examples are len being a function instead of a method, and the entire os.path module. Once, I needed a quick script to process a few files in some nested directories. I’d only been learning Ruby for about three days, and I wanted it done quickly, so I used Python. I struggled for an hour to write a Python program using os.path.walk, and finally decided to look into how Ruby would do it. I had the thing done in Ruby in ten minutes.

Other parts of Python make sense in a tortured sort of way, like writing ' '.join([1, 2, 3]) instead of, say, join([1, 2, 3], ' ') to join some strings. Yeah, fine, I understand that it makes some kind of sense to think of it as “String ‘ ‘! Join the list [1, 2, 3]!” rather than “List [1, 2, 3]! Join yourself using the string ‘ ‘!”. Just like it makes some kind of sense that C++ uses virtual void flagellate(int flagellant) = 0; to declare a pure virtual function, since you’re initializing the function’s pointer to null. But just because they kind of make sense when you think really hard about them, that doesn’t make them obvious (like the Zen of Python claims to want), and definitely doesn’t justify the kind of condescending blather that the Python FAQ presents in response to the question of why it’s ' '.join([1, 2, 3]).

Maybe it’s because I started as a natural language guy, in English and linguistics, that Ruby makes so much sense to me. People complain that it has too many different ways of accomplishing the same thing. It definitely flies in the face of Python’s “There should be one obvious way to do it”. On the other hand, Python itself frequently chooses ways to do things which aren’t incredibly obvious, like the aforementioned os.path.walk. That there is only one way to do it, if anything, just makes the non-obvious nature of that way hurt even more, because there’s no way to get out of it.

To me, Python is designed like a formal system, while Ruby’s design is more like a natural language. Of course, there are languages much more like formal systems than Python; there’s Lisp, and Haskell, and Scala. Even Java, once you stop looking at syntax and warts like arrays and primitive types, and just look at its efforts to be “objects all the way down”. But Python seems to have aspired to the same kind of minimalism as Lisp. Python aspires to rid itself of synonyms and homonyms, to make all antonyms expressible by sticking “not” at the front. Ruby takes on extra syntactic baggage like “unless” and suffers methods with different names that do the same thing, in the name of being more obvious and readable to humans. Python believes that explicit is better than implicit. Sometimes you have to write more for the sake of making it explicit. Ruby tolerates weirdnesses like the flip-flop operator, which does a boatload of implicit stuff behind the scenes, on the principle that it’s fairly common to do something like that, so you can memorize one set of implicit behavior.

Please don’t take this to mean I think Ruby is better than Python. I said that Ruby, the language, and several of the libraries, fit my style of thinking better than Python. And even more, please don’t take this to mean that I think crazy things like Perl’s implicit $_ variable are okay. Ruby, I think, gets just the right amount of redundancy and implicit behavior; if Python has too little, then Perl has far, far too much. Ruby, I think, gets just the right distance between natural language and formal system; Scheme and Haskell are a little more like a formal system than I prefer, but Cucumber and AppleScript get much closer to natural language than they should. Cucumber and AppleScript are wonderfully easy to read, but horrible to write, while Scheme and Haskell are nice to write, and the end product is very aesthetically pleasing in a classical sort of way, but can be rather hard to read.

So, given that I liked the Ruby language better than the Python language, why haven’t I dumped Python yet? Third-party libraries.

Ruby has amazing libraries related to the web. Rails has been extensively talked up. I made another visit to it after I put some serious effort into learning Ruby, and I found it a lot more pleasant than I remembered. By contrast, I started to find Django code-y and somewhat laborious. Sinatra is also well-liked. Nokogiri blows away any HTML/XML processing library anywhere that I can think of; Python’s Beautiful Soup is great for scraping HTML, but less useful for XML, while Clojure’s Enlive is great for scraping and templating, but again, less useful for XML than Nokogiri. Let’s not even talk about Python’s standard library HTML and XML parsers; the HTML parser is a horrible Java-wannabe library that makes you implement your own API, while the XML parsers are cumbersome and limited.

Ruby’s libraries in other areas that are of interest to me are less impressive. Python has NumPy, SciPy, Natural Language Toolkit, BioPy. Clojure puts in an appearance with Incanter and OpenNLP, and there are some Java libraries here that you can use. Ruby has basically nothing in scientific computing, NLP, or bioinformatics. It could be great for all of these areas, because they’re all interdisciplinary, and I suspect that people from outside math and computer science would find Ruby easier to just get than other languages, the same way I did. But the Ruby community’s main concern is the web, and it shows.

If Python were a bad enough language, I wouldn’t feel so conflicted. But Python’s not a bad language; it’s a great language with a few annoying facets, most of them superficial. I like Ruby the language better, but Python’s sure better than a cup of hot Java in the face, to say nothing of being skewered by a C++. Most of the libraries I want are in Python. Some are in Ruby, but the ones that aren’t are huge ones that aren’t likely to be replicated in Ruby any time soon. Python is, in fact, probably the only language that has all the libraries you could ever want, is pretty nice to read and write, and can also get you a job. And I would have been happy with it forever, if that stupid Ruby hadn’t come along.

But I’m also mad at Ruby for creating this internal conflict for another reason: if I had never learned of its existence and just committed to Python, maybe I would know it well enough to have a job by now.

Stupid Ruby.

Understanding Dynamic Arrays and Linked Lists

Almost all sequential data structures are implemented using either dynamic arrays or linked lists. Both of them are linear structures that grow as you add new items, but sometimes you can really torpedo your program if you use the wrong one. Here I’ll talk about the differences between them, and when one or the other makes sense.

Arrays: Like the Cereal Aisle

Arrays are like the cereal aisle at the grocery store.

cereal-aisle

When you’re shopping and you see the cereal you want, it’s easy to just reach out and grab a box, no matter where it is on the shelf. If we want a box of Honey Nut Cheerios, we just find Honey Nut Cheerios and grab it; we don’t have to even glance at other cereals, and we don’t have to move anything or reorganize anything. As we’ll see later, linked lists do not have this property.

If we worked in the store and wanted to switch the positions of two cereals—say we wanted to swap Frosted Cheerios and Multi-Grain Cheerios—that’s also pretty easy. We just take out every box of Frosted Cheerios, put them on a cart, then take out every box of Multi-Grain Cheerios and put them in Frosted Cheerios’s former spot, then take the boxes of Frosted Cheerios off the cart and put them where Multi-Grain was.

On the other hand, it’s hard to add new things to the cereal aisle. Say that General Mills comes out with a new Crispy Baklava Cheerios, and we wanted to put it next to Honey Nut since they both have a honey theme to them. We’d have to move down whatever that is next to Honey Nut in the picture (Chocolate Cheerios?). But then to move down Chocolate, we’d have to move down what’s next to it, in the purple box. And to move down the purple boxes, we’d have to move down the blue boxes next to them, and so on. If the shelf has space left at the end, and we’re not too fussy about where the new cereal goes, it’s pretty fast to stick it in at the end. It won’t be with the other Cheerios, but maybe we can do a swap so that it’s at least in the Cheerios section, even if it’s not right next to Honey Nut. If the shelf is full, we just can’t do it; we have to replace something, or move everything to a new shelf with more space.

This is also like arrays. Arrays are fixed-size. If you run out of space, you have to allocate a new array with more space and move everything over to it. If you want to put something in the middle, you have to move down everything that comes afterward. Both of these things can end up being huge wastes of time if you do a lot of them.

There’s another way you could deal with arrays having a fixed size: make an array that’s way bigger than you think you’ll ever need. Then you shouldn’t ever run out of space. But this wastes memory; when you make an array of 200 elements to hold a string that contains someone’s first and last names, which usually comes out to less than 40 characters, you’re holding on to memory that could be used for other things. Sure, modern computers have lots and lots of memory, but there’s no need to squander it when there are better options, just for the off chance that you’ll have to handle Finnish names.

Dynamic arrays are arrays that deal with these problems automatically. They grow bigger as you add elements to them, so you don’t have to worry about running out of space. They also balance speed and memory usage; a typical dynamic array implementation sometimes has to allocate new chunks of memory, but it also leaves itself room to grow, while not squandering memory by holding on to tons of space that it won’t need.

C++’s STL vector, Java’s ArrayList, and Python’s list all use dynamic arrays. In Python, you can do this

ls = []
for k in range(1000000):
    ls.append(k)

and it isn’t horrendously slow or wasteful of memory, because Python uses a clever trick behind the scenes. (On my machine, that code runs in about half a second, and my machine is pretty second-rate.) For one thing, every time a Python list needs to expand, it doesn’t just allocate enough space for one more item; it allocates enough for a few. 1 That way we’re not wasting too much memory on taking space which might never be used. On the other hand, we don’t need to allocate new memory and move everything over as often. Since allocating and moving over are time-consuming, this makes a big difference. If we only allocated three extra spaces for each allocation, for example, we’d have to reallocate and move a third as often.

Python also saves on memory allocation by reusing blocks of memory. Sometimes, when a data structure isn’t used anymore in the program, Python doesn’t actually get rid of the memory it uses. Instead, Python puts the memory in a “recycling bin”. The next time it needs memory, it looks in the recycling bin to see if any of the memory blocks there are the size it needs. If so, Python just erases the data in that memory block and stores the new data in it.

C++ and Java use similar schemes to save space and time. If you look at the methods for the vector and ArrayList classes, you’ll find that some of them refer to both a size and a capacity. The size is the number of things currently stored in the array that backs up the structure. The capacity is the amount of memory space in that array. When size == capacity, it’s time to allocate a new block of memory and move everything over.

Linked Lists: Like being a Train Conductor

Linked lists are like being on a train.

4carLinkTrain

Say you’re the conductor on a train and you need to find a specific passenger because it turns out he’s a stowaway with fake papers and you need to throw him off. You don’t know exactly which car he’s in. You’re at the front of the train. In order to find this passenger, you have to walk back through the cars until you see him. He might be in the second car. He might be in the last car. In the worst case, he might have jumped the train a few stops back because he knew you were on to him, and you have to search the entire train for a passenger who’s not even there.

This is how it is searching in a linked list. Since the nodes are hooked together by pointers and are stored at arbitrary places in memory, you have no choice but to look at the data in every node, following the pointers along the list, until you find what you’re looking for or come to the end. It’s not like the cereal aisle where you can just scan for what you want, then grab it right off the shelf.

But being on a train has some advantages, too. Say our engine blows and we need a new locomotive. Just pull into the train yard, unhook the locomotive, drag it into the depot, and then drag on a new one. The caboose is also easy to replace. And if we want to add a new first-class sleeper car in the middle, that’s also not so bad; we can uncouple the cars at the point we want to insert the new car, drag them apart, and use the shunting rails to get the new car moved in between. Then we just couple the new car to both halves of the old train, and we’re done.

Inserting new items in the middle or at the front of a linked list is quick and easy, because all you have to to do is reassign some pointers. The pointers are like the train couplings in the example; you just have to reconnect them, and you have a new node at the center of the list.

Unlike a dynamic array, which can grow, but makes you pay the huge cost of reallocating and moving everything, a linked list can keep on growing, one node at a time, until it’s filled all of memory, without incurring too much of a penalty. You never have to move anything, because it’s easy to attach a new node at the end or wherever you want it.

Where you use them

The dynamic array types (Python’s list, Java’s ArrayList, and C++’s vector) are usually the workhorse sequential collection of their language. Linked lists are usually used for more specialized collections. Since it’s quick and easy to add things at the beginning or end, and you can keep growing them without moving anything, they’re good for stacks and queues.

Linked lists are also good because their structure is flexible. You can do all sorts of weird things with them. For instance, there is the circular linked list. This is a linked list whose final node has a pointer to the first node, so when you traverse the list, you end up at the beginning again when you’re done.

One advantage of doing things this way is that you reduce the risk of dereferencing a null pointer in languages like C and C++. Rather than checking whether the next node is null, you can check whether it’s the same as the head; if you accidentally access the null pointer while you’re doing this, your program will crash, but if you happen to access the head of the list it’s no big deal.

Another advantage of circular linked lists is how clean and quick it is to reuse space. Memory allocation is fairly expensive, so there might be times when you want to keep the same nodes around and just reassign their data. You can make a queue that reuses nodes with a circular linked list. Keep around pointers to the head and the node where the next item will be added. When you remove an item from the queue, just point the head pointer at the next node. Now your queue has another “empty” node at the end. When you insert, check that you’re not trying to insert at the head. This sort of deletion is called lazy deletion, because you haven’t really deleted anything; you’ve just marked it unused (by switching the position of the head pointer).

When it comes to weird things you can do with linked lists, though, nothing beats the skip list. But you’ll have to read about that on your own, because I’m out of time. I hope this post has shed some light on the differences between the two common linear data structures.

  1. See this blog for more on Python’s list implementation.