You Won’t Be Remembered For Your Code

A lot of people who are first getting into the software industry seem to think that they’re going to make some kind of impact with their code. The dumber ones think it’s going to be some kind of huge splash, like people are going to print their code on giant wall scrolls and put it up in museums so future programmers can come through and admire its structure and that amazing little trick with the lambda. The smarter ones know it will be a small impact, but they still think it will be an impact of some kind.

Allow me to rain on your parade: you will never be remembered for your code. Code in industry software is endlessly erased and rewritten. Even that picture of a penis you carved into the wall of a bathroom stall in sixth grade and then initialed like it was some great work of art is more likely to still be there, because replacing the walls of bathroom stalls is expensive and laborious. But erasing your clever little trick with the lambda and replacing it with ten lines of plodding that even the biggest dunderhead couldn’t possibly fail to understand takes about five minutes, and people who erase code are on salary, so it didn’t even cost the company five minutes of work, whereas people who replace bathroom stall walls get paid hourly. Plus, now even the biggest dunderhead couldn’t possibly fail to understand that code, so they can fire the programmer who replaced your clever little lambda trick and hire the biggest dunderhead for a lower salary.

No one is remembered for code. Steve Jobs is remembered for product design and marketing. Bill Gates will be remembered for being insanely rich. Programming languages are products, so programming language creators are also remembered as product designers, not for their wonderful code, and they’re easily forgotten once no one is using the language anymore; give it twenty years and Larry Wall will be a random piece of programming trivia alongside Jon Ousterhout, Jean Ichbiah, and Niklaus Wirth. Same goes for frameworks, which are even more disposable than programming languages. If anyone remembers David Heinemeier Hansson or Rails at all, it won’t be for some ingenious metaprogramming trick he used in Active Record.

And these are the really famous people, the people whose code actually affected the way thousands of others do their jobs. Even they will not be remembered for how wonderful their code was. So trust me when I say that you won’t be remembered for your great code.

However, if your code sucks, it will keep your memory alive for a while. Every time someone goes into a sloppily formatted file with dozens of unnecessary global variables, lots of overlong methods, and comments that say x++; // Add one to x, they’ll head on over to Git and find out who wrote this crap. If you wrote enough of the code, you might even become something of a legend for a while. Of course, the other programmers will make a point of extinguishing your imprint from the code, so your code is still headed for death, and your legend will live on only in the tales the other programmers tell of the crappy code they suffered with until they could sneak in enough refactoring to eliminate it. You’ll be remembered, but you’ll be remembered the same way Tom Green is remembered, as an irritating bore that everyone raced to expurgate as quickly as humanly possible.

So don’t get into software if you think code is a good way to be remembered. It’s not. Product design is more likely to get you remembered by someone, but that’s not foolproof either; not many people, even in programming circles, have heard of Ward Cunningham, who invented the wiki, or Dan Bricklin, who invented the spreadsheet, even though those are both products at least as important as anything Steve Jobs ever made.

Advertisements

JSON in Java: A Case Study in Pain

Dealing with JSON in most dynamically typed languages is dead simple: you have a function that can inflate a string into the language’s native hash map type, and you have another function that can serialize the language’s native hash map type into a string of JSON. These languages all support nested maps, some sort of list, and strings, numbers, and booleans well enough to automatically map from JSON’s simple data types into the language’s native types.

Dealing with JSON in the emphatically statically typed, the holdover that’s still hanging around like a bad hangover, the one, the only Java will eventually make you wish you were dead.

I’ll be up front with you: this isn’t going to be a deep or insightful read. It’s mostly going to be a rant, with lots of hate for Java and static typing, but mostly static typing as seen in Java, because I’ve never tried to work with JSON in languages like Haskell with more sophisticated type systems, so you Haskell lovers are free to stay smug in the knowledge that you work with a better class of type system, nay, not a class in sight, but rather a category. If you love Java and static typing, you might want to tune out, because I am not at all interested in hearing for the hundredth time how dynamic languages force you to write unit tests for the stuff that static languages check at compile time. This is also not meant as a top-to-bottom indictment of static typing; it’s an examination of how static typing utterly fails to solve one specific problem in an elegant way.

To start with, I’m going to present the problem. Then I’ll examine what I find to be the root causes of this problem, and then rant a little about some features Java will never get that I think could help this problem.

The Problem

There are two major things you need to be able to do with JSON in your programming language: turn a string of it into your language’s native data types so you can work with the data, and turn your language’s native data types into a string of it so you can send it across a network.

JSON—Javascript Object Notation—is based on a restriction of Javascript’s native data types. Looking at the standard, we see that it supports objects, arrays, strings, numbers, booleans, and null. On the face of it, it seems like we should be able to represent these types with native Java types: maps for objects, lists for arrays, and Java’s string, number, boolean, and null for those types. But maps and lists in Java have to be restricted to a single type, so to represent JSON values like {"status": true, "message": "Status normal"} or [23, 'hello', false], we have to use a Map<String, Object> and a List<Object>.

Unfortunately, this almost immediately leads to problems. Java lists and maps don’t support being stringified as valid JSON, so you have to write custom code to do it. Actually writing this code isn’t too hard as long as you remember that JSON supports arbitrarily nested lists and objects, so you have to make sure your serialization is applied recursively until you bottom out. But it has all the usual disadvantages that come when you write custom code for fundamental operations instead of getting it from a library: you have to test it, support it, and maintain it yourself, and it’s harder to bring into new projects since you can’t just let your build tool grab it from Maven Central.

Working with Objects in Java is rather painful; if you want to do anything with them, you have to cast them. This requires you, the programmer, to know what type every one of those objects actually is. Look at this rather ugly bit of code:

Map<String, Object> jsonMap = getJsonMap();
List<Object> jsonList = jsonMap.get("theList");
if ((Boolean)jsonList.get(2)) {
    Integer idNumber = (Integer)jsonList.get(0);
    User user = getUserById(idNumber);
    user.setPreferredGreeting((String)jsonList.get(1));
    user.save();
}

Not only is this code ugly, it also loses the much-vaunted type safety of Java. You could get runtime type errors if it turns out that the first item of jsonList is not actually an Integer or the second item is not actually a String.

We can solve the ugliness problem a bit by throwing out the built-in Java map and list and using a special set of classes to represent JSON data types. This is what org.json or Jackson and GSON’s tree structure use. Instead of a Map, we’ll use a JsonObject; instead of a List, we’ll use a JsonArray; and instead of using Java strings, numbers, and booleans, we’ll use a JsonPrimitive. We’ll make all of these classes implement the JsonType interface.

The JsonObject class will behave like a Map<String, JsonType>. The JsonArray will behave like a List<JsonType>. The JsonPrimitive will store its value as a string, or possibly an Object. It will have a set of three methods: getAsNumber, getAsString, getAsBoolean, and possibly corresponding isNumber, isString, isBoolean. We glossed over the extra complication that the Javascript/JSON number type can be either integral or floating point; we also glossed over the extra complication that JSON numbers don’t have a specified size, so what we get could potentially overflow an Integer or Double. We could handle those problems here either by making getAsNumber return a BigDecimal or making separate getAsInteger, getAsDouble, getAsBigInteger, getAsBigDecimal methods. Or we could decide that overflow will be rare enough that we’ll tolerate errors when it happens, and make getAsNumber return a Double.

With this new set of classes, we can rewrite the above example:

JsonObject root = getJsonMap();
JsonArray array = root.get("theList");
if (array.get(2).getAsBoolean()) {
    User user = getUserById(array.get(0).getAsInteger());
    user.setPreferredGreeting(array.get(1).getAsString());
    user.save();
}

It’s at least less ugly without all those casts and temporary variables, and since the types make it obvious that we’re dealing with JSON, we don’t have to name everything jsonWhatever. But we’ve still lost type safety; if we do array.get(2).getAsBoolean() and that item isn’t a boolean, we’ll still get a runtime type error. And these custom JSON types are inevitably less convenient to work with than the standard library data types. What options do we have if we want to iterate over a JsonArray? Does it implement Iterable<T>, which will let us use the “smart for-loop”, or are we stuck with the C-style for-loop? Can we get the entry set of a JsonObject and iterate over all its keys and values if we want to? Do these classes implement Collection<T>, making them streamable?

This solution with separate JsonWhatever classes is the quick and dirty way of solving the JSON problem in Java. Imagine that your car breaks down in the middle of the desert. You have a tarp, a roll of duct tape, and a long pole in the back, so you tape the tarp to the pole, tape the pole to your car, and you’ve got a sail. The JsonWhatever classes are the homemade car sail of JSON handling. It does nothing whatsoever to fix the fundamental problem, but it does kind of get you moving again, in a way that’s slightly better than walking or pushing the car.

Most high-powered JSON libraries in Java go with a more sophisticated approach, borrowed from ORMs such as Hibernate. The Java language really really wants you to specify everything precisely, categorically, and at a low level: this class has these exact five fields with this five exact names, and this one’s an Integer, and this one’s a String, and this one’s a NosehairInspectorFactory, and so on. So why not create a class that specifies what your JSON will look like, precisely, categorically, and at a low level?

This is what Jackson and GSON do. They include a class, called ObjectMapper in Jackson, that can take a normal Java object and turn it into a string of JSON, or take a string of JSON and a normal Java class and create an object of that class from the string. Here’s an example:

class MyObject {
    private Integer myId;
    private String myPreferredGreeting;
    private Boolean isHasBeenGreeted; // Note, this is what you get when you insist on starting all boolean names with "is".
    private MyOtherObject myOtherObject;
    private List<Integer> allMyBuddiesIds;

    // All that getter and setter nonsense that Java insists on...
}

Jackson can map between this JSON and an object of that class:

{
    "myId": 23,
    "myPreferredGreeting": "Whassup dorks?",
    "isHasBeenGreeted": false,
    "myOtherObject": {
        "myFavoriteDrink": "Rolling Rock",
        "myFavoriteFood": "Cheese pizza with the crust cut off",
        "myLifePhilosophy": "YOLO bros"
    },
    allMyBuddiesIds: [21, 33, 49]
}

You might be wondering exactly how it does that. The answer is magic. As previously mentioned, I hate magic. But Jackson is magic from top to bottom. Except when it’s not.

The basic idea, to give a little more detail, is that Jackson can parse a JSON string into some intermediate form (which may or may not be the tree model it also offers, its version of our JsonWhatever types above; I haven’t looked into Jackson’s source code far enough to know). From there it can use reflection, possibly assisted by annotations you put on your class, to look for a way to map from the JSON object to your class. If you’re familiar with dependency injection, it’s pretty much the same idea: Jackson looks for a constructor or some setter methods it can use to inject the values from the JSON into an object of the class you provide. Then we have a regular Java object, referred to in this context as a POJO (“Plain old Java object”) since it’s not a special JSON-whatever object.

This sounds amazing. And it is pretty amazing, for JSON with all primitive types and no nesting. Unfortunately, Jackson doesn’t play well with generic types, so you end up having to use raw types for maps and lists, bringing us back to our original problem where we have to know what types are stored in maps and lists and cast them when we extract them.

The other problem with this is that some APIs return JSON responses that look like this:

{
  "statuses": [
    {
      "coordinates": null,
      "favorited": false,
      "truncated": false,
      "created_at": "Mon Sep 24 03:35:21 +0000 2012",
      "id_str": "250075927172759552",
      "entities": {
        "urls": [

        ],
        "hashtags": [
          {
            "text": "freebandnames",
            "indices": [
              20,
              34
            ]
          }
        ],
        "user_mentions": [

        ]
      },
      "in_reply_to_user_id_str": null,
      "contributors": null,
      "text": "Aggressive Ponytail #freebandnames",
      "metadata": {
        "iso_language_code": "en",
        "result_type": "recent"
      },
      "retweet_count": 0,
      "in_reply_to_status_id_str": null,
      "id": 250075927172759552,
      "geo": null,
      "retweeted": false,
      "in_reply_to_user_id": null,
      "place": null,
      "user": {
        "profile_sidebar_fill_color": "DDEEF6",
        "profile_sidebar_border_color": "C0DEED",
        "profile_background_tile": false,
        "name": "Sean Cummings",
        "profile_image_url": "http://a0.twimg.com/profile_images/2359746665/1v6zfgqo8g0d3mk7ii5s_normal.jpeg",
        "created_at": "Mon Apr 26 06:01:55 +0000 2010",
        "location": "LA, CA",
        "follow_request_sent": null,
        "profile_link_color": "0084B4",
        "is_translator": false,
        "id_str": "137238150",
        "entities": {
          "url": {
            "urls": [
              {
                "expanded_url": null,
                "url": "",
                "indices": [
                  0,
                  0
                ]
              }
            ]
          },
          "description": {
            "urls": [

            ]
          }
        },
        "default_profile": true,
        "contributors_enabled": false,
        "favourites_count": 0,
        "url": null,
        "profile_image_url_https": "https://si0.twimg.com/profile_images/2359746665/1v6zfgqo8g0d3mk7ii5s_normal.jpeg",
        "utc_offset": -28800,
        "id": 137238150,
        "profile_use_background_image": true,
        "listed_count": 2,
        "profile_text_color": "333333",
        "lang": "en",
        "followers_count": 70,
        "protected": false,
        "notifications": null,
        "profile_background_image_url_https": "https://si0.twimg.com/images/themes/theme1/bg.png",
        "profile_background_color": "C0DEED",
        "verified": false,
        "geo_enabled": true,
        "time_zone": "Pacific Time (US & Canada)",
        "description": "Born 330 Live 310",
        "default_profile_image": false,
        "profile_background_image_url": "http://a0.twimg.com/images/themes/theme1/bg.png",
        "statuses_count": 579,
        "friends_count": 110,
        "following": null,
        "show_all_inline_media": false,
        "screen_name": "sean_cummings"
      },
      "in_reply_to_screen_name": null,
      "source": "Twitter for Mac",
      "in_reply_to_status_id": null
    },
    {
      "coordinates": null,
      "favorited": false,
      "truncated": false,
      "created_at": "Fri Sep 21 23:40:54 +0000 2012",
      "id_str": "249292149810667520",
      "entities": {
        "urls": [

        ],
        "hashtags": [
          {
            "text": "FreeBandNames",
            "indices": [
              20,
              34
            ]
          }
        ],
        "user_mentions": [

        ]
      },
      "in_reply_to_user_id_str": null,
      "contributors": null,
      "text": "Thee Namaste Nerdz. #FreeBandNames",
      "metadata": {
        "iso_language_code": "pl",
        "result_type": "recent"
      },
      "retweet_count": 0,
      "in_reply_to_status_id_str": null,
      "id": 249292149810667520,
      "geo": null,
      "retweeted": false,
      "in_reply_to_user_id": null,
      "place": null,
      "user": {
        "profile_sidebar_fill_color": "DDFFCC",
        "profile_sidebar_border_color": "BDDCAD",
        "profile_background_tile": true,
        "name": "Chaz Martenstein",
        "profile_image_url": "http://a0.twimg.com/profile_images/447958234/Lichtenstein_normal.jpg",
        "created_at": "Tue Apr 07 19:05:07 +0000 2009",
        "location": "Durham, NC",
        "follow_request_sent": null,
        "profile_link_color": "0084B4",
        "is_translator": false,
        "id_str": "29516238",
        "entities": {
          "url": {
            "urls": [
              {
                "expanded_url": null,
                "url": "http://bullcityrecords.com/wnng/",
                "indices": [
                  0,
                  32
                ]
              }
            ]
          },
          "description": {
            "urls": [

            ]
          }
        },
        "default_profile": false,
        "contributors_enabled": false,
        "favourites_count": 8,
        "url": "http://bullcityrecords.com/wnng/",
        "profile_image_url_https": "https://si0.twimg.com/profile_images/447958234/Lichtenstein_normal.jpg",
        "utc_offset": -18000,
        "id": 29516238,
        "profile_use_background_image": true,
        "listed_count": 118,
        "profile_text_color": "333333",
        "lang": "en",
        "followers_count": 2052,
        "protected": false,
        "notifications": null,
        "profile_background_image_url_https": "https://si0.twimg.com/profile_background_images/9423277/background_tile.bmp",
        "profile_background_color": "9AE4E8",
        "verified": false,
        "geo_enabled": false,
        "time_zone": "Eastern Time (US & Canada)",
        "description": "You will come to Durham, North Carolina. I will sell you some records then, here in Durham, North Carolina. Fun will happen.",
        "default_profile_image": false,
        "profile_background_image_url": "http://a0.twimg.com/profile_background_images/9423277/background_tile.bmp",
        "statuses_count": 7579,
        "friends_count": 348,
        "following": null,
        "show_all_inline_media": true,
        "screen_name": "bullcityrecords"
      },
      "in_reply_to_screen_name": null,
      "source": "web",
      "in_reply_to_status_id": null
    },
    {
      "coordinates": null,
      "favorited": false,
      "truncated": false,
      "created_at": "Fri Sep 21 23:30:20 +0000 2012",
      "id_str": "249289491129438208",
      "entities": {
        "urls": [

        ],
        "hashtags": [
          {
            "text": "freebandnames",
            "indices": [
              29,
              43
            ]
          }
        ],
        "user_mentions": [

        ]
      },
      "in_reply_to_user_id_str": null,
      "contributors": null,
      "text": "Mexican Heaven, Mexican Hell #freebandnames",
      "metadata": {
        "iso_language_code": "en",
        "result_type": "recent"
      },
      "retweet_count": 0,
      "in_reply_to_status_id_str": null,
      "id": 249289491129438208,
      "geo": null,
      "retweeted": false,
      "in_reply_to_user_id": null,
      "place": null,
      "user": {
        "profile_sidebar_fill_color": "99CC33",
        "profile_sidebar_border_color": "829D5E",
        "profile_background_tile": false,
        "name": "Thomas John Wakeman",
        "profile_image_url": "http://a0.twimg.com/profile_images/2219333930/Froggystyle_normal.png",
        "created_at": "Tue Sep 01 21:21:35 +0000 2009",
        "location": "Kingston New York",
        "follow_request_sent": null,
        "profile_link_color": "D02B55",
        "is_translator": false,
        "id_str": "70789458",
        "entities": {
          "url": {
            "urls": [
              {
                "expanded_url": null,
                "url": "",
                "indices": [
                  0,
                  0
                ]
              }
            ]
          },
          "description": {
            "urls": [

            ]
          }
        },
        "default_profile": false,
        "contributors_enabled": false,
        "favourites_count": 19,
        "url": null,
        "profile_image_url_https": "https://si0.twimg.com/profile_images/2219333930/Froggystyle_normal.png",
        "utc_offset": -18000,
        "id": 70789458,
        "profile_use_background_image": true,
        "listed_count": 1,
        "profile_text_color": "3E4415",
        "lang": "en",
        "followers_count": 63,
        "protected": false,
        "notifications": null,
        "profile_background_image_url_https": "https://si0.twimg.com/images/themes/theme5/bg.gif",
        "profile_background_color": "352726",
        "verified": false,
        "geo_enabled": false,
        "time_zone": "Eastern Time (US & Canada)",
        "description": "Science Fiction Writer, sort of. Likes Superheroes, Mole People, Alt. Timelines.",
        "default_profile_image": false,
        "profile_background_image_url": "http://a0.twimg.com/images/themes/theme5/bg.gif",
        "statuses_count": 1048,
        "friends_count": 63,
        "following": null,
        "show_all_inline_media": false,
        "screen_name": "MonkiesFist"
      },
      "in_reply_to_screen_name": null,
      "source": "web",
      "in_reply_to_status_id": null
    },
    {
      "coordinates": null,
      "favorited": false,
      "truncated": false,
      "created_at": "Fri Sep 21 22:51:18 +0000 2012",
      "id_str": "249279667666817024",
      "entities": {
        "urls": [

        ],
        "hashtags": [
          {
            "text": "freebandnames",
            "indices": [
              20,
              34
            ]
          }
        ],
        "user_mentions": [

        ]
      },
      "in_reply_to_user_id_str": null,
      "contributors": null,
      "text": "The Foolish Mortals #freebandnames",
      "metadata": {
        "iso_language_code": "en",
        "result_type": "recent"
      },
      "retweet_count": 0,
      "in_reply_to_status_id_str": null,
      "id": 249279667666817024,
      "geo": null,
      "retweeted": false,
      "in_reply_to_user_id": null,
      "place": null,
      "user": {
        "profile_sidebar_fill_color": "BFAC83",
        "profile_sidebar_border_color": "615A44",
        "profile_background_tile": true,
        "name": "Marty Elmer",
        "profile_image_url": "http://a0.twimg.com/profile_images/1629790393/shrinker_2000_trans_normal.png",
        "created_at": "Mon May 04 00:05:00 +0000 2009",
        "location": "Wisconsin, USA",
        "follow_request_sent": null,
        "profile_link_color": "3B2A26",
        "is_translator": false,
        "id_str": "37539828",
        "entities": {
          "url": {
            "urls": [
              {
                "expanded_url": null,
                "url": "http://www.omnitarian.me",
                "indices": [
                  0,
                  24
                ]
              }
            ]
          },
          "description": {
            "urls": [

            ]
          }
        },
        "default_profile": false,
        "contributors_enabled": false,
        "favourites_count": 647,
        "url": "http://www.omnitarian.me",
        "profile_image_url_https": "https://si0.twimg.com/profile_images/1629790393/shrinker_2000_trans_normal.png",
        "utc_offset": -21600,
        "id": 37539828,
        "profile_use_background_image": true,
        "listed_count": 52,
        "profile_text_color": "000000",
        "lang": "en",
        "followers_count": 608,
        "protected": false,
        "notifications": null,
        "profile_background_image_url_https": "https://si0.twimg.com/profile_background_images/106455659/rect6056-9.png",
        "profile_background_color": "EEE3C4",
        "verified": false,
        "geo_enabled": false,
        "time_zone": "Central Time (US & Canada)",
        "description": "Cartoonist, Illustrator, and T-Shirt connoisseur",
        "default_profile_image": false,
        "profile_background_image_url": "http://a0.twimg.com/profile_background_images/106455659/rect6056-9.png",
        "statuses_count": 3575,
        "friends_count": 249,
        "following": null,
        "show_all_inline_media": true,
        "screen_name": "Omnitarian"
      },
      "in_reply_to_screen_name": null,
      "source": "Twitter for iPhone",
      "in_reply_to_status_id": null
    }
  ],
  "search_metadata": {
    "max_id": 250126199840518145,
    "since_id": 24012619984051000,
    "refresh_url": "?since_id=250126199840518145&q=%23freebandnames&result_type=mixed&include_entities=1",
    "next_results": "?max_id=249279667666817023&q=%23freebandnames&count=4&include_entities=1&result_type=mixed",
    "count": 4,
    "completed_in": 0.035,
    "since_id_str": "24012619984051000",
    "query": "%23freebandnames",
    "max_id_str": "250126199840518145"
  }
}

That’s from Twitter’s API reference. Now imagine writing a Java class to represent that response. Now imagine doing it again for whatever other insane responses Twitter wants to fling at you.

Jackson has what feels like hundreds of bizarre annotations and configuration options that let you control the translation process between a JSON string and your class, or get around problems like JSON responses suddenly getting new fields and mutually referential objects causing stack overflow errors due to infinite recursion. It’s a huge, complex library, and while there’s plenty of information out there, it can be hard to word a search query in the right way to find a solution to your specific problem. At my company, we briefly looked at migrating from org.json to Jackson, but it was going to be a massive amount of work to figure out the right magical incantations to make Jackson not break and then either go mark up all our database objects with the right annotations or write new view-model classes to use for JSON translation. And since none of us were Jackson experts, we had a steep learning curve ahead of us.

Imagine your car breaks down in the middle of the desert, but you happen to be right next to an abandoned auto shop with a set of books and tools. You don’t know the first thing about cars, other than where the gas goes, but you figure you’ve got a good brain, so you sit down with a book and start reading. After six months of alternately reading and tinkering on some of the forgotten wrecks left on the hydraulic lifts of this abandoned auto shop, you’re ready to start on your car. You eventually get it kind of working, though it can’t go above 37 mph, makes a weird stuttering noise as you drive, and you have to stop every 15 miles and go tighten some bolts, and also you just spent six months camped out in an abandoned auto shop in the middle of the desert. Here you’ve addressed the fundamental problem that your car is broken, and what you came up with is definitely better than a homemade car sail, but was it really worth that six months you spent in that abandoned auto shop?

The root cause, and how to fix it

The root cause of all these problems is Java’s static typing. Well, to be fair, it’s also Javascript’s dynamic typing. JSON is based on Javascript, so it’s perfectly fine with lists of mixed types, maps of string to whatever you please, and arbitrarily nested self-referential objects. Languages like Python, Ruby, PHP, and Clojure, which are dynamically typed and have a similar set of primitives to Javascript, are also fine with that, so you can trivially map any JSON object to a native data type in those languages.

Java is not fine with any of that. It demands lists of a single type and maps of strings to a single type, and it asks that you please tell it beforehand just how deep you’re going to nest those self-referential objects. You can cheat your away around this by declaring the type as Object, but working with Objects is tedious, ugly, and loses you the only advantage to having all these restrictions.

How can we fix this? Here a few features that would definitely help. They’re listed in order of how likely they are to be added to the Java language, starting at “Maybe in Java 26” and ending at “You’ve got a better chance of convincing your company to rewrite everything in Erlang”.

Optional dynamic typing

C# supports a type keyword dynamic. This makes a variable dynamically typed; the compiler basically pretends that it can be any type and supports any operation, and allows a runtime exception to be thrown if you do the wrong thing with it.

This would, at the very least, make the JsonWhatever strategy a lot easier: instead of needing a JsonPrimitive with all those conversion methods, just make the JsonObject and JsonArray store values of type dynamic.

This would also make Jackson’s problem with generic types easier to deal with: instead of binding to lists of Object or maps of string to Object, it can bind to a list of dynamic or map of string to dynamic.

This approach still gives up type safety, but it makes giving up type safety a lot easier to work with and less painful.

Union types

Languages like Haskell and ML support something called union types. Essentially you can define a new type which can hold a value of any of a given set of types.

Imagine that Java had a syntax like type JsonVal = Map<String, JsonVal> | List<JsonVal> | Integer | Double | String | Boolean | null to define a new type called JsonVal. This declares that a JsonVal can be any of those listed types, but no other. If our code ever tries to assign a FizzingWhizbeeFactory to a JsonVal, the compiler will complain.

This can buy us both freedom from casting and arbitrary nesting. The compiler has a list of possible types, so it doesn’t need us to tell it exactly which one a given value is; if we call a method on a JsonVal, it knows where it can look for that method, and if it can’t find the method anywhere, it can flag a compile-time error. It doesn’t need us to cast for it. And the JsonVal type is defined recursively: a JsonVal can be a Map<String, JsonVal> or a List<JsonVal>. This gets us the arbitrary nesting we need to accurately represent JSON.

There are other challenges to getting union types right in Java. For instance, if I have a union type type MyType = A | B and C is a subtype of B, can MyType be an instance of C? Can I cast an instance of C to B and assign it to MyType? Can I then downcast an instance of MyType which is assigned an instance of B to C if that’s what it really is? Or do we just do what generics did, throw the whole mess out, and say that any type which isn’t explicitly listed in the declaration of the union type is invalid? But if we could get union types in Java, it would go a long way towards solving our JSON problem in a type-safe way.

A Dedicated JSON type

This is the easiest option for programmers, and the most deranged from a language design standpoint.

Scala, Visual Basic, and Kawa (a dialect of Scheme on the JVM) have XML literals. In Scala, for example, you can write this:

val people = 
    <people>
        <person firstName="Ted" 
                lastName="Bundy" 
                gender="M" />
        <person firstName="Jeffery" 
                lastName="Dahmer" 
                gender="M" />
        <person firstName="Charles" 
                lastName="Manson" 
                gender="M" />
        <person firstName="Rasmus" 
                lastName="Lerdorf" 
                gender="M" />
    </people>

So in Java, we could just modify the language to let you write this:

Json someJson = {
    "people": [
        { "firstName": "Adolf", "lastName": "Hitler" },
        { "firstName": "Idi", "lastName": "Amin" },
        { "firstName": "Pol", "lastName": "Pot"},
        { "firstName": "Hiro", "lastName": "Mashima"}
    ]
}

The Json type can have a static method, Json.fromString, that inflates a string into JSON, and its toString method can be overridden to serialize a Json into a string.

Scala can sort of do XML literals as part of the normal language, because Scala’s design is already deranged. In Java, the language would have to be modified. But there is precedent for this: arrays in Java are already just random tacked on garbage with special syntax and special behavior that applies to no other part of the language. And Java’s principal use today is as a server-side web programming language; given how important JSON is to web programming, it might be worth modifying the core language to add this.

Still, this is the least likely of the options. Adding lambdas and method references was a massive deal, and those had already proven their worth in dozens of other languages over a period of almost fifty years. Java is not a language that likes revolutionary features; it prefers to wait until every language in the universe, even C++, has a feature, and then panic and squeeze something on top of the existing language features that’s just good enough to disunite the opposition.

Conclusion

Dealing with JSON in Java is quite painful, and it’s because JSON is inherently dynamically typed due to its heritage in Javascript’s native object type, and Java’s static type system is too rigid to accommodate it in an elegant way.

Since JSON is used for communicating between systems and components of a system, the difficulty of working with JSON in Java can affect how you choose to design your API responses. You might choose to favor flat objects over nested, as we used to do at my company. Instead of something like this:

{
    "room": {
        "name": "library",
        "carpetType": "Persian rug"
    },
    "killer": {
        "name": "Colonel Mustard",
        "weapon": "lead pipe"
    }
}

You might choose this:

{
    "roomName": "library",
    "roomCarpetType": "Persian rug",
    "killerName": "Colonel Mustard",
    "killerWeapon": "lead pipe"
}

In Java, with Jackson, the second form requires a single class with four strings. The first form requires a class with two subclasses, each with two strings, and probably a few annotations to make sure everything happens correctly when you bind from a string. So up front, the first form is more complicated to work with in Java, and this extra complexity just gets more pronounced as your JSON responses get more complex.

But using the second form causes other problems. Since JSON keys have no guaranteed order, you could send the response to a client like a browser and have it print with the keys all randomized, which makes the response much harder to read if you need to debug. If a client of your API only cares about the room, they have to loop over all the keys and find the ones that start with room instead of being able to just grab the room object. You can write that annoying code every time, or you can write some stupid little helper function that you have to document and test to loop over a JSON object and find all the keys with a certain prefix. You’ll probably also have to duplicate this function across various code bases in your stack. This tradeoff might be worth it if you just have a bunch of Java code communicating (e.g. an Android app and a Spring API), but if you have other languages in your stack, like Javascript or Python, you’re forcing them to conform to the lowest common denominator.

If you’re writing web code nowadays, JSON is a fact of life, and it’s a great transmission format: it’s simple, readable, and powerful, easy to parse, able to represent almost any kind of data in human readable format. It’s elegant and easy to use, a big improvement in a lot of ways over RPC formats like COM and CORBA, unreadable binary formats, and XML. It’s a travesty that Java has such terrible support for something so elegant, widespread, and fundamental.

Web Programming: A Prophecy

Over the course of the 20th and 21st Centuries, a lot of jobs that used to require human labor have been turned over to robots. Factory jobs are going to robots. Mining jobs are going to robots. Fighter jet and spy plane pilot jobs are going to robots. Retail jobs are going to robots, with self checkouts and restaurants like Eatsa.

I used to believe, smugly, that programming was too complex and difficult to ever be taken over by robots. When I read an article by some business commentator that claimed programming jobs were going to be this generation’s equivalent of the high-wage factory jobs of the 1950s and 1960s, I scoffed. Until I actually did some of what passes for programming in industry.

It’s pretty much all been lamented before: modern programming in a corporate environment is mostly writing little bits of glue code to combine a bunch of libraries written by some distant genius. Mike Taylor’s version was the first version of this lament that I saw, back when I was still in school and under the impression that industry programming was at least a little bit like personal or school programming. But I’m not here to lament. I’m here to make a prophecy.

In my current job, I am a full-stack web developer. I write the server-side code. I write the client-side code. I write the HTML and CSS. I create the database schema and write the SQL scripts to update the database. Is it complex and difficult? Kind of. But it’s complex and difficult because I have to have passing familiarity with Java, Javascript, HTML, CSS, and SQL, and work through all the amazing ways they don’t fit together well. My part in the whole mess is to drop in fairly simple, monotonous bits of boilerplate code to grab things from the database and insert them into an HTML document. Then I manipulate another mess of frameworks and libraries to show it all nice and pretty and make dynamic behavior on the page. Between the server-side framework, the Javascript framework, Bootstrap, and various other libraries, about 90% of the work is done by magical frameworks that just use little spludges of repetitive code I write to get that last 10% of the way towards becoming a web app.

I find it extremely hard to believe that there’s no way to automate that last 10%. There has to be a tool that makes it possible for someone to drop in those repetitive little globs of boilerplate just the way they like, and get a web application out of it.

As a full-stack web developer, I also offer some advice on design, UX, and business needs. But I’m not a designer, a UX pundit, or an expert on the business needs. My degree is in computer science. My expertise is programming. But there’s damn little programming in the code I write, and nothing I’d call computer science. The main hurdle is the delicate little interconnections between these rickety, rigid frameworks, the brittle wooden bridges I have to cross by writing repetitive boilerplate. But, per hypothesis, there has to be a way to create a tool that automates that repetitive boilerplate. Without that, there’s no programming left in creating the sort of custom line of business app that I work on. You could get together a designer, a UX pundit, and an expert on the needs of the business, and have them use this tool to create whatever web app they want. There are actually already tools that are close to this. Squarespace. Drupal and Joomla. WordPress is a little closer to full-stack web development, but I see job ads for “Wordpress programmers” rather than “PHP programmers” or “Front end programmers”, which suggests to me that WordPress is a specialized tool. As I was writing this I discovered WebML, which already sounds pretty close to what I was envisioning.

Once I realized that, I saw how right Mr. or Ms. Business Commentator whose name I’ve forgotten was in his or her comparison of software developer jobs to factory jobs. Right now, tools like Squarespace and Joomla still have certain limits, and there’s cachet in having a custom web app, so everyone and their dog needs a full-stack web developer to make one for them. Trucking companies need web apps; banks need web apps; government agencies need web apps; churches need web apps; even the Amish probably have a full-stack web developer on their payroll to make an Amish web app for them. (Job perks include hot apple Brown Betty in the kitchen, bonuses paid in handmade furniture, and a company carriage for the commute.) But once this magical tool whose future existence I’ve postulated comes to pass, none of these people will need full-stack web developers. Except the Amish; they need someone who’s okay being in a building with electricity every day. But everyone else will just have business experts make the web app. This might be the product manager, or the head of the department that’s going to use the web app, or the lady down the hall who knows Excel and VBA really well and consequently always gets stuck doing everyone’s spreadsheets. But there’s no reason to bring a programmer into the company and pay them a bunch of money and have them learn the business requirements when you can just have someone who already knows the business requirements use the tool to build the web app.

This doesn’t mean that all programming jobs will suddenly disappear. There will still be programmers. You can’t have a business person build something with a prepackaged tool when you don’t have any business requirements and no one’s ever built anything like it before. Google-scale stuff isn’t going anywhere. Google-level programmers aren’t going anywhere. But people like this guy are going to lose their jobs, or they’ll be slotted in at a lower pay grade to build web apps with the new tools. Even people who aren’t like that guy, people who don’t obstinately hold it as a personal tenet that no one should know more advanced technical subjects like Big O and data structures and we should all just concentrate on understanding business requirements and spelling Dijkstra’s name right, will lose their jobs in droves. They’ll lose their jobs because they’ll have spent careers working on simple little full-stack web apps built on existing frameworks instead of on Google-scale stuff, and suddenly businesses won’t need that anymore, or at least they’ll be paying a lot less for it.

So how long do we, the generic full-stack web developers working on dinky little web apps, have before the apocalypse? My guess is no more than ten years. The tools are already getting there, and they keep advancing. With single-page apps and faster connections, users are already being trained to expect more lag in web apps, so the kind of performance tuning that an experienced developer does will become less relevant. And businesses would love to be able to fire a bunch of expensive programmers and squeeze a little more work out of the business people. They’d love to be able to stop stocking soda and Terra chips for us arrogant, overpriced keyboard jockeys.

A few questions

At this point, you might be thinking my prophecy is wrong and the era of anyone who knows Javascript and one server-side framework being able to find jobs anywhere at relatively high salaries will last forever. And you might be right. The odds are against it, but you might be right.

However, I’m going to try and preemptively address a few questions I can see people having about this prophecy.

Won’t programmers just switch to programming the tools?

No. Well, yes, some of them will. But there aren’t nearly enough jobs working on the tools for everyone who now works as a generic full-stack web dev to get one. The world only needs two or three competing tools in each space, but even if a dozen more manage to scrape by, the scale won’t match the number of people currently working on dinky custom web apps alone for non-tech companies. Think about it: have you ever written a framework? And if you have, did anyone use it? Even though it sometimes feels like there are new frameworks for Javascript coming out on a daily basis, only about three or four actually get widely used, and most framework creators don’t get paid to work on them. The ones that do get used, the ones whose creators do get paid to work on them, are the ones like Angular and React that are backed by large tech companies.

Writing the tools will be the kind of Google-scale programming that will hang on after the Twilight of the Generic Full-Stack Web Devs. But for every former generic full-stack web dev who switches to working on tools, there will be at least ten who get shut out because of significantly lower demand for programmers, and probably another five who switch to using the new tools to do their own jobs at a significant pay cut just to stay employed. Companies in all industries needing to have their own custom web app and hiring a generic full-stack web dev to glue together some frameworks for them has created a supply of generic full-stack web devs that won’t be nearly as much in demand once business people can make the dinky custom business app.

Isn’t programming too complex and difficult to automate like you describe?

I hope if you’ve done the kind of generic full-stack web development I’m talking about here, you know that’s not true. The frameworks already do most of the work. Disorganization and the dismal state of web standards is the only thing holding us back from a single isomorphic client/server framework that lets us build monolithic web apps in a single language the way we used to build desktop apps. And with Node.js, client-side MVC frameworks, and Bootstrap, we keep getting closer to that one framework to find them all and in the DOM bind them.

There are other kinds of programming that I don’t foresee being automated so easily: hardware programming, big data, machine learning, cryptography, and of course the tools such as compilers, IDEs, frameworks, servers, and databases. Programming those things is complex and difficult, and programmers working on those things will continue to have lucrative careers. This prophecy will largely affect people not working in the tech industry, but working on web apps (and possibly mobile apps) in other industries.

Didn’t this already fail?

There have been various attempts to create tools that business people could use to make custom applications, e.g. CASE tools. Many failed. But one, COBOL, turned out to be one of the most successful programming languages of all time.

It’s possible that efforts to create the super tool I’m prophesying here will also fail. But I think modern day has one big thing that makes such a tool more likely to thrive: Google. Older tools were expensive proprietary things, and you had to read a gigantic manual to figure out how to use them. But nowadays, if you need help, you can Google for answers, or you can ask a question on a Stack Exchange site. And there will probably be blogs and tutorials and help guides for the tools, and maybe even public online documentation by the manufacturers, the way Oracle does with Java.

If you’re a busy business expert trying to make a web app, this makes a huge difference: instead of having to spend hours reading through a giant manual and days figuring out the best way to use what your tool gives you, you can just Google for an answer and copy it verbatim, like programmers do.

I’m a generic full-stack web developer and I believe in your prophecy. What should I do?

As a prophet of doom, I sadly don’t have any constructive suggestions for you, other than to consider where your career might go after this doom comes to pass.

If you enjoy the technical side of things, you can try to become an expert in one of those other areas of programming that won’t be wiped out in this apocalypse. If you find cryptography or machine learning too imposing, go for stuff like dev ops and database adminstration. Or if you’re more of a front end person and you like design or UX, you could head in that direction.

Alternately, if you like business requirements and management, like this guy, go full speed in that direction. I don’t, so I won’t be joining you.

What are you planning to do once the prophecy comes to pass?

Once my career in generic full-stack web development dries up, I’m going to take a few months to backpack around Europe and find myself. In a small Italian village on the Amalfi coast, I will rescue a stunningly beautiful woman from three drunken Sigma Chi boys on Spring Break from Penn State. The only daughter of the owner of a struggling trattoria, she will have the heart of poet and the sensitive wit of a Regina Spektor lyric. Our love will blossom like the last flowers of June, and after months of sexual tension and exquisite trattoria food, we will finally consummate our relationship. But the next morning, I’ll receive word that my beloved Albanian bulldog, Bullard, has died. Stricken with grief, I will weep tears of manly remorse, and rue that I was not at Bullard’s side in his final hours. I will return to the United States to further weep, and to visit his grave under the shade of a poplar tree in my parents’ backyard. But in the ashes of this tragedy a fire will kindle within me, and I’ll start a popular YouTube channel in which I talk about the wonders of Italy and Albanian bulldogs while wearing an oilskin duster in burnt sienna, a color that brings out my eyes. As my channel gains subscribers, I’ll start a companion blog and podcast and soon become a widely recognized blogger and YouTube celebrity.

So you see, I’ve got this all figured out. The end of generic full-stack web development will ultimately be a good thing for me. Maybe it can be a good thing for you too.

Every Freaking Tech Job Ad Ever

We’re looking for a superstar COBOL ninja to join our team! Applicants must have at least 40 years experience with COBOL. COBOL.NET experience preferred. You’ll spend every day hacking away on our exciting, disruptive legacy banking product, making it more robust and scalable as well as more exciting and disruptive. We are an exciting, young, hip, radical, groovy team of hackers, rockstars, ninjas, and wizards who love tech, startups, and tech startups, as well as disruption, excitement, and disruptive excitement, particularly in tech! We’re changing the face of banking by using the latest, most exciting technologies to disrupt all the old ideas about storing money and replace them with exciting, disruptive new ones! So come join us and remake the face of banking by making hacky temporary workarounds to problems with our mainframe OS that were fixed thirty years ago by upgrading to Windows 3.1!

Requirements:

  • 40 years COBOL experience.
  • Experience working with decimal computers
  • Teletype experience
  • Experience with mainframes
  • Experience with banking
  • Experience with banking mainframes
  • Experience with exciting, disruptive developments in legacy banking mainframes
  • PhD. in Computer Science, Computer Engineering, or Nucular Engineering
  • 10+ years software experience
  • Must be under 24 years of age
  • Must be able to lift up to five grams
  • Great team player who loves working in teams and can’t do anything alone. Anything.
  • Be a hip, radical, groovy ninja who’s excited about disrupting banking with disruptive legacy mainframe systems

Nice to have:

  • Experience with TCL, MUMPS, or RPG
  • Experience with Visual Foxpro
  • COBOL.NET
  • Dataflex experience
  • Juggling three or more balls
  • Knowledge of wilderness survival
  • Willing to spend business trips living in a van down by the river
  • A pulse

Introducing MonkeyType.js!

When I needed to choose a Javascript framework for a work project, I put a lot of thought into it. I tried Angular, Angular 2, Ember, React, Redux, Backbone, Dojo, Vue, Mithril, Marionette, CanJS, Knockout, Polymer, and js_of_ocaml. None of them worked. They all just felt wrong.

What I eventually realized is that these frameworks felt wrong because they don’t use the right abstraction. Modeling web apps as MVC, MVVM, MVP, MVT, MVASDFASDHSFGSDFGHQRTQRTAFGLKJNNGJSKL, these are all the wrong abstractions.

That’s why I decided to create my own Javascript framework. Instead of using broken models of abstraction like Model-View-Controller or Model-View-View Model, it uses MOT: Monkeys On Typewriters. And I called its name MonkeyType.js.

I know what you’re thinking: what does a stupid front-end engineer who took on $157,000 in student loans to follow his passion and get a degree in Lithuanian literature with a minor in Cloud Shape Interpretation Studies, laid around on his mom’s couch for five years, got an online Master’s degree in Motivational Speaking, worked as a tutor for left-handed children, decided to enroll in divinity school, dropped out after accruing another $34,000 in debt, lucked into a plum job at a grocery store making $15 an hour where his only real duties were printing signs for the manager to stick in the break room, learned HTML, CSS, and Javascript after the manager asked him to make a website for her side business selling purebred blue fawn Azawakhs (also known as the Tuareg Sloughi according to the AKC), found out you could make money with this Javascript stuff, and finally moved out of his mom’s house, know about designing a framework? Way ahead of you! MonkeyType.js is based firmly on MATH! It uses the celebrated Infinite Monkey Theorem, described by famed French measure theorist Félix Édouard Justin Émile Borel in 1913, to created a model of UI firmly based on MATH!, with grounds in MATH!, concretely resting on a foundation of MATH! to create a uniquely appropriate abstraction coming, like all the greatest and most appropriate abstractions, from MATH!

The Infinite Monkey Theorem states that given infinite monkeys banging the keys on infinite typewriters for an infinite length of time, the probablity that Hamlet will appear somewhere on one of the monkey’s typewriters is equal to 1, i.e. one of the monkeys almost surely produces Hamlet. MonkeyType.js takes this theorem to heart and uses it to create an abstraction for UI programming that’s so wrong, it couldn’t be more right. The best part about a client-side framework based on the Infinite Monkey Theorem is that the Infinite Monkey Theorem is MATH!, and uses terms like almost surely in confusing ways that only make sense if you know MATH!, and are otherwise just confusing and misleading! You have to be really smart to know MATH!, so you also have to be really smart to understand MonkeyType.js. No dummies who only know how to make solid, functional user interfaces that are intuitive and aesthetically pleasing; only the best front-end engineers need apply to use MonkeyType.js, because with MonkeyType.js, you have to be able to do all of those things, and also know MATH!, and not dumb kiddy stuff like fractions, but REALLY HARD MATH! that famous French guys invented. Everyone knows French stuff is really cool and hard to understand; the other night I watched Amelie, and I had no idea what was going on, but it was French, so I can just go up to people and say “Hey, have you seen Amelie? It’s French” and they know how smart and cool I am to be able to understand French stuff. And then they’re even more impressed when they find out I’m the inventor of MonkeyType.js and I understand MATH!

A typical MonkeyType.js program has two parts: the monkeys, and the typewriters. The monkeys represent the data. They’re sort of like the model in other frameworks, except with more MATH!, and more French. The reason monkeys are a better abstraction for data than a model is that models are stupid and for babies. When I was a stupid baby I got this model of an airplane for my birthday. It was wood. I kept trying to build it and getting splinters in my fingers that really freaking hurt. Finally I got a splinter under my fingernail, and it hurt so bad that I got mad and threw that dumb airplane model across the room and it broke, and my mom had to come in and hug me. I wouldn’t stop crying, so finally my mom had to take me to the zoo to make me quiet down. I didn’t stop crying until I saw the monkeys. I was only fifteen years old, but I kept that love of monkeys my whole life, and I never cried again, until I got home and stepped on a broken piece of the airplane model and got a splinter in my foot.

The typewriters represent the appearance, sort of like the view in other frameworks, except with more MATH! Typewriters are way better than views. I hope I don’t have to explain this one; views remind me too much of 1985’s A View to a Kill, and then I spend all day mumbling “Dance into the fire” under my breath and my smarmy co-workers who pretend to be my friends but secretly laugh at me because they all hacked into the Pentagon’s computers when they were ten years old and I didn’t learn to program until I was forty-seven start talking behind my back.

The monkeys pound on the typewriters to create the website. Unlike other dumb frameworks that have something like a controller standing between the model and view, MonkeyType.js lets the monkeys pound directly on the typewriters. Here’s where the really cool part comes in. Obviously we can’t have infinite monkeys and infinite typewriters on a modern computer with finite memory, so MonkeyType.js uses a clever optimization called a monkey barrel. Instead of infinite monkeys, the monkey barrel contains about twelve or thirteen monkeys that it keeps recycling. Every time you ask for a monkey, the monkey barrel gives you a monkey. It might be the same monkey you got last time. It might be a different one. Since one monkey’s as good as another monkey, with this approach, you effectively have infinite monkeys. The typewriters are stored internally in a circular buffer, which gives the impression that there are infinite typewriters, but it’s actually just, like, twelve or thirteen typewriters on a loop. And to create the impression of infinite time, MonkeyType.js is coded in an excruciatingly dumb way so that all the pages hang for at least a minute before doing anything, and when they do do something, they just load a spinner that swirls for another minute or two before the actual page you were looking at finally loads. This impression of infinite time is increased even further in production enviroments, where hundreds of trackers and advertising services also get to load before your content. This makes MonkeyType.js the perfect framework for rich media sites like Buzzfeed, Huffpost, and ZergNet, where every page is so obviously loading five hundred ads before it even thinks about delivering the content you asked for, because the advertising is actually the content and the content was actually just a bait-and-switch to get you to look at ads. But who cares, because we deliver ads with a firm basis in MATH!

In conclusion, where other JS frameworks give you broken abstractions based around dumb ideas like being easy to learn, easy to write, or easy to use to create great UIs, MonkeyType.js gives you abstractions based on MATH! and awesome stuff like monkeys and typewriters. Who cares if it causes memory leaks in your browser? It’s MATH! I hope you’ll consider MonkeyType.js for your next project, and finally learn what a real abstraction looks like.

A Paean for the Proxy

These verses were composed in trying times. Upon a summer’s day in August, the Year of Our Lord Two Thousand and Sixteen, it suddenly became indispensable that I should write a small bit of boilerplate, many times scrawled carelessly upon my IDE window, and never once committed to memory, with the end of orchestrating that the Play Framework, on which firmament rested my code, accomplish some minute but needful end, I remember not what. Raising anchor and casting off into the wide seas of Tela Totius Terrae, I came upon a small inlet at Stack Overflow which seemed probable to house the lore I sought. Clicking upon it, I chanced to provoke the wrath of the Proxy, the gatekeeper which guards our office network. Three times I gave the pass-word, and three times the Proxy denied me, and at last, defeated, I turned to other matters.

In the proximal hours of this mortifying rebuff, I anathematized the Proxy thrice over, and thrice maledicted it, doubling the trials it had lain upon my back like three lashes of a cat-o-nine-tails. But in the following hours I reflected on the happenings, and it came to me that I was like a child before a good shepherd, squealing to be let beyond the pasture, and the Proxy was the good shepherd, the pass-word prompt his crook across the gates, an inexorable guardian, keeping my toe upon the path of truth and my ken upon the course of righteousness. And upon that realization I composed these verses, which I dedicate to the Proxy, in admiration, may it ever watch over us sinful software developers, and keep us from temptation.


Oh Proxy, wondrous, on the net
Above us like the moon, so silvery grey
By blocking sites you never let
We software developers go astray.

You keep us, and guard us, and ensure that we
are never tempted from our tasks
You maintain our morality
But sometimes relent, if we ask.

Oh Proxy, how glorious are thee
A shepherd for this software flock
Forgive us if we question thee
On why you chose Stack Overflow to block.

A Comparison of Four Algorithms Textbooks

At some point, you can’t get any further with linked lists, selection sort, and voodoo Big O, and you have to go get a real algorithms textbook and learn all that horrible math, at least a little. But which book? There are tons of them.

I haven’t read every algorithms book out there, but I have read four of them. Maybe my experience with these four can help guide your decision. The four books are Algorithms, by Dasgupta, Papadimitriou, and Vazirani (hereafter called Dasgupta); Introduction to Algorithms, by Cormen, Leiserson, Rivest, and Stein (hereafter called CLRS); The Algorithm Design Manual, by Steve Skiena (hereafter called Skiena); and The Art of Computer Programming, Volumes 1-3, by Donald Knuth. I’ll do a five-point comparison, going over the prose style, code use, mathematical heaviness, breadth and depth of topics, and position on the continuum between theoretical and practical of each book.

There’s one thing you should know before I start: Dasgupta is available for free online, while Knuth and CLRS aren’t (well, they probably are, but not legally). So that might make your decision for you. I haven’t been able to determine if Skiena is legally available online. The book is posted in either PDF or HTML on a few legit-ish looking sites, but Skiena’s own page for it doesn’t mention anything about it being freely available, so proceed with caution.

Does it have to be one of these four?

Not at all. There are lots of good algorithms books that I’ve never read. You can even learn a lot by just reading CS.SE and Wikipedia. But these four are the ones I’ve personally used. Many people like one of these four, but they do reflect my taste and biases. But even if you decide to go with a different book, this overview might at least tell you what features to notice.

Prose style

A good algorithms book will usually explain its topics in three ways: with a natural language prose explanation, with some kind of implementation code, and with mathematical notation. When you’re starting out, the prose is usually the most important part, because it takes you from “What the hell is a binary search?” to “Got it, how do I write actual code for a binary search?”

For me, the winner on prose style has to be Knuth. He’s just a masterful writer. You might have to read some of his sentences twice, but that’s only because he gets across twice as much information as a lesser writer. Knuth’s prose explanations got me to understanding on several topics I’d despaired of ever getting with other books, like B-trees and merge sort.

Skiena is also an excellent prose stylist. Where Knuth is elegant and flowing, like the John Milton of algorithms, Skiena is direct and sharp, like the Ernest Hemingway of algorithms. Knuth and Skiena are good at different things: Skiena is good at finding a simple, direct way to explain things which are traditionally made more complicated than they need to be. His overview of Big O is amazingly clear and simple, as long as you have at least some memory of calculus. On the other hand, some topics are inherently really complicated, and this is where Knuth shines: he’ll give you several different ways to view a complicated topic, and chances are at least one of them will work for you.

Knuth is much quicker to jump to math, while Skiena mostly eschews math and tries to keep everything intuitive. We’ll discuss that more in the section on mathematical heaviness.

Dasgupta has a pretty good prose style too, more patient and beginner-friendly than either Skiena or Knuth. Sometimes, I found Dasgupta too verbose and felt the authors belabored the obvious too much, but other times that was an asset rather than a weakness (e.g. in the material on Fast Fourier Transforms).

CLRS probably has the weakest prose style of these four books. It’s closer to the standard math text style, written with a formal and distant air, sometimes favoring precision over clarity. It’s strictly functional, but tolerable if you already have some understanding of the topic.

Code use

Knuth’s big weakness is code. He uses two different notations, pseudocode and MIX assembly language, his imaginary assembly for his imaginary computer. I find both of them extremely difficult to follow.

The problems with MIX should be pretty obvious: it’s assembly language, so it’s just hard to read. Not only that, it’s pretty different from the MIPS, ARM, or x86 assembly that modern readers might have seen. It’s designed to be run on either a decimal or binary architecture and assumes a six-bit word. Knuth put a ton of effort into creating this imaginary machine and its assembly language. Since it’s made up, MIX assembly is still technically pseudocode; but MIX is to pseudocode as Quenya is to pseudo-languages. Newer editions of TAoCP use MMIX, which was redesigned to reflect modern assembly languages. I haven’t seen it yet, but I imagine it’s still hard to read since it’s assembly language.

Knuth also uses high-level pseudocode, but I find that hard to read too because it’s organized like an unstructured language with goto statements. If I were planning to implement the algorithms in an unstructured language, it would probably be fine, but I’ve always found that there’s a nontrivial translation process between Knuth’s unstructured pseudocode and structured pseudocode suitable for implementation in a modern language.

CLRS and Dasgupta both use high-level pseudocode that resembles Pascal, although not too slavishly. CLRS expresses some things as if they were methods or fields of an object, in a semi-object oriented way. Skiena also does some of this, but in addition he uses real C code.

A lot of modern books use C or Java throughout, which might be more to some people’s tastes. These books reflect my taste, and I like my algorithms language-independent. I don’t mind how Skiena uses C—he uses it mainly to show how to implement something when there’s a significant gap between theory and practice. But I’m glad he stuck to pseudocode for the most part.

Mathematical Heaviness

In this category, going from heaviest to lightest, we have roughly Knuth > CLRS > Dasgupta > Skiena. Dasgupta and Skiena are pretty close, while there’s a significant gap between them and CLRS. There’s a minor gap between CLRS and Knuth, and it really depends on the topic, but CLRS just didn’t take on as many inherently mathematical topics as Knuth did (especially in Knuth’s Volume 2, Seminumerical Algorithms, where he deals with random number generation and efficient arithmetic).

In general, Dasgupta is geared towards people (like students) who just recently learned a little math and are going to learn a little more, but haven’t internalized it all yet. Skiena is geared towards people (like working programmers, or graduate students) who’ve learned some math and forgotten a lot of it. Skiena is also student friendly, but he stresses practicality more than anything, and sometimes, in practice, you gotta do some math.

CLRS is also for students, but it’s more at the graduate student level, especially the later “Selected Topics” chapters that can sometimes get pretty heavy. Knuth seems more geared towards graduate students near the end of their Ph.Ds and practicing researchers.

All four books require you to know some combinatorics and a little number theory, and to have at least a vague memory of limits and function growth from calculus. That’s about it for Skiena and Dasgupta, though Skiena seems to expect you to have internalized the math a little more than Dasgupta does. The basic chapters of CLRS are just a little beyond Skiena in math background, but some topics get quite heavily mathematical. Knuth gets pretty heavily mathematical in almost every topic.

I’ve been conflating two things in the discussion so far, since they’re sort of linked: actual knowledge of theorems and axioms in mathematics, and the ability to read mathematical writing and proofs and follow an author’s reasoning (what’s often called “mathematical maturity”). Knuth requires the most technical knowledge of mathematics of the four here, and he’ll also stretch your reading ability pretty far; but CLRS will also stretch your reading ability, and Dasgupta is no cakewalk either, although I do think Dasgupta qualifies as a picnic. Skiena doesn’t have any proofs, but he kind of sneaks in proof-like things with his “war stories”, which usually read quite a bit like proofs, extended with discussion of some of the approaches he tried that didn’t work out.

If you’re getting your first algorithms book and you’re a CS undergrad, Dasgupta or Skiena is easiest to follow. CLRS will stretch you a little, but it’s manageable. If you’re a math or physics undergrad, CLRS shouldn’t be too hard and Knuth might be doable.

Breadth and Depth of Topics

Dasgupta is the loser here; not only does the book cover fewer topics than the others, the topics it chooses to cover are poorly organized and somewhat eccentric. I’m not sure why the authors threw in the awful chapter on quantum computing at the end; it’s totally incomprehensible unless you already understand quantum mechanics, which is no mean feat.

CLRS has the best balance of breadth and depth: it covers basic data structures; some more advanced ones like red-black trees, B-trees, and the union-find data structure; algorithmic paradigms like greediness and dynamic programming; graphs, shortest paths, and network flows; sorting; amortized analysis; and an assortment of other topics such as number theoretic algorithms, cryptography, linear programming, string matching, and NP completeness.

Skiena covers about the first third of CLRS, but he does a lot more with NP complete problems and how to design approximation schemes for them than CLRS does.

Knuth, of course, is the master of depth. Volume 1 covers math background and fundamental data structures; Volume 2 covers random number generation and arithmetic; Volume 3 covers searching and sorting, going through various sort routines and some more advanced data structures, such as B-trees, as well as developing the whole theory behind hash tables and how to choose hashing functions and table sizes; and Volume 4 covers combinatorial algorithms. Volume 4 is split into three subvolumes; right now, only Volume 4A has actually come out.

If you want to get just one book, I would get Skiena or CLRS. They include all the most important topics both in practice and for undergraduate classes. Skiena, as a bonus, even includes some computational geometry material, in case you’re doing video games or computer graphics.

Theoretical vs Practical

Skiena is the most relentlessly practical of this bunch. Knuth was actually pretty practical at the time Volume 1 came out, but he became less and less practical as time went on because people stopped writing things in assembly or using tape drives. Both give you implementation tips to make your code perform better, although Knuth’s can be hard to capitalize on since you’re not writing in assembly.

CLRS and Dasgupta are both theoretical in the sense that they mostly ignore the computer. Everything in CLRS will work, but sometimes their approach is too “straight from the theory”, as I discovered when I implemented Karp-Rabin according to their discussion and put it on Code Review.SE after struggling with numerous overflow-related bugs, only to have someone suggest a different approach that rectified all the performance issues and handled overflow elegantly.

Dasgupta and CLRS are both still good books, and if you’re just starting out learning algorithms, don’t get too caught up on this issue. Write your code in Python or Ruby, some quick and easy language that also ignores the metal. You’ll get the idea of the implementation, and you can deal with the issues around implementing it in some other language later. (Python even looks like the pseudocode in CLRS.)

Conclusion

I probably use CLRS the most, because of its breadth of coverage and because its pseudocode is so much easier to follow than Knuth’s. If I don’t understand the concept or the mathematical underpinnings, I’ll go to Knuth for the deep discussion, and the excellent prose style.

I just got Skiena recently, but in the future, I expect him to usurp CLRS somewhat. He covers most of the same material, but his prose style is more direct and his approach is more practical. Skiena is excellently organized and has a catalogue of problems and their algorithmic solutions in the second half of his book, good for browsing when you’re stumped.

I don’t use Dasgupta that much. Dasgupta was the text for my undergrad algorithms class, and while it was good in that capacity, it’s not really a book that continues to be useful once you’re past the first-semester course, mainly because of the lack of breadth in coverage and the eccentric organization and choice of topics.