JSON in Java: A Case Study in Pain

Dealing with JSON in most dynamically typed languages is dead simple: you have a function that can inflate a string into the language’s native hash map type, and you have another function that can serialize the language’s native hash map type into a string of JSON. These languages all support nested maps, some sort of list, and strings, numbers, and booleans well enough to automatically map from JSON’s simple data types into the language’s native types.

Dealing with JSON in the emphatically statically typed, the holdover that’s still hanging around like a bad hangover, the one, the only Java will eventually make you wish you were dead.

I’ll be up front with you: this isn’t going to be a deep or insightful read. It’s mostly going to be a rant, with lots of hate for Java and static typing, but mostly static typing as seen in Java, because I’ve never tried to work with JSON in languages like Haskell with more sophisticated type systems, so you Haskell lovers are free to stay smug in the knowledge that you work with a better class of type system, nay, not a class in sight, but rather a category. If you love Java and static typing, you might want to tune out, because I am not at all interested in hearing for the hundredth time how dynamic languages force you to write unit tests for the stuff that static languages check at compile time. This is also not meant as a top-to-bottom indictment of static typing; it’s an examination of how static typing utterly fails to solve one specific problem in an elegant way.

To start with, I’m going to present the problem. Then I’ll examine what I find to be the root causes of this problem, and then rant a little about some features Java will never get that I think could help this problem.

The Problem

There are two major things you need to be able to do with JSON in your programming language: turn a string of it into your language’s native data types so you can work with the data, and turn your language’s native data types into a string of it so you can send it across a network.

JSON—Javascript Object Notation—is based on a restriction of Javascript’s native data types. Looking at the standard, we see that it supports objects, arrays, strings, numbers, booleans, and null. On the face of it, it seems like we should be able to represent these types with native Java types: maps for objects, lists for arrays, and Java’s string, number, boolean, and null for those types. But maps and lists in Java have to be restricted to a single type, so to represent JSON values like {"status": true, "message": "Status normal"} or [23, 'hello', false], we have to use a Map<String, Object> and a List<Object>.

Unfortunately, this almost immediately leads to problems. Java lists and maps don’t support being stringified as valid JSON, so you have to write custom code to do it. Actually writing this code isn’t too hard as long as you remember that JSON supports arbitrarily nested lists and objects, so you have to make sure your serialization is applied recursively until you bottom out. But it has all the usual disadvantages that come when you write custom code for fundamental operations instead of getting it from a library: you have to test it, support it, and maintain it yourself, and it’s harder to bring into new projects since you can’t just let your build tool grab it from Maven Central.

Working with Objects in Java is rather painful; if you want to do anything with them, you have to cast them. This requires you, the programmer, to know what type every one of those objects actually is. Look at this rather ugly bit of code:

Map<String, Object> jsonMap = getJsonMap();
List<Object> jsonList = jsonMap.get("theList");
if ((Boolean)jsonList.get(2)) {
    Integer idNumber = (Integer)jsonList.get(0);
    User user = getUserById(idNumber);
    user.setPreferredGreeting((String)jsonList.get(1));
    user.save();
}

Not only is this code ugly, it also loses the much-vaunted type safety of Java. You could get runtime type errors if it turns out that the first item of jsonList is not actually an Integer or the second item is not actually a String.

We can solve the ugliness problem a bit by throwing out the built-in Java map and list and using a special set of classes to represent JSON data types. This is what org.json or Jackson and GSON’s tree structure use. Instead of a Map, we’ll use a JsonObject; instead of a List, we’ll use a JsonArray; and instead of using Java strings, numbers, and booleans, we’ll use a JsonPrimitive. We’ll make all of these classes implement the JsonType interface.

The JsonObject class will behave like a Map<String, JsonType>. The JsonArray will behave like a List<JsonType>. The JsonPrimitive will store its value as a string, or possibly an Object. It will have a set of three methods: getAsNumber, getAsString, getAsBoolean, and possibly corresponding isNumber, isString, isBoolean. We glossed over the extra complication that the Javascript/JSON number type can be either integral or floating point; we also glossed over the extra complication that JSON numbers don’t have a specified size, so what we get could potentially overflow an Integer or Double. We could handle those problems here either by making getAsNumber return a BigDecimal or making separate getAsInteger, getAsDouble, getAsBigInteger, getAsBigDecimal methods. Or we could decide that overflow will be rare enough that we’ll tolerate errors when it happens, and make getAsNumber return a Double.

With this new set of classes, we can rewrite the above example:

JsonObject root = getJsonMap();
JsonArray array = root.get("theList");
if (array.get(2).getAsBoolean()) {
    User user = getUserById(array.get(0).getAsInteger());
    user.setPreferredGreeting(array.get(1).getAsString());
    user.save();
}

It’s at least less ugly without all those casts and temporary variables, and since the types make it obvious that we’re dealing with JSON, we don’t have to name everything jsonWhatever. But we’ve still lost type safety; if we do array.get(2).getAsBoolean() and that item isn’t a boolean, we’ll still get a runtime type error. And these custom JSON types are inevitably less convenient to work with than the standard library data types. What options do we have if we want to iterate over a JsonArray? Does it implement Iterable<T>, which will let us use the “smart for-loop”, or are we stuck with the C-style for-loop? Can we get the entry set of a JsonObject and iterate over all its keys and values if we want to? Do these classes implement Collection<T>, making them streamable?

This solution with separate JsonWhatever classes is the quick and dirty way of solving the JSON problem in Java. Imagine that your car breaks down in the middle of the desert. You have a tarp, a roll of duct tape, and a long pole in the back, so you tape the tarp to the pole, tape the pole to your car, and you’ve got a sail. The JsonWhatever classes are the homemade car sail of JSON handling. It does nothing whatsoever to fix the fundamental problem, but it does kind of get you moving again, in a way that’s slightly better than walking or pushing the car.

Most high-powered JSON libraries in Java go with a more sophisticated approach, borrowed from ORMs such as Hibernate. The Java language really really wants you to specify everything precisely, categorically, and at a low level: this class has these exact five fields with this five exact names, and this one’s an Integer, and this one’s a String, and this one’s a NosehairInspectorFactory, and so on. So why not create a class that specifies what your JSON will look like, precisely, categorically, and at a low level?

This is what Jackson and GSON do. They include a class, called ObjectMapper in Jackson, that can take a normal Java object and turn it into a string of JSON, or take a string of JSON and a normal Java class and create an object of that class from the string. Here’s an example:

class MyObject {
    private Integer myId;
    private String myPreferredGreeting;
    private Boolean isHasBeenGreeted; // Note, this is what you get when you insist on starting all boolean names with "is".
    private MyOtherObject myOtherObject;
    private List<Integer> allMyBuddiesIds;

    // All that getter and setter nonsense that Java insists on...
}

Jackson can map between this JSON and an object of that class:

{
    "myId": 23,
    "myPreferredGreeting": "Whassup dorks?",
    "isHasBeenGreeted": false,
    "myOtherObject": {
        "myFavoriteDrink": "Rolling Rock",
        "myFavoriteFood": "Cheese pizza with the crust cut off",
        "myLifePhilosophy": "YOLO bros"
    },
    allMyBuddiesIds: [21, 33, 49]
}

You might be wondering exactly how it does that. The answer is magic. As previously mentioned, I hate magic. But Jackson is magic from top to bottom. Except when it’s not.

The basic idea, to give a little more detail, is that Jackson can parse a JSON string into some intermediate form (which may or may not be the tree model it also offers, its version of our JsonWhatever types above; I haven’t looked into Jackson’s source code far enough to know). From there it can use reflection, possibly assisted by annotations you put on your class, to look for a way to map from the JSON object to your class. If you’re familiar with dependency injection, it’s pretty much the same idea: Jackson looks for a constructor or some setter methods it can use to inject the values from the JSON into an object of the class you provide. Then we have a regular Java object, referred to in this context as a POJO (“Plain old Java object”) since it’s not a special JSON-whatever object.

This sounds amazing. And it is pretty amazing, for JSON with all primitive types and no nesting. Unfortunately, Jackson doesn’t play well with generic types, so you end up having to use raw types for maps and lists, bringing us back to our original problem where we have to know what types are stored in maps and lists and cast them when we extract them.

The other problem with this is that some APIs return JSON responses that look like this:

{
  "statuses": [
    {
      "coordinates": null,
      "favorited": false,
      "truncated": false,
      "created_at": "Mon Sep 24 03:35:21 +0000 2012",
      "id_str": "250075927172759552",
      "entities": {
        "urls": [

        ],
        "hashtags": [
          {
            "text": "freebandnames",
            "indices": [
              20,
              34
            ]
          }
        ],
        "user_mentions": [

        ]
      },
      "in_reply_to_user_id_str": null,
      "contributors": null,
      "text": "Aggressive Ponytail #freebandnames",
      "metadata": {
        "iso_language_code": "en",
        "result_type": "recent"
      },
      "retweet_count": 0,
      "in_reply_to_status_id_str": null,
      "id": 250075927172759552,
      "geo": null,
      "retweeted": false,
      "in_reply_to_user_id": null,
      "place": null,
      "user": {
        "profile_sidebar_fill_color": "DDEEF6",
        "profile_sidebar_border_color": "C0DEED",
        "profile_background_tile": false,
        "name": "Sean Cummings",
        "profile_image_url": "http://a0.twimg.com/profile_images/2359746665/1v6zfgqo8g0d3mk7ii5s_normal.jpeg",
        "created_at": "Mon Apr 26 06:01:55 +0000 2010",
        "location": "LA, CA",
        "follow_request_sent": null,
        "profile_link_color": "0084B4",
        "is_translator": false,
        "id_str": "137238150",
        "entities": {
          "url": {
            "urls": [
              {
                "expanded_url": null,
                "url": "",
                "indices": [
                  0,
                  0
                ]
              }
            ]
          },
          "description": {
            "urls": [

            ]
          }
        },
        "default_profile": true,
        "contributors_enabled": false,
        "favourites_count": 0,
        "url": null,
        "profile_image_url_https": "https://si0.twimg.com/profile_images/2359746665/1v6zfgqo8g0d3mk7ii5s_normal.jpeg",
        "utc_offset": -28800,
        "id": 137238150,
        "profile_use_background_image": true,
        "listed_count": 2,
        "profile_text_color": "333333",
        "lang": "en",
        "followers_count": 70,
        "protected": false,
        "notifications": null,
        "profile_background_image_url_https": "https://si0.twimg.com/images/themes/theme1/bg.png",
        "profile_background_color": "C0DEED",
        "verified": false,
        "geo_enabled": true,
        "time_zone": "Pacific Time (US & Canada)",
        "description": "Born 330 Live 310",
        "default_profile_image": false,
        "profile_background_image_url": "http://a0.twimg.com/images/themes/theme1/bg.png",
        "statuses_count": 579,
        "friends_count": 110,
        "following": null,
        "show_all_inline_media": false,
        "screen_name": "sean_cummings"
      },
      "in_reply_to_screen_name": null,
      "source": "Twitter for Mac",
      "in_reply_to_status_id": null
    },
    {
      "coordinates": null,
      "favorited": false,
      "truncated": false,
      "created_at": "Fri Sep 21 23:40:54 +0000 2012",
      "id_str": "249292149810667520",
      "entities": {
        "urls": [

        ],
        "hashtags": [
          {
            "text": "FreeBandNames",
            "indices": [
              20,
              34
            ]
          }
        ],
        "user_mentions": [

        ]
      },
      "in_reply_to_user_id_str": null,
      "contributors": null,
      "text": "Thee Namaste Nerdz. #FreeBandNames",
      "metadata": {
        "iso_language_code": "pl",
        "result_type": "recent"
      },
      "retweet_count": 0,
      "in_reply_to_status_id_str": null,
      "id": 249292149810667520,
      "geo": null,
      "retweeted": false,
      "in_reply_to_user_id": null,
      "place": null,
      "user": {
        "profile_sidebar_fill_color": "DDFFCC",
        "profile_sidebar_border_color": "BDDCAD",
        "profile_background_tile": true,
        "name": "Chaz Martenstein",
        "profile_image_url": "http://a0.twimg.com/profile_images/447958234/Lichtenstein_normal.jpg",
        "created_at": "Tue Apr 07 19:05:07 +0000 2009",
        "location": "Durham, NC",
        "follow_request_sent": null,
        "profile_link_color": "0084B4",
        "is_translator": false,
        "id_str": "29516238",
        "entities": {
          "url": {
            "urls": [
              {
                "expanded_url": null,
                "url": "http://bullcityrecords.com/wnng/",
                "indices": [
                  0,
                  32
                ]
              }
            ]
          },
          "description": {
            "urls": [

            ]
          }
        },
        "default_profile": false,
        "contributors_enabled": false,
        "favourites_count": 8,
        "url": "http://bullcityrecords.com/wnng/",
        "profile_image_url_https": "https://si0.twimg.com/profile_images/447958234/Lichtenstein_normal.jpg",
        "utc_offset": -18000,
        "id": 29516238,
        "profile_use_background_image": true,
        "listed_count": 118,
        "profile_text_color": "333333",
        "lang": "en",
        "followers_count": 2052,
        "protected": false,
        "notifications": null,
        "profile_background_image_url_https": "https://si0.twimg.com/profile_background_images/9423277/background_tile.bmp",
        "profile_background_color": "9AE4E8",
        "verified": false,
        "geo_enabled": false,
        "time_zone": "Eastern Time (US & Canada)",
        "description": "You will come to Durham, North Carolina. I will sell you some records then, here in Durham, North Carolina. Fun will happen.",
        "default_profile_image": false,
        "profile_background_image_url": "http://a0.twimg.com/profile_background_images/9423277/background_tile.bmp",
        "statuses_count": 7579,
        "friends_count": 348,
        "following": null,
        "show_all_inline_media": true,
        "screen_name": "bullcityrecords"
      },
      "in_reply_to_screen_name": null,
      "source": "web",
      "in_reply_to_status_id": null
    },
    {
      "coordinates": null,
      "favorited": false,
      "truncated": false,
      "created_at": "Fri Sep 21 23:30:20 +0000 2012",
      "id_str": "249289491129438208",
      "entities": {
        "urls": [

        ],
        "hashtags": [
          {
            "text": "freebandnames",
            "indices": [
              29,
              43
            ]
          }
        ],
        "user_mentions": [

        ]
      },
      "in_reply_to_user_id_str": null,
      "contributors": null,
      "text": "Mexican Heaven, Mexican Hell #freebandnames",
      "metadata": {
        "iso_language_code": "en",
        "result_type": "recent"
      },
      "retweet_count": 0,
      "in_reply_to_status_id_str": null,
      "id": 249289491129438208,
      "geo": null,
      "retweeted": false,
      "in_reply_to_user_id": null,
      "place": null,
      "user": {
        "profile_sidebar_fill_color": "99CC33",
        "profile_sidebar_border_color": "829D5E",
        "profile_background_tile": false,
        "name": "Thomas John Wakeman",
        "profile_image_url": "http://a0.twimg.com/profile_images/2219333930/Froggystyle_normal.png",
        "created_at": "Tue Sep 01 21:21:35 +0000 2009",
        "location": "Kingston New York",
        "follow_request_sent": null,
        "profile_link_color": "D02B55",
        "is_translator": false,
        "id_str": "70789458",
        "entities": {
          "url": {
            "urls": [
              {
                "expanded_url": null,
                "url": "",
                "indices": [
                  0,
                  0
                ]
              }
            ]
          },
          "description": {
            "urls": [

            ]
          }
        },
        "default_profile": false,
        "contributors_enabled": false,
        "favourites_count": 19,
        "url": null,
        "profile_image_url_https": "https://si0.twimg.com/profile_images/2219333930/Froggystyle_normal.png",
        "utc_offset": -18000,
        "id": 70789458,
        "profile_use_background_image": true,
        "listed_count": 1,
        "profile_text_color": "3E4415",
        "lang": "en",
        "followers_count": 63,
        "protected": false,
        "notifications": null,
        "profile_background_image_url_https": "https://si0.twimg.com/images/themes/theme5/bg.gif",
        "profile_background_color": "352726",
        "verified": false,
        "geo_enabled": false,
        "time_zone": "Eastern Time (US & Canada)",
        "description": "Science Fiction Writer, sort of. Likes Superheroes, Mole People, Alt. Timelines.",
        "default_profile_image": false,
        "profile_background_image_url": "http://a0.twimg.com/images/themes/theme5/bg.gif",
        "statuses_count": 1048,
        "friends_count": 63,
        "following": null,
        "show_all_inline_media": false,
        "screen_name": "MonkiesFist"
      },
      "in_reply_to_screen_name": null,
      "source": "web",
      "in_reply_to_status_id": null
    },
    {
      "coordinates": null,
      "favorited": false,
      "truncated": false,
      "created_at": "Fri Sep 21 22:51:18 +0000 2012",
      "id_str": "249279667666817024",
      "entities": {
        "urls": [

        ],
        "hashtags": [
          {
            "text": "freebandnames",
            "indices": [
              20,
              34
            ]
          }
        ],
        "user_mentions": [

        ]
      },
      "in_reply_to_user_id_str": null,
      "contributors": null,
      "text": "The Foolish Mortals #freebandnames",
      "metadata": {
        "iso_language_code": "en",
        "result_type": "recent"
      },
      "retweet_count": 0,
      "in_reply_to_status_id_str": null,
      "id": 249279667666817024,
      "geo": null,
      "retweeted": false,
      "in_reply_to_user_id": null,
      "place": null,
      "user": {
        "profile_sidebar_fill_color": "BFAC83",
        "profile_sidebar_border_color": "615A44",
        "profile_background_tile": true,
        "name": "Marty Elmer",
        "profile_image_url": "http://a0.twimg.com/profile_images/1629790393/shrinker_2000_trans_normal.png",
        "created_at": "Mon May 04 00:05:00 +0000 2009",
        "location": "Wisconsin, USA",
        "follow_request_sent": null,
        "profile_link_color": "3B2A26",
        "is_translator": false,
        "id_str": "37539828",
        "entities": {
          "url": {
            "urls": [
              {
                "expanded_url": null,
                "url": "http://www.omnitarian.me",
                "indices": [
                  0,
                  24
                ]
              }
            ]
          },
          "description": {
            "urls": [

            ]
          }
        },
        "default_profile": false,
        "contributors_enabled": false,
        "favourites_count": 647,
        "url": "http://www.omnitarian.me",
        "profile_image_url_https": "https://si0.twimg.com/profile_images/1629790393/shrinker_2000_trans_normal.png",
        "utc_offset": -21600,
        "id": 37539828,
        "profile_use_background_image": true,
        "listed_count": 52,
        "profile_text_color": "000000",
        "lang": "en",
        "followers_count": 608,
        "protected": false,
        "notifications": null,
        "profile_background_image_url_https": "https://si0.twimg.com/profile_background_images/106455659/rect6056-9.png",
        "profile_background_color": "EEE3C4",
        "verified": false,
        "geo_enabled": false,
        "time_zone": "Central Time (US & Canada)",
        "description": "Cartoonist, Illustrator, and T-Shirt connoisseur",
        "default_profile_image": false,
        "profile_background_image_url": "http://a0.twimg.com/profile_background_images/106455659/rect6056-9.png",
        "statuses_count": 3575,
        "friends_count": 249,
        "following": null,
        "show_all_inline_media": true,
        "screen_name": "Omnitarian"
      },
      "in_reply_to_screen_name": null,
      "source": "Twitter for iPhone",
      "in_reply_to_status_id": null
    }
  ],
  "search_metadata": {
    "max_id": 250126199840518145,
    "since_id": 24012619984051000,
    "refresh_url": "?since_id=250126199840518145&q=%23freebandnames&result_type=mixed&include_entities=1",
    "next_results": "?max_id=249279667666817023&q=%23freebandnames&count=4&include_entities=1&result_type=mixed",
    "count": 4,
    "completed_in": 0.035,
    "since_id_str": "24012619984051000",
    "query": "%23freebandnames",
    "max_id_str": "250126199840518145"
  }
}

That’s from Twitter’s API reference. Now imagine writing a Java class to represent that response. Now imagine doing it again for whatever other insane responses Twitter wants to fling at you.

Jackson has what feels like hundreds of bizarre annotations and configuration options that let you control the translation process between a JSON string and your class, or get around problems like JSON responses suddenly getting new fields and mutually referential objects causing stack overflow errors due to infinite recursion. It’s a huge, complex library, and while there’s plenty of information out there, it can be hard to word a search query in the right way to find a solution to your specific problem. At my company, we briefly looked at migrating from org.json to Jackson, but it was going to be a massive amount of work to figure out the right magical incantations to make Jackson not break and then either go mark up all our database objects with the right annotations or write new view-model classes to use for JSON translation. And since none of us were Jackson experts, we had a steep learning curve ahead of us.

Imagine your car breaks down in the middle of the desert, but you happen to be right next to an abandoned auto shop with a set of books and tools. You don’t know the first thing about cars, other than where the gas goes, but you figure you’ve got a good brain, so you sit down with a book and start reading. After six months of alternately reading and tinkering on some of the forgotten wrecks left on the hydraulic lifts of this abandoned auto shop, you’re ready to start on your car. You eventually get it kind of working, though it can’t go above 37 mph, makes a weird stuttering noise as you drive, and you have to stop every 15 miles and go tighten some bolts, and also you just spent six months camped out in an abandoned auto shop in the middle of the desert. Here you’ve addressed the fundamental problem that your car is broken, and what you came up with is definitely better than a homemade car sail, but was it really worth that six months you spent in that abandoned auto shop?

The root cause, and how to fix it

The root cause of all these problems is Java’s static typing. Well, to be fair, it’s also Javascript’s dynamic typing. JSON is based on Javascript, so it’s perfectly fine with lists of mixed types, maps of string to whatever you please, and arbitrarily nested self-referential objects. Languages like Python, Ruby, PHP, and Clojure, which are dynamically typed and have a similar set of primitives to Javascript, are also fine with that, so you can trivially map any JSON object to a native data type in those languages.

Java is not fine with any of that. It demands lists of a single type and maps of strings to a single type, and it asks that you please tell it beforehand just how deep you’re going to nest those self-referential objects. You can cheat your away around this by declaring the type as Object, but working with Objects is tedious, ugly, and loses you the only advantage to having all these restrictions.

How can we fix this? Here a few features that would definitely help. They’re listed in order of how likely they are to be added to the Java language, starting at “Maybe in Java 26” and ending at “You’ve got a better chance of convincing your company to rewrite everything in Erlang”.

Optional dynamic typing

C# supports a type keyword dynamic. This makes a variable dynamically typed; the compiler basically pretends that it can be any type and supports any operation, and allows a runtime exception to be thrown if you do the wrong thing with it.

This would, at the very least, make the JsonWhatever strategy a lot easier: instead of needing a JsonPrimitive with all those conversion methods, just make the JsonObject and JsonArray store values of type dynamic.

This would also make Jackson’s problem with generic types easier to deal with: instead of binding to lists of Object or maps of string to Object, it can bind to a list of dynamic or map of string to dynamic.

This approach still gives up type safety, but it makes giving up type safety a lot easier to work with and less painful.

Union types

Languages like Haskell and ML support something called union types. Essentially you can define a new type which can hold a value of any of a given set of types.

Imagine that Java had a syntax like type JsonVal = Map<String, JsonVal> | List<JsonVal> | Integer | Double | String | Boolean | null to define a new type called JsonVal. This declares that a JsonVal can be any of those listed types, but no other. If our code ever tries to assign a FizzingWhizbeeFactory to a JsonVal, the compiler will complain.

This can buy us both freedom from casting and arbitrary nesting. The compiler has a list of possible types, so it doesn’t need us to tell it exactly which one a given value is; if we call a method on a JsonVal, it knows where it can look for that method, and if it can’t find the method anywhere, it can flag a compile-time error. It doesn’t need us to cast for it. And the JsonVal type is defined recursively: a JsonVal can be a Map<String, JsonVal> or a List<JsonVal>. This gets us the arbitrary nesting we need to accurately represent JSON.

There are other challenges to getting union types right in Java. For instance, if I have a union type type MyType = A | B and C is a subtype of B, can MyType be an instance of C? Can I cast an instance of C to B and assign it to MyType? Can I then downcast an instance of MyType which is assigned an instance of B to C if that’s what it really is? Or do we just do what generics did, throw the whole mess out, and say that any type which isn’t explicitly listed in the declaration of the union type is invalid? But if we could get union types in Java, it would go a long way towards solving our JSON problem in a type-safe way.

A Dedicated JSON type

This is the easiest option for programmers, and the most deranged from a language design standpoint.

Scala, Visual Basic, and Kawa (a dialect of Scheme on the JVM) have XML literals. In Scala, for example, you can write this:

val people = 
    <people>
        <person firstName="Ted" 
                lastName="Bundy" 
                gender="M" />
        <person firstName="Jeffery" 
                lastName="Dahmer" 
                gender="M" />
        <person firstName="Charles" 
                lastName="Manson" 
                gender="M" />
        <person firstName="Rasmus" 
                lastName="Lerdorf" 
                gender="M" />
    </people>

So in Java, we could just modify the language to let you write this:

Json someJson = {
    "people": [
        { "firstName": "Adolf", "lastName": "Hitler" },
        { "firstName": "Idi", "lastName": "Amin" },
        { "firstName": "Pol", "lastName": "Pot"},
        { "firstName": "Hiro", "lastName": "Mashima"}
    ]
}

The Json type can have a static method, Json.fromString, that inflates a string into JSON, and its toString method can be overridden to serialize a Json into a string.

Scala can sort of do XML literals as part of the normal language, because Scala’s design is already deranged. In Java, the language would have to be modified. But there is precedent for this: arrays in Java are already just random tacked on garbage with special syntax and special behavior that applies to no other part of the language. And Java’s principal use today is as a server-side web programming language; given how important JSON is to web programming, it might be worth modifying the core language to add this.

Still, this is the least likely of the options. Adding lambdas and method references was a massive deal, and those had already proven their worth in dozens of other languages over a period of almost fifty years. Java is not a language that likes revolutionary features; it prefers to wait until every language in the universe, even C++, has a feature, and then panic and squeeze something on top of the existing language features that’s just good enough to disunite the opposition.

Conclusion

Dealing with JSON in Java is quite painful, and it’s because JSON is inherently dynamically typed due to its heritage in Javascript’s native object type, and Java’s static type system is too rigid to accommodate it in an elegant way.

Since JSON is used for communicating between systems and components of a system, the difficulty of working with JSON in Java can affect how you choose to design your API responses. You might choose to favor flat objects over nested, as we used to do at my company. Instead of something like this:

{
    "room": {
        "name": "library",
        "carpetType": "Persian rug"
    },
    "killer": {
        "name": "Colonel Mustard",
        "weapon": "lead pipe"
    }
}

You might choose this:

{
    "roomName": "library",
    "roomCarpetType": "Persian rug",
    "killerName": "Colonel Mustard",
    "killerWeapon": "lead pipe"
}

In Java, with Jackson, the second form requires a single class with four strings. The first form requires a class with two subclasses, each with two strings, and probably a few annotations to make sure everything happens correctly when you bind from a string. So up front, the first form is more complicated to work with in Java, and this extra complexity just gets more pronounced as your JSON responses get more complex.

But using the second form causes other problems. Since JSON keys have no guaranteed order, you could send the response to a client like a browser and have it print with the keys all randomized, which makes the response much harder to read if you need to debug. If a client of your API only cares about the room, they have to loop over all the keys and find the ones that start with room instead of being able to just grab the room object. You can write that annoying code every time, or you can write some stupid little helper function that you have to document and test to loop over a JSON object and find all the keys with a certain prefix. You’ll probably also have to duplicate this function across various code bases in your stack. This tradeoff might be worth it if you just have a bunch of Java code communicating (e.g. an Android app and a Spring API), but if you have other languages in your stack, like Javascript or Python, you’re forcing them to conform to the lowest common denominator.

If you’re writing web code nowadays, JSON is a fact of life, and it’s a great transmission format: it’s simple, readable, and powerful, easy to parse, able to represent almost any kind of data in human readable format. It’s elegant and easy to use, a big improvement in a lot of ways over RPC formats like COM and CORBA, unreadable binary formats, and XML. It’s a travesty that Java has such terrible support for something so elegant, widespread, and fundamental.

Advertisements

4 thoughts on “JSON in Java: A Case Study in Pain

  1. You can deal with JSON in a more systematic way, the generic code is not that hard to write class that will perfom more or less generic mapping to Json (in practice just use Google library). If you still want to use Java and your bare hands, you can write the following code (I was too lazy to perform all that reflective operations so I used annotations to resque myself from a bit of work). Here is a code sketch:
    Annotation to help us to create collections

    @Retention(value = RetentionPolicy.RUNTIME)
    @Target(value = ElementType.FIELD)
    public @interface CollectionOf {
    Class[] type();
    }

    Annotation to help us resolve entity key:
    @Retention(value = RetentionPolicy.RUNTIME)
    public @interface JsonEntity {
    String name() default "";
    }

    Some quick and dirty code to create objects:

    public class JsonMapper {
    private static final Class VALUE_TYPE = JsonEntity.class;

    /*unchecked exception*/
    public static class JsonException extends Error {

    public JsonException(Exception reason) {
    super(reason);
    }
    }

    /*overloading for convinience*/
    public static T map(String json, Class destType) {
    return map(Json.createReader(new StringReader(json)).readObject(),
    destType);
    }

    /* Map parse objects to classes */
    public static T map(JsonObject parsed, Class destType) throws JsonException {
    try {
    //wrap exception, of course map is still unsafe
    return (T) unsafeMap(parsed, destType);
    } catch (InstantiationException | IllegalAccessException
    | ClassNotFoundException ex) {
    throw new JsonException(ex);
    }
    }

    /**
    * Primitive map method
    * You could have used more reflection to get private fields wia getters/ setters
    * @param parsed parsed object
    * @param destType destination type
    * @return object
    * @throws all kinds of disasters
    */
    private static Object unsafeMap(JsonObject parsed, Class destType)
    throws InstantiationException, IllegalAccessException,
    ClassNotFoundException, InstantiationException {

    Object result = destType.newInstance();

    //to keep things simple we acess onlyt public fields
    Field[] fields = destType.getDeclaredFields();

    for (Field field : fields) {
    JsonEntity entity = field.getAnnotation(VALUE_TYPE);

    /* Get key */
    String key = entity == null || entity.name().isEmpty()
    ? field.getName().toLowerCase()
    : entity.name();

    //get class, will be null if we have reference type
    Class type = field.getType();

    String name = type.getCanonicalName();

    switch (name) {
    //handle primitive types
    case "int":
    case "java.lang.Integer":
    field.setInt(result, parsed.getInt(key));
    break;
    case "double":
    case "java.lang.Double":
    field.setDouble(result, parsed.getJsonNumber(key).doubleValue());
    break;
    case "long":
    case "java.lang.Long":
    field.setLong(result, parsed.getJsonNumber(key).longValue());
    break;
    case "java.lang.String":
    field.set(result, parsed.getString(key));
    break;

    //handle some collections
    case "java.util.List":
    /*
    * you can perform some additional work to get type
    * without annotation
    */
    Class listType
    = field.getAnnotation(CollectionOf.class).type()[0];

    List resultList = new ArrayList();

    switch (listType.getCanonicalName()) {
    case "java.lang.Integer":
    resultList.addAll(toList(key, parsed,
    JsonString.class,
    (x) -> ((JsonNumber) x).intValue()));
    break;
    case "java.lang.Double":
    resultList.addAll(toList(key, parsed,
    JsonNumber.class,
    (x) -> ((JsonNumber) x).doubleValue()));
    break;
    case "java.lang.String":
    resultList.addAll(toList(key, parsed,
    JsonString.class,
    (x) -> ((JsonString) x).getString()));
    break;
    default:
    resultList.addAll(toList(key, parsed,
    JsonObject.class,
    (x) -> map((JsonObject) x, listType)));
    }
    field.set(result, resultList);
    break;
    }
    }

    return result;
    }

    /* Helper method */
    private static List toList(String key, JsonObject parsed,
    Class clazz,
    Function mapper) {
    return parsed.getJsonArray(key)
    .getValuesAs(clazz).stream().map(mapper)
    .collect(Collectors.toList());
    }
    }

    Of course, such code is left to the framework, so you’ll never do it yourself, but the idea is: if something is painful, maybe you’re using the wrong instrument?

    • Example usage:

      class Student {
      
          @JsonEntity(name = "name")
          public String name;
      
          @JsonEntity(name = "surname")
          public String surname;
      
          @JsonEntity(name = "marks")
          @CollectionOf(type = Integer.class)
          public List marks;
      
          @JsonEntity(name = "avg")
          public double avg;
          
          @JsonEntity(name = "colleagues")
          @CollectionOf(type = Student.class)
          public List colleagues;
      
          @Override
          public String toString() {
              return name + " " + surname + " " + marks.toString() + "\\" + avg 
                      + " and his collegues:" + colleagues.toString(); 
          }
      }
      
      /**
       *
       * @author Home
       */
      public class JavaApplication2 {
      
          private static final Logger log = Logger.getLogger(JavaApplication2.class.getName());
      
          /**
           * @param args the command line arguments
           */
          public static void main(String[] args) {
              try {
                  /* Lets deserialize one Paul parker and two clones of John smith */
                  Student student = JsonMapper.map(
                          "{\"name\":\"Paul\","
                          + "\"surname\":\"Parker\","
                          + "\"marks\":[5,5,5,5,5],"
                          + "\"avg\":5,"
                          + "\"colleagues\":"
                          + "[{\"name\":\"Jonh\","
                          + "\"surname\":\"Smith\","
                          + "\"marks\":[1,2,3,4,5],"
                          + "\"avg\":3,"
                          + "\"colleagues\":[]},"
                          + "{\"name\":\"Jonh\","
                          + "\"surname\":\"Smith\","
                          + "\"marks\":[1,2,3,4,5],\"avg\":3,"
                          + "\"colleagues\":[]}]}", Student.class);
                  System.out.println(student.toString());
              } catch (JsonMapper.JsonException ex) {
                  log.log(Level.SEVERE, "failed to deserialize, reason: " + ex.getMessage());
              }
          }
      }
      

      • Thanks for your comments!

        The hand-rolled example at the front was just meant to demonstrate the thought process that someone might use to approach the problem of representing JSON in Java. It’s not something I would actually do, it’s a demonstration to show people who might not be familiar with Java (e.g. several of my colleagues, who have worked with dynamically typed languages such as Javascript and Python for their entire careers) the thought process that leads to libraries like org.json, Jackson and GSON.

        Your example code looks similar to Jackson: inflating the JSON string into an intermediate data structure, then reflectively binding it to an instance of some class annotated with instructions on how to perform the binding. My two big problems with this approach are the necessity of defining a class with all the annotations, and the complexity of implementation that comes from the reflective binding process. In Python, for instance, you can just do json.loads(some_json_string) and you get back a native dictionary type, with no need to define and annotate a class. And the reflective binding process leads to some of the most unreadable and hard to diagnose errors I’ve ever seen; to really understand what’s wrong, you have to already have a pretty good understanding of several low-level details of how the library is implemented.

        If the JSON you’re working with is simple, neither of these problems come up, but if you’re working with complex JSON like the example from the Twitter API in the post, working with this approach becomes painful. On a work project I had to deal with some massive responses from the Salesforce API: several megabytes worth of oddly structured, crazily nested bramble of objects and arrays. The hoops I had to jump through and weird type errors I had to deal with from Jackson convinced me that although this approach might be the best way to deal with JSON in Java, it was far from an optimal way to deal with JSON.

  2. Yes, with JSON you need to leave off the typing, which is problematic. In the end you find yourself dealing with low level API’s like SAX for xml (and analogous for JSON, just pointed that one out as a best-known example).

Comments are closed.