Working on legacy code can lead you to seeing some things you’d never otherwise see. A few years ago I read the blog post “A Beautiful Race Condition”. It has a very detailed explanation of what can happen under the hood of a java.util.HashMap, which is not thread safe, when someone uses it as if it was. This was an interesting but hypothetical topic for me, because the problem is known and I’d been working on web services where we avoided sharing objects between threads.

Fast forward a few years and a couple of positions, and now I’m working on making a good-sized monolith able to scale. One day the operations team contacted me to say that one of our production servers was setting off alarms because its CPU usage was hitting 100%. When I got a thread dump from the problem server, what did I see but

  java.util.HashMap.getEntry(HashMap.java:347)
  java.util.HashMap.containsKey(HashMap.java:335)
  name.changed.to.protect.the.guilty.Example.getSomething(Example.java:1234)

(Unfortunately I can’t show the real code because it’s proprietary)

There were multiple threads stuck in this state. Seeing the stack trace triggered just enough recollection that I was able to google up that blog post. Rereading it confirmed that the description matched, which pointed out where I needed to go to do the fix.

For our servers it seems that one stuck thread will cause 1 CPU to report full usage. Two stuck threads a on 4 CPU VM caused 50% CPU usage for the entire server, and four stuck threads a on 4 CPU VM caused 100% CPU usage for that server. The system was otherwise responsive, so these threads weren’t preventing other threads from getting CPU cycles. That was fortunate, because the only way to get the CPU usage back down is to restart the JVM, and the only way to prevent the problem from recurring is to fix the code and deploy a new version.

It’s straightforward to make sure you’re not passing HashMap objects from one thread to another. A case to watch out for is using a HashMap as the value in a cache – if two different threads both get cache hits on the same key, the HashMap value object will end up shared by those threads. Since the contents of a cache are intended to be shared, any value stored in a cache should be a ConcurrentHashMap or one of the immutable Map implementations that’s available.

Recently I’ve been working towards migrating a legacy application from Java 6 to Java 8. While the language and the JVM are backwards compatible, there have still been a few issues that I thought I would document in the hope that it might help someone else.

There are number of unit tests which set up test data in the database before they start. (I know there are lots of reasons not to do that, but when an application has as few unit tests as this one does you hang on to whatever you’ve got.) One of these tests passed consistently with Java 6, but failed with Java 8. The failure suggested a problem with the test data.

Here’s a snippet of the code. The methods are the traditional ones from the JUnit 3 days. In the debugger I could see that data was being created successfully by the setup method, so how could there be a problem with it?

@Override
@Before
public void setUp() {
    // initialization code...
    // more initialization code...
}

@Override
@Before
public void tearDown() {
    super.tearDown();
}

It turned out that the problem was right in front of me but hard to spot: the tear down method was annotated with @Before. What happens when there are two @Before methods? If the tear down runs before the setup it’s okay, there’s just nothing actually deleted. But if the tear down runs after the setup it cleans out all the test data just before the test runs.

How this code ran reliably with Java 6 I do not know. To fix it for Java 8, I actually just removed the tear down method since (as you may have noticed) it didn’t do anything useful anyway.

Recently I’ve been working towards migrating a legacy application from Java 6 to Java 8. While the language and the JVM are backwards compatible, there have still been a few issues that I thought I would document in the hope that it might help someone else.

The biggest issue I’ve seen is unit test failures caused by unspecified sort ordering – the order in which items are returned from a collection when there isn’t an explicit ordering. Somewhere along the way, the order in which items are returned by the java.util classes when there isn’t an explicit sort order changed. With Java 6 results were returned in an order that was consistent, test run after test run. With Java 8 the order has changed. (It may still be consistent from run to run – I haven’t tested that.)

For example, one unit test iterated over the items in a Set to verify their contents. This worked in Java 6, repeatably. When run with Java 8 the same items are returned, but in a different order, so the verification failed. I had to revise that test to not depend on the order of items returned by the iterator.

Another test sorted items by timestamp, but assigned a number of them the same timestamp. That was fixed by tweaking the timestamps to have different values in the seconds field.

Then there was one where there was a bug in a production comparator: when the current object was not null and the other object was null, it treated them as equal despite a unit test which clearly expected nulls at the end. Somehow this worked with Java 6. In that case I didn’t change the unit test at all, just fixed the comparator.

Check 1, 2 , 3 …

Is this thing on?