The Codebase of a National Lab I Worked At
During my freshman year, I lived in a magical world. In this magical freshman world, students sometimes turned in sloppy code, but professionals always took the time to write nice, maintainable codebases. Everybody, as far as I could tell, followed best practices. Everybody used Git and GitHub, and proper branching protocols were always followed. This dream was shattered the summer after my freshman year when I went to work for a national laboratory.
I will not be mentioning this lab by name in this post for the sake of plausable deniability, but if you figure it out... well, if they wanted me to put more effort into not publicly complaining about them, they should have put more effort into their codebase.
This lab was a top security lab. I was given Property Protection clearance, but I knew people who had Q clearance. Or rather, I think they had Q clearance, but when I asked them what their clearance was, they didn't tell me and had to report that I had asked them about what clearance they were.
My job while interning at this lab the summer after my freshman year was to update a six million line codebase from Java 8 to Java 11. The codebase did not have unit tests. Well, it had some, but nobody knew what percent of the codebase had test coverage. Part of my job was writing unit tests while I was adding new code. The reason why I had to create new unit tests, rather than update old ones, was that the part of the codebase I was working on did not have test coverage.
The codebase was presumably in Java 8, but while inspecting it I came across a file that said it was from the Java 4 library. But it was not imported from the library; it was just a code file claiming to be from the Java 4 library sitting in our Java 8 codebase. Perhaps the file was lost? I scrolled down and saw that at one point in the file, there was a constant setting a limit regarding the number of threads that could be spawned while running the program that the file managed (I don't recall exactly what it did). There was a comment declaring that the constant was too low in the original library, so they copied and pasted the library file and manually changed the constant. This file had stayed there since the days of Java 4. I don't know if there were any repercussions for increasing the thread count more than the library had intended; perhaps our computers were so strong that it did not matter. It was, after all, a national lab.
Around the middle of my internship, there was a code rollout. This was basically a "push to production", except that for the day of the push, it was all hands on deck because they knew there were going to be errors. Errors that would cause the people working in the extremely high-tech and high-security physics labs to complain. This sort of thing happened about every six months when they rolled out production.
The lab did not use Git; it used Accurev. I prefer Git to Accurev simply because I'm familiar with Git, but Accurev was presumably the right call for such a large company. But in order to update something with Accurev, you had to go through a ton of checks, so there was just dead code hanging around everywhere because nobody wanted to go through the whole ordeal of updating it. My supervisor's supervisor claimed this was not supposed to happen.
Many of the files seemed to have not been updated in a while. The copyright message for the main GUI claimed that the software was the property of the University of California, even though the lab had not been part of the UC system since 2007. I was working there in 2019. Some of the other code files were copyrighted from 1999, before I was born. Perhaps this code was just very future-proof!
At one point I was describing to one of the programmers that I had to add tests to cover the code that I was changing. They were shocked that the tests didn't already exist and were concerned by the lack of test coverage in the codebase. When I told this to my boss, he said that the idea of writing tests to cover the whole codebase was a fairly new idea, and that having full code coverage wasn't realistic and nobody actually programmed like that. He also thought that it was silly to chase after warnings, because warnings are not really errors: they are just complaints that the IDE makes, and the IDE is just a tool. You can suppress warnings, and a different IDE configuration might give you different warnings or no warnings at all.
However, it is not entirely fair of me to criticise my supervisor, because as a 19-year-old who had spent exactly one year in university, I was hardly a programming expert myself. Part of my job was creating a script that looked at all of the imported Java libraries and their dependencies. There were several hundred libraries, and they all had quite a few dependencies. This was the 10th circle of Java Library Dependency Hell. My solution for this was a bash script that took 10 minutes to run. Towards the end of the project I realized my mistake and translated it to Python, resulting in code that took maybe 1 or 2 seconds to run. Why on Earth did I think bash was a good idea? Because this was my very first internship and I had zero idea what I was doing. I told a more senior engineer about this, and he consoled me by saying I was still better than some of the full-time programmers at the lab. I sincerely hope he was sugarcoating things to try and make me feel better, because otherwise I'm very concerned with the quality control.
So you may be wondering, why was the code at such a fancy lab this bad? Well, my supervisor's supervisor claimed it was because they had trouble recruiting young people. But this doesn't make sense: even if you are good at code, you will still age like everybody else. If you age, you will become old. This proccess, while sometimes unfortunate, does not decrease your ability to code. In fact, you often become better after so many years of practice. Being old and being good at code are not mutually exclusive.
I think a part of it is that they don't pay as much as FAANG+? It could also be a cycle of bad code: their codebase is so bad that they can't hire good people, and the codebase just becomes worse as a result. It is also possible that the lab's focus is not programming, and that the programming only exists to support the amazing scientific research taking place at the facility, so code quality takes a back seat. This is arguably not a bad thing, since in a national lab, science remains more of a priority than the program which is written to support it. But still, the research is high-stakes, so much so that its supporting codebase would at the very least be expected to have a reasonable amount of test coverage.
You could certainly argue, if we're being honest, that this codebase is not much worse than the average legacy codebase. And, as much as I disparage the codebase, the reality is that the research was getting done—and getting done well—despite the codebase, so clearly something was going right. Maybe moving fast and breaking things is the right way to go in this situation.
What bothers me about this situation though is that this lab is arguably the highest stakes lab I have worked in. I know that the world doesn't neccecarily assign its most talented individuals to its hightest stakes projects, but this is an egregious missalocation of talent. With work this important taking place, the code quality has no right to be this bad. On the bright side however, I can console myself when I make a mistake in my programming by reminding myself that no matter how bad the code I write gets, chances are, it is probably still better than the codebase at the national lab.
Questions, comments, or criticism?
Join the discussion!
HackerNews Lobste.rs