The title of my previous post, Software Development: the Secret of Success, was originally meant to be tongue in cheek, as you might have guessed. Like that of another post, The Secret of Academic Success. It was a play on that title.
Then I got ambitious and tried to find a secret but the best I could come up with was “get help … “; which is a lame and obvious platitude. But at the last moment, at the end of the post, I had a revelation: “write for the future” or, more generally, “build for the future”.
OK, sounds like another lame and obvious platitude. But hear me out.
It’s a platitude because who would disagree that it’s good to write for the future? But on the surface it’s not practical advice for at least two reasons:
One, how do we know what the future holds? If we don’t know what’s coming, how can we write for it?
Two, it’s going to take extra time/effort/resources/money which we may not have. We may have a boss who wants something now and doesn’t give a toss for the future.
Nevertheless I still think it’s good advice. Sometimes rules that are on the surface obviously wrong are true at a deeper level. Marshall McLuhan has many such sayings. One example is, “the effect precedes the cause”. I think a lot about that one.
A hard earned lesson
The need to write for the future was brought home by direct experience with research software I used, as I briefly explained in the previous blog post. At one point we had a wonderful Lucid interpreter, written originally (in C) by C. Ostrum and majorly upgraded by A. Faustini. It had excellent syntactic and runtime error checking; integers, reals, atoms, strings, lists; user defined functions; interactive I/O; and an escape to UNIX. It ran all the programs in the 1985 book plus powerful array (field) capabilities.
Now that interest in dataflow and Lucid is picking up I would love to make it available. I have the source. But nobody can run Lucid on it because the source won’t compile. Putting it through a modern C compiler produces literally hundreds of errors and warnings. Apparently 30 years ago C compilers were much more tolerant.
The pLucid interpreter (that’s what we’re talking about) was not, on the whole, written for the future. I take a lot of the blame for that. I kept asking for more features now, instead of (say) insisting that the code go through lint without any reports. I was the boss who didn’t give a toss.
You can say in my defense, how was I to anticipate that I’d need to compile it three decades in the future? Well I had many reasons to anticipate this.
Anyone with any experience in software should realize that even professionals seriously and regularly underestimate the length of time that a program is going to be around, and the challenges it is going to face.
The canonical example is Y2K. The programs affected were mostly written in the 50s and 60s when the programmers couldn’t imagine that three or four decades later they’d still be running. It seemed wasteful of resources at the time to include “19” in year. (In fact much less was at stake. They could have used one bit to distinguish 19.. and 20.. years and had software that worked thru 2099).
I concede that in some ways the pLucid interpreter was written for the future. For a start it was written in C and C did not go away – though many thought it would be replaced by something more high level. And it ran under UNIX on a terminal. Terminals were overshadowed by GUIs but they didn’t disappear.
The experience with the popshop code was the other end of the spectrum. It was what people call “overengineered”. It was highly modular (dozens of informal C modules). Most of the routines were tiny, maybe half a dozen lines of code. Commented code. There were several layers of abstraction, strictly obeyed (C has no facilities for enforcing abstraction, this was programmer discipline).
Each module was a folder with several files. These included a file of procedures, a list of other modules needed (“imports”), a file of global declarations, a file of initialization code, etc. A special too we produced, called sloth, integrated the modules, called in the imports in logical order etc and produced a huge .c file. The total software effort was far more than that involved in producing a straightforward interpreter. The idea was to make it easy to generate intensional software – like an intensional spreadsheet.
(Sloth suffered from a particularly egregious example of present-oriented programming. Obviously I kept the source for sloth. But to save a few lousy bytes, I compressed it into .C format. And now only a couple of decades later, I discover that the decompressing routines are no longer available. Gone. Lost.)
By the measure, “I want something now” it was a failure. No useful application was ever produced, just test versions. It was almost completed but I abandoned in when I realized I’d never solve the memory management problem. C has no garbage collector and I would never have tracked down the last memory leak.
That was a mistake, and an example of not writing for the future. I should have realized that memory would become plentiful and even leaky programs would be useful up to a point. I should have realized that eventually something like C would appear with automatic memory management.
The future is bright
I made a similar programming-for-the-present error many many years ago with my attempts to implement my ‘Griff’ macro processor. I dithered between doing it in 7040 assembler, FORTRAN IV, and LISP (the only practical choices at the time).
7040 assembler was too difficult (and would have been a dead end). LISP was too slow and FORTRAN IV would probably have been slow, plus difficult. So I abandoned Griff.
Big mistake. Either the FORTRAN or the LISP options would have soon given me a usable interpreter, once the hardware and software caught up. FORTRAN would have been trickier but more portable.
For a long time I attributed the failure of the popshop to the classic mistake of overengineering, and that of Griff to underpowered hardware. In fact if I’d had the outlook of writing for the future I would have finished them both and soon reaped the benefits.
In both cases I was a bit ahead of my time. But that doesn’t have to be fatal. You have to write for the future and – as the French say – give time time.
It was the popshop that really came through in the end. Eventually Python arrived and I discovered I could basically transcribe the popshop software to Python. All the hard work had been done – for example, presenting Lucid expression as an abstract data type.
How do you write for the future?
While it’s true that we can’t predict the future, by now we know a lot about the future of IT and this knowledge can be used as a guide for writing (or more generally building) for the future.
For a start we know that hardware is soon going to be significantly more powerful. Processor speeds may not increase by much but machines will have more cores and GPUs will play a bigger role. Memory will be bigger and faster as will disk speeds and capacities. And advances in compiler technology will produce clever code from straight forward programs.
As a result software performance will improve dramatically without the need for complicated rewrites. Rod Burstall once said that “efficiency is the enemy of clarity”. So one form of writing for the future is to write relatively simple if suboptimal algorithms and let future hardware and compiler improvements take care of performance issues.
(I wish someone had made this point when I was working on Griff so many years ago.)
On the other side of the coin we can be almost certain that our software will eventually have to support far greater volumes of data, of queries, of transactions and the like than we currently experience. One example is the unemployment application processing systems in New York that because of Corona virus shutdowns had to process hundreds of thousands of applications – they failed under the load.
In this case writing for the future means producing software that can quickly be scaled up – by orders of magnitude if necessary. Again this is favored by simple algorithms and data structures
Another trend we can take for granted is that the software we produce will be in use much longer than we can imagine, in spite of our current plans. The New York software also suffered here – it was written decades ago in COBOL. Years of austerity ruled out upgrades.
Once we are realistic about how long our software is going to be used, we see several future-proofing strategies.
For one, documentation is very important. Imagine the situation of some software engineer maintaining a program that was written before she was born, in a language neither she or any of her colleagues has ever learned. Without documentation she is lost.
The same can be said for basic practices such as having meaningful identifier names and naming schemes that tell you something about what the variable denotes; such as “getTemperature” and “setTemperature”.
It’s clearly important to carefully choose the language and system. All else being equal it should be a language that’s been around and will stay around, and it should be a very stable release. This rules out, for example, Python 2 and the absolute latest version of Python 3.
A pattern emerges
In this way we could continue listing known trends we have to take into account and fairly obvious strategies to deal with them. For example we can be confident that errors will be found and extra features will be requested. This calls for documentation, again, and general purpose (as opposed to very specific) code.
However a pattern emerges, that almost every practice (say, information hiding and abstraction) that is considered Good Software Engineering is future oriented. So isn’t “writing for the future” just being a good software engineer?
Unfortunately not. There are a whole set of practices that are vital but not future oriented. They all have the goal of gaining a temporary advantage. And these practices may violate Good engineering principles.
An example, given above, is to write a complex but efficient algorithm to solve a critical performance issue. This gives the software an advantage, but one that is temporary in the sense that hardware advances make the complex solution eventually unnecessary.
Another example is skipping the documentation stage in the rush to get to market. This may save a lot of time (and is very common) but down the road the lack of documentation can become a serious, and permanent, disadvantage.
A judgement issue
So in summary, people who produce software have to manage a contradiction between doing things that are good in the short term, and using Good practices that benefit the long term. When I propose “write for the future” I don’t mean it as an absolute – I don’t mean always choose the long term good. Any software project (especially if a business is based on it) will need short term advantages.
What I mean by “write for the future” is that this should be the default choice. Write for the present – for an immediate advantage – only if the advantage is 1) real and 2) really needed. I don’t believe there is any fixed, more precise rule that makes this choice. It’s always a judgement call that depends on the context.
In general if the software is not entangled in the market – not, for example, an implementation of a key feature of a system being sold – then it’s much easier to choose the future. This includes research software, open source systems, and personal side projects.
But where economic competition is involved, the pressure for short term advantage can be irresistible. I believe this explains why, after decades of research in software engineering, so much commercial software is so unreliable.
Build for the future
It should be clear that the choose the future principle applies to domains other than software. The general preferred engineering approach is to build for the future.
There are several examples of spectacularly successful projects that owe their spectacular success to future-oriented engineering. Two examples come to mind: the internet and the personal computer.
The internet was designed and built as a general purpose highly resilient communication network long before the web let alone email were conceived. It has stood up to wave after wave of communication-heavy applications and even the Corona pandemic didn’t shut it down.
Personal computers preceded the internet but were almost perfectly suited to being nodes in a packet switching network. The basic reason is that it was a general purpose device designed when people wanted word processing and storing recipes.
In fact there is a theme running through these and other future-oriented successes: redundancy. For example, making a tool general purpose is itself already a form of redundancy – of devoting more than the bare minimum of resources to solve a problem.
Other examples come to mind: in education, teach for the future; in health, treat for the future; in athletics, train for the future.
I eat my own dog food. In this essay I anticipate objections and counter them before they’re raised. In fact this is an example of an effect preceding a cause: the effect is my counter argument, the cause is the objection.
Which brings us back to McLuhan. He was an expert at writing for the future. His “global village” and “medium is the message” concepts preceded the web by almost three decades. In fact McLuhan’s ideas are an outstanding example of an effect long preceding the cause.
So also is the whole idea of writing for the future. The cause is software disasters down the road. The effect is writing now to avoid them.