What slow software really costs

This week I measured our own archive search. Two seconds per query. It had never bothered me before, and that is the interesting part.

Two seconds is too short to complain about and too long to be free. It is the classic case of effort that shows up nowhere: not in the books, not in a project report, not in the conversation with the IT partner. It spreads across the day in slices, and nobody books slices.

This Sunday edition works the case through. With numbers I measured myself this week.

The harmless half: the bill in hours

Take ten employees. Each searches for something thirty times a day — in the archive, the CRM, the filing system. Each search takes two seconds longer than it needs to.

That is ten minutes a day. Forty hours a year. A full working week nobody notices, because it arrives in two-second slices.

The number is unspectacular, and it is still useful: it turns a feeling into a quantity. “The software is just slow” cannot be budgeted. “Forty hours a year” can.

The expensive half: what nobody writes down

The second bill is bigger and appears in no table. It starts the moment searching becomes tiring.

Then a colleague phones instead of looking it up. Then someone keeps their own list because the one in the system takes too long to load. Then a second, unofficial system grows next to the official one: a spreadsheet on a laptop, notes in the mail client, a folder on the desktop.

Those shadow archives are the real cost. They are not backed up, not searchable, not handed over. Whoever leaves the company takes them along. And they do not come from carelessness — they are a reasonable response to a system slower than your own head.

The connection is uncomfortably simple: the longer a search takes, the less searching happens. And the less searching happens, the less the system holding the data is worth.

Case one: one setting, factor 15

Back to the two seconds. The cause was not overload, not an undersized server, not badly written software.

The database was running on the factory settings from the installation package. The memory it is allowed to keep for frequent access was set to 128 megabytes. The indexes it needs for that are 8,247 megabytes in total. So on every query it read from disk what it could long have kept in mind.

One line changed, service restarted. 2,025 milliseconds became 90 to 136. A factor of 15, without new hardware, without a line of code.

The point is not that somebody was sloppy. Factory settings are chosen so the software starts on any machine — from a developer laptop to a large server. They are a compromise for everyone and therefore right for nobody. The only question is whether anyone looks again after installation.

Case two: the invoice that arithmetic made go away

The second finding of the same week: we were short on memory. The quote for the next machine up was 138 francs a month.

The reflex is to buy it. That is not wrong, just expensive — and often the second-best answer. So we measured first who actually creates the load.

Six databases shared one machine. Four of them are 27 gigabytes together and get queried a few times a day. Two are 414 megabytes together and carry 112 million transactions.

The smallest item does the most work, by more than a thousandfold. And because both sat in the same pot, they kept evicting each other from the cache: eight gigabytes of indexes do not fit into four gigabytes of cache, and the constantly working small databases kept being pushed out by the large, quiet ones.

The fix was not an invoice but a split by working pattern. The large, rarely queried holdings now sit on a machine that runs anyway and has memory to spare. The small, constantly working ones stay where the programs that use them are.

Cost: zero instead of 138 francs a month. And reversible at every stage.

Case three: the move nobody noticed

That leaves the question of how you move something like that without stopping operations. The usual answer is a maintenance window on Sunday evening.

There is another way, and the trick is old. Between the programs and the database you put a broker. From then on all programs talk to it instead of to the database directly. During the move it gets two commands: pause — it holds every open connection and collects the requests. Resume — it sends them to the new location. For the programs nothing happened, except that one request took a moment longer.

The decisive part is the order: this broker is introduced while everything still runs at the old location. Then the risky part is over before the move even begins.

And it is not only good for that one day. It pools connections, and we run a service that opens a new one on every call. At 112 million transactions that is the difference between “runs” and “runs well”.

What of this applies to you

You probably do not have a vector database. That does not matter, because none of the three cases is about vectors.

1. Factory settings are a compromise for everyone. Ask your IT partner when they last reviewed the settings of your most important application. If the answer is “at installation”, you probably have headroom.
2. Measure before you buy. The question is never “is the server big enough”, the question is “who creates the load”. Surprisingly often that is not the same data that takes up the space.
3. Slow is a symptom, not a property. “The system is just slow” is not a diagnosis. Which query, how long, how often per day — those are three numbers, and they are collected in an afternoon.

And the point above all of it: the most expensive consequence of slow software is not the lost time. It is the habit that forms around it. Whoever has spent six months learning that searching does not pay off will not search again even once it is fast.

How long do you wait — and how often a day?