UK COVID-19 Cases Missed Due to Excel Glitch
James Tapsfield (also: Hacker News):
The extraordinary meltdown was caused by an Excel spreadsheet containing lab results reaching its maximum size, and failing to update. Some 15,841 cases between September 25 and October 2 were not uploaded to the government dashboard.
[…]
The problems are believed to have arisen when labs sent in their results using CSV files, which have no limits on size. But PHE then imported the results into Excel, where documents have a limit of just over a million lines.
The technical issue has now been resolved by splitting the Excel files into batches.
But while CSV files can be any size, Microsoft Excel files can only be 1,048,576 rows long – or, in older versions which PHE may have still been using, a mere 65,536. When a CSV file longer than that is opened, the bottom rows get cut off and are no longer displayed. That means that, once the lab had performed more than a million tests, it was only a matter of time before its reports failed to be read by PHE.
Excel recently turned 35, and it sounds like they were using the old .xls format that was superseded in 2003. The .xlsx format has higher limits, but even 1,048,576 rows is less than one doubling away from the 516K total cases in the UK that Google currently reports.
Previously:
5 Comments RSS · Twitter
No... it's even more embarrassing. They were using 1 column per patient. There's a 16384 column limit because columns are supposed to be used for features, and rows for individual items.
The UK is unable to find an intern who knows how to use Excel correctly for a matter of vital economic importance, let alone an intern who can write a python script to access an SQL DB. No wonder Brexit is such a mess...
Nevermind, it seems the BBC says you're right: https://www.bbc.com/news/technology-54423988
Either way it's pathetic.
As the author of a database program I am obviously biased, but it always dismays me to see any spreadsheet program being used for database tasks. It's super common though. I guess it's better than using a word processor, which occasionally happens also.
It's a terrible, terrible indictment of the British Government that they cannot bring the appropriate technical competence to bear for something so crucially important - critical even - to the nation's health and economy. It's a breath-taking failure and heads really should roll...
