National Archive File Format Time Bomb 233
geordie_loz writes "The BBC is reporting that the UK National Archive is warning of old formats being a 'ticking time-bomb' where data is going to be lost because of incompatibility in newer versions of software, and software not existing at all. More surprisingly, Microsoft has offered a solution via the OOXML format."
Tagging beta... (Score:2, Insightful)
MS should not own the standard (Score:2)
Re:MS should not own the standard (Score:5, Informative)
Re: (Score:3, Informative)
While the GP may or may not have been exactly sure what they were referring to, it doesn't make them wrong.
Re: (Score:2)
Such precise terms as (Score:4, Informative)
Becuase if you want to include bugs etc, then no, it doesn't support each and every 2007 feature.
If you mean supporting tables, nested documents, embedded graphs, scripting and so on, yes.
It may not be "click the same buttons" feature correct nor probably the "run the same VB code" compatible.
Take a look at some of the people on the board that devised ODF. They include the US National Archives. Print media. Archivists.
Y'know, people who KNOW DOCUMENTS.
As to the remainder of your questions, there is a process, it does have to go through comittee (else how does everyone else know how to implement the new standard? MS doesn't have this problem since they only want themselves to know their updated standard). It is XML so it is extensible (decode the initialism). The process will take as long as it takes. Much the same as Vista will take as long as it takes to get SP 1 out.
I don't see how these latter issues are something that is a part of ODF and not any form of standardisation that OfficeXML will have to have to go through for anyone other than MS to implement...
Re:One thing I'd like to know (ODF question) (Score:5, Informative)
Yeah, it's XML. Also, unlike OOXML, ODF uses namespaces, so you can create a separate standard if you don't want to muck around with ODF.
It would depend. The thing about changing standards is that it causes problems for all sorts of people. There is a real need for a stable and standardized document format that just doesn't change, or if it does, very slightly.
Re:MS should not own the standard (Score:5, Funny)
Rubbish. I've worked at places with an Open Office format. Basically they open the office to any monkey who turns up for a job interview and a handful of people have to make up for their incompetence.
Idiots (Score:4, Insightful)
There are so many idiots in this state of the affairs:
1. the idiots which decided to build huge archive with undocumented proprietary format
2. idiots which believe they can't find even a single copy of the software they need
3. idiots who didn't store a single copy of the software that reads the format, together with the archive (not very far from obvious, is it).
4. idiots who want to convince other idiots that OOXML is an open format (versus straight XML serialization of the whatever binary DOC was in the source code base at the time in MS)
Re:Idiots (Score:4, Interesting)
Please give me a link to a copy of the Professional Write 3 (PW) software app. for MSDOS 6.
Yep, I had that very problem some years ago when I was cleaning my room and found several 5 1/4 disquettes which contained the
Re:Idiots (Score:4, Insightful)
Re:Idiots (Score:5, Insightful)
The system wasn't thought up any more than a library thinks up all the books it contains.
Doesn't matter. (Score:2)
And being a government, these files are INCREDIBLY important.
Why haven't they been converted? Really, all their DIGITAL archives should be in a single format by now.
All those books are in a single format. And paper records can last a LOT longer than digital records. The
Re:Doesn't matter. (Score:5, Interesting)
Why haven't they been converted? Really, all their DIGITAL archives should be in a single format by now.
No, they shouldn't. You usually want 3 formats:
- the original format of the document. Whatever whichever idiot happened to write (or record, or video) it in, you absolutely want the original in your records.
- a searchable format (eg OCR'd text from scanned image docs)
- a rendered format. (eg an image or pdf, or svg - something open enough that you can continue to show how the doc would have looked). The appropriate rendered format varies. Paper is not an appropriate format for storing CCTV footage, for example
If you're very, very lucky the original is both searchable and viewable; like, say, HTML. It gets more complicated too, because you often want to store a redacted copy of the document (think of the Onion story 'CIA realise they've been using black highlighter pen all these years') and you want that searchable too, so you have to keep a redacted searchable format too... and of course, some of the records are on actual paper. Have you started worrying about the fading inks in the originals yet?
BTW you can't restrict the format of the original. Consider an email from a corporate bidding for a govt contract, with attachments. They need to keep those.
- Mr. E
PS, posting anon because I have dealings with the national archives, and don't want to speak for my company.
Re: (Score:3, Insightful)
1/2 pentabyte = 20 bits? (Score:5, Funny)
A pentabyte is 5 bytes, right? How hard is it to store 20 bits on paper? ;)
(I assume petabyte (10^15 or 2^50, depending on convention) is the word you're looking for.)
That's why I said "depending on convention" (Score:2)
Re: (Score:2)
Right, 1 pentabyte = 40 bits, which is why I stated that 1/2 pentabyte (taken from the GGP post, and in the subject heading) is 20 bits. 1/2 of 40 is 20. ;)
Re: (Score:3, Insightful)
Er, even if you translate it into other languages, they'll evolve too. Try reading Old French much? And translation also leaves you with the headache of reconciling various translations and figuring out which is "more correct" (IIRC the Bible has this problem). It would be a much better idea to make redundant copies, to guard against bitrot and store them as physically apart as possible.
Re: (Score:3, Interesting)
For example, I keep a copy of DOS and Win3.1 ISOs (about 20MB total) and Norton Commander (3 floppy images!) on a DVDR, along with a copy of Virtual PC. This lets me recreate a Windows 3.1 virtual PC anytime I want.
Now.... You can do that now. However, in 100 years, will this be possible? You do not know what the future brings. Let's not even talk about 1000 years and beyond. Now; you backed this stuff up on a DVD and you die tomorrow. Your kids keep the data, and when they die a historian speciali
Re: (Score:3, Interesting)
My copy of Office XP won't activate on any of the computers I currently own (the hardware it was originally activated on is long-dead), and that's only 5 years old.
Re: (Score:2, Interesting)
Along with Professional File (database product)....
Re: (Score:2)
You're suggesting the National Archives have the resources and intelligence (as in, research and know-how) of a single guy who found several 5 1/4 disks while cleaning his room.
Well, thanks. I laughed a lot.
Overestimate? (Score:2)
Re: (Score:2)
While it may be hard to find the program itself anymore, you can usually find something that can read the files.
Re: (Score:3, Interesting)
Which seems reasonable at a time when "everyone" has a computer that'll read it, for example when it comes to image viewers there's software covering literally hundreds of formats without issue.
2. idiots which believe they can't find even a single copy of the software they need
It's supposed to be an archive, not a "well we'll have to dig up a copy of the software, I'll get back to you in some months.
3. idiots who didn't st
Re: (Score:2)
If Microsoft has taken the job of taking some binary BLOB and make it into something human-readable, OOXML or not, then I say you'd have an easier time converting OOXML to something readable in OpenOffice than not.
If it's a text document, then you might be able to parse the OOXML regardless and understand most of the formatting. If it's a spreadsheet, then many of the parts of the OOXML spec are ALSO binary LOBs and you are no better off. If it's something that OOXML doesn't support, you are out of luck. At least ODF provides ways to package other formats along with itself in a transparent fashion, is completely documented and supported by multiple vendors.
Cheers,
Toby Haynes
Re: (Score:3, Insightful)
That's easier said than done. You'd have to keep multiple copies of everything, including hardware, up to the point where you're confident you have a stable standard - probably the power mains - and that's if you're not worried about violating licenses. Of course, with the advent of online apps, there is no way to snapshot the entire ecosystem of servers and softwa
Re: (Score:3, Insightful)
It's not just about the software... (Score:3, Interesting)
I'm sure that most of the archive data created today is stored on something like DVDs but, as recently as the early 1990s, the official long-term storage medium for the UK government was Syquest 44MB removable cartridge hard drives [wikipedia.org].
I know that I have a working 44MB drive (well, when I last fired it up, which would have been sometime last decade) somewhere in my attic but I doubt that too many of these drives are still in existance.
I only hope that the
Re: (Score:2)
Wrong! It was actually secret option C: accidentally drop the last Syquest drive, thus dooming civilization!
Re: (Score:2)
This is the case, especially with computer magazines. Sensible publishers will have used self unpacking executables and/or the ZIP format.
Finding a device to read floppy disks and CD-ROM's is straightforward enough. But trying to find the relevant application which runs
Re: (Score:2)
Re: (Score:2)
It's ASCII, of course. There's nothing wrong with using Word, VisiCalc, Wordstar or whatever, but just save a bloody text version of your document, and its guaranteed that some
Re: (Score:2)
You forgot the idiots who don't remember/were never aware of a time when there was no common platform, OS, hardware, or even, for that matter, alphabet encoding, and when nearly all files were saved as a dump of the in-memory structure for efficiency.
At my old print shop, we used to save all the cool/funny business cards that came through the door. And one of the nicest Helvetica versions I've ever seen ("Helios").
Someone probably still has all that. O
Re: (Score:2)
And if you suggest using Access 2007, just forget it, I already tried, and the documenter add-on is the biggest POS ever. This is using a legally installed copy of Office 2007 on XP Pro on my wife's desktop.
I am sticking with
Re: (Score:2)
Re: (Score:2)
In a hundreds years microsoft will no longer exist so what do the historians do when they uncover a stack of magnetic tapes/dvds that contains documents in
Re: (Score:2)
In a hundreds years microsoft will no longer exist so what do the historians do when they uncover a stack of magnetic tapes/dvds that contains documents in .doc format? (.doc or whatever other format you can think of.) They may have the hardware to transfer the data to newer computers and storage but the secrets to translating .DOC were lost years ago when microsoft went bankrupt..
It isn't that complicated. If historians can piece together egyptian hieroglyphics from the Rossetta Stone, then they certainly can extract some plain text from a .doc file.
.doc file, they probably won't be able to figure out opendocument any better, because it is just as silly to believe that opendocument will be any more common 1000 years from now than microsoft word documents.
And if they can't figure out a
Re: (Score:3, Interesting)
It rests on
1) Physical storage medium -- whether this is Flash, Hard Drive, Optical Medium, [NV]RAM, etc., all these technologies may be very difficult to retrieve data from, especially if the level of technology happens to go down in the future (say, global thermonuclear war). Even if data is retrieved, there's no guarantee that it's intact after 1000 years (the dyes in CDs will have decomposed by that
Re: (Score:2)
The question isn't if the format will still be in use, the question is this: How hard will it be to write a converter to whatever the standard is in the future? With ODF, you need to uncompress a zip archive and parse some reasonably simple XML. With OOXML, you need to
Use SGML (Score:5, Funny)
Use TeX (Score:2, Interesting)
Also, LaTeX will get you nicer documents than any WYSIWYG word processor in less time (once you know it
Re: (Score:3, Funny)
2:2 And on the seventh day God said
2:3 And God watched gcc running and sanctified it, because it would have taken Him at least two weeks to write the whole thing in machine code.
The big lie... (Score:5, Informative)
to give it a proper name, the format is "Microsoft Open Office XML", they deliberately went to a lot of trouble to pick a name that's as easily to confuse as possible with OpenOffice
Re: (Score:3, Informative)
Re: (Score:3, Funny)
You don't understand the format then. Office Open XML is the ultimate in
upgrades free money for MS (Score:2)
Open Formats (Score:2)
Obviously... (Score:5, Funny)
Oh yeah, their solution? Virtualised Windows 3.1. And obviously in 15 years you'll have to virtualise Vista in order to run the Win3.1 virtual machine to run Word. And Microsoft will be paid a license for each application and level of virtualisation.
You couldn't make this stuff up.
Re: (Score:2)
Re: (Score:2)
Microsoft are providing virtualisation so they can run old software in order to convert it into newer formats, not to have a load of nested virtualised operating systems like Russian dolls that have to be paid for in perpetuity.
The idea that an institution like the British Library, which is run by people bright enough to make you look like a dead match, would accept such a preposterous
Re: (Score:2)
You really need open / published formats for archiving.
Re: (Score:2)
Bright people don't make tech decisions (Score:5, Interesting)
Unfortunately, those bright people don't get to make technical decisions.
The British Library recently introduced SED [www.bl.uk], an electronic document delivery system. With SED, you can order electronic copies of journal papers and articles from their archives. Great idea! Previously, you had to wait for the documents to come through the post, and that would take a week or so. Now you get them by email in a couple of working days.
Except that the documents are crippled by Adobe DRM, which imposes the following restrictions:
- You can only view them using certain specific versions of Acrobat Reader (6 or 7) - the latest version is not recommended [www.bl.uk].
- The software only works on Windows 2000 or XP. No Linux support, no Mac support. Vista might work, but again, it's not recommended.
- You can only look at each document for a limited time, and you can only print it once.
So, if you want to use the service, you'd better hope that you have (a) the right version of Windows, (b) the right version of Acrobat Reader, (c) a reliable net connection, and, most importantly, (d) a very reliable printer that won't chew up the document. Unless you're a filthy dirty pirate, of course.If Adobe managed to convince the British Library to put up with this ridiculous system, I am sure that Microsoft will have no difficulty convincing them about their archive "solution". If SED is anything to go by, it'll be another awful implementation of a great idea.
Re: (Score:2, Interesting)
What about printing it on this [cups-pdf.de]?
Re: (Score:2)
Oh and by the way you can use Acrobat 8 now.
Re: (Score:2)
Ah, a minor improvement. It did strike me as particularly incompetent that Adobe's DRM scheme did not even work with the latest version of their own product, but then DRM is all about incompatibility and frustration for legitimate users.
Re: (Score:2)
Microsoft are providing virtualisation so they can run old software in order to convert it into newer formats
That doesn't make any sense. Microsoft already know the file format, just write a bit of software which will read in the old and write out the new format. We're talking terabytes of information here. it isn't as if you can just open each file manually and choose "export as".
The idea that an institution like the British Library, which is run by people bright enough to make you look like a dead match, would accept such a preposterous idea is insulting.
You seem to have a remarkable faith in institutions. They're planning to use OOXML. I think insulting them is entirely fair.
Let me quote:
Adam Farquhar, head of e-architecture at the British Library, praised Microsoft for its adoption of more open standards.
Stop and consider that quote for a moment. Given your vast experience of the history of IT, does
More surprisingly!? No, UNsurprisingly (Score:4, Interesting)
As "well intentioned" as Microsoft may be, Microsoft's Open XML cannot be anything but proprietary when its code references Windows and Office API functions rather than more precise data format information as with ODF. (For more information about this, you might search out the arguments against making OOXML an ISO standard.)
Re: (Score:2)
Doesn't open source solve this (Score:5, Informative)
Also--as noted, the OOXML format is a nonsolution for this nonproblem. It seems like it would be a waste of effort--why convert a bunch of files to a format that may die just as quickly as any other format, when you can just leave the file as is and open it in OOo (assuming I'm correct that they won't stop read support for dead formats)?
Also, it seems to me that no current format or any future format will ever solve this nonproblem because formats will always change as new functionality is continually added. The better solution is to keep this a nonproblem by having open source software that can read old file formats.
Re: (Score:2)
Well there is always maintenance work involved. Things change all the time, so does software. It could well be that in 20 years OOo won't support MS formats anymore for whatever reason, unless people actively
Re: (Score:2)
What needs to be done is for all these archival agencies
Real Issue (Score:2)
The only solution IMHO is _open and documented_ interfaces, protocols, programs, data types and hardware. In the future they won't be able to read our disks and files. They just can try to build a machine that reads our disks and files - for which they need documentation how they work.
surprise? (Score:5, Insightful)
Brilliant.
"What, the shit I sold you yesterday stinks? Try this new shit, it's great and it has none of the problems of the old one."
That's what you hire PR people for.
How about some *helpful* suggestions (Score:5, Insightful)
To kick things off here's one:
Keep EVERYTHING in the simplest possible format. ASCII would seem sensible, since its the content we care about, not the formatting. (although that wouldn't help our Asiatic brethren much). Then Keep decent records of HOW you can read that format. With examples of the software and hardware. do this bit on PAPER. V. Tough Paper (or rock, or plastic or whatever). Update the explanations every other year, to put it in language the next gen will understand. Maybe also have instructions on how to translate the simple format to less simple things.
I guess, basically, its a case of KISS and then *provide a persistent and regularly updated 'Rosetta Stone'* for latecomers to work from.
As a side branch, this kind of reminds me of discussions I read about a while back of how to warn future generations about Nuclear Waste dumps (y'know, the really nasty stuff with half-lives in the thousands of years range). I don't think anyone ever came up with a decent answer....
Re: (Score:2)
Comment removed (Score:5, Funny)
Re: (Score:2)
http://pardus-larus.student.utwente.nl/~pardus/pr
Re: (Score:3, Interesting)
ASCII would seem sensible, since its the content we care about, not the formatting.
No, the formatting is important as well. Sometimes 'the medium is the message', and that whole bunch of artsy crap we geeks would prefer to ignore. - Just think of it as an engineering challenge in order to make the pain go away.
You always archive the original (unless you have a batch; then you sample one and call it the original), and that original can be in just about any format, hand-written, coffee-stained, in sanskrit. When scanning a document into an electronic archive the ideal would be to have OCR
Re: (Score:3, Interesting)
plausible, and wrong." -- H.L.Mencken, The Divine Afflatus (1917).
Ok, you started by identifying one problem - asian languages. In fact, pretty much every non-US language since you said ASCII and not Latin1. So we can extend that to UTF-8 with no problems, except there's probably a huge table just for the 100000 characters or so, even though the spec is quite short.
But then, you have only characters, which is probably fine for basic text. How ab
Re: (Score:3, Interesting)
We fought a lot with this at Siemens (Sietec) about fifteen years ago, when trying to decide what format to use on stackers full of 12" WORM disks, which were just nicely becoming useful for large-scale archival storage in those days. We needed format that would outlast the disks, which probably meant 50-100 years assuming normal replacement/turnover.
We ended up with the bottom level being a WORM standard, which was served out to users via the NFS standard, which was reasonably close to a Unix filesystem
Re: (Score:2)
I think we're talking about electronic documents here, not about other mediums. ASCII isn't perfect, and it still leaves problems as far as binary d
Microsoft lecturing about open standards?!? (Score:2)
You can almost hear the slime dripping. (Score:2)
The real problem seems to be the credulous morons in charge of the National Archives project.
I've never understood this arguement... (Score:2, Informative)
Re: (Score:2)
solution (Score:2)
That emulator itself needs to be Open Source so that you can port it to future platforms. Otherwise, you'd be faced with running an emulator in an emulator in an emulator in an emulator in an emulator...
Keeping around multiple conversions certainly doesn't hurt. Converters vary in quality and the resulting conversions will themselves vary in future compatibility.
OOXML is not a solution (Score:2)
The OOXML spec is huge (expensive to implement from the spec) and complex, and the mean
Re: (Score:2)
OOXML is XML -- if you want to extract plain text from it just feed it through a XML parser and strip all the tags. You can do something similar with Office's format, but the solution will be far less perfect and contain lots of junk.
In fact, I just tried that. One of my
Re: (Score:2)
Thanks for explaining why OOXML is not a solution (Score:2)
Precisely my point. If the layout and non-text information in the file matters, then you've thrown it away. If it doesn't, then why are you bothering to put it in the archive?
You can do something similar with Office's format, but the solution will be far less perfect and contain lots of junk.
Yes, and (as I noted) I've done the same thing, and it's a relatively crude way of reverse-engineering
called "hiring the fox to guard the henhouse"(n/t) (Score:2)
Time bomb ? (Score:2)
Here in the colonies ... (Score:2)
Xena converts digital records into two preservation formats.
* Bitstream version. This is a metadata-wrapped bitstream version of the
You have *got* to be kidding me... (Score:2)
We live in an age when brand-new, undocumented, *encrypted* file formats are deciphered within days or weeks. You're telling me that in a few decades, NOBODY will be able to figure out a spreadsheet or word-processing document?
Re: (Score:2, Insightful)
Re: (Score:2)
1: I figured that to do this, you need to know how the 100-year-old system works. That's no problem now, since there's still enough of the old hardware (and its documentation) lying around. But some day, those will have turned to dust. Your archive better contain complete information on all the old data for
IBM (Score:3, Informative)
It has been a while since I worked on an AS/400 system... so anyone with updated info please feel free to correct me if things have changed.
It seems like a no-brainer.
Link: http://en.wikipedia.org/wiki/AS/400 [wikipedia.org]
Re: (Score:2)
so much for 100% compatibility
Re: (Score:2, Informative)
I can still double click on
For example, here is a version of Visicalc from 1981!
http://www.bricklin.com/history/vcexecutable.htm [bricklin.com]
Re: (Score:2)
How is it such an achievement on a platform of their own design?
Re: (Score:3, Insightful)
Re: (Score:2)
That's not backwards compatibility, that's just emulation. I can run C64 programs on Windows too using emulation, but it would be wrong to say Windows is backwards compatible with the C64. Windows is pretty
Re: (Score:2)
That's not backwards compatibility, that's just emulation.
Re: (Score:2)
(Not if. When.)
Re: (Score:3, Informative)
Yes, that was sarcastic, but you deserved it.
Re: (Score:2)
Re: (Score:2, Informative)