State of Optical Preservation?

[segaloco]

At what point are the engineers paid enough times more than whole other people to do these things in a way that is actually accessible? Is their being considered valued at twice or more than other living humans not enough? I'm so tired of "but its hard" from engineering. Hard is scraping for food on the streets. My expectation is literally just "You think you're worth two or more of some other people? Put on your lab coat and fix this or shut up."

Engineers are not magicians that deserve no expectations. Eventually we need to expect the things made for us actually serve us, or we need to rethink the pedestal on which we place the barons of half-assed-stop-gaps-turned-solutions. That we continue to accept this good enough is so insulting to how many people starve so these luminaries of industry can ride on half-specified products and ignorance of consumer expectations....this is how we stratify our own working class, making garbage for each other while profit is made off our backs either way. The least we can do is not have such stopgaps be the legacy we leave for each other while our bosses laugh all the way to the bank...

[Pokun]

Well, there's a limit on how much you will be able to do in a given amount of time, so meeting the expected goal will have priority. Long-term planning of what consequences your work may have for people a 100 years later would have very low priority.

[Joe]

Sounds like serial transmission over modems being an adjunct of the voiceband POTS service rather than something catered to in QA from the beginning.

That's a remarkably apt comparison, considering that early audio CD players would happily allow you to listen to data tracks. Some old data CDs have audio hidden in their data tracks, which - thanks to the scrambling - looks like random gibberish if you don't know to look for it. CD-i games use hidden audio to warn you not to listen to the data!

What is the point of the Rainbow books if not to be prescriptive of the data formats? It should be unambiguous enough that there isn't wiggle room for these weird little variations in the first place.

Most of the weird little variations aren't allowed by the standards, but they were used anyway either because the discs didn't need to work outside of their target device or because it made the discs harder to copy. The one place where the standards do have wiggle room is the alignment between subcode and audio, which is perfectly reasonable when you consider how hard it would be to notice even relatively large offsets between the two while listening to a CD and how expensive it would have been to build a CD player with enough computing power to decode the entire bitstream in a single microcontroller.

Ironic, given the point of a memory format is to preserve information...

And it works perfectly well for that. But what you're asking for is not just preserving information, but preserving all the information the disc could contain. Do you have the same complaint about dumping the raw flux transitions off of floppy disks? Sure, many disks don't actually need that level of detail to preserve everything, but some do, and sometimes you find information that would have been lost without ever knowing it was there.

[Pokun]

I think CDs was designed as a medium for distributing audio, not necessarily preserving it. Though the history of old phonograph records going bad must have been in the mind of Sony and Phillips when they designed it.

early audio CD players would happily allow you to listen to data tracks. Some old data CDs have audio hidden in their data tracks, which - thanks to the scrambling - looks like random gibberish if you don't know to look for it. CD-i games use hidden audio to warn you not to listen to the data!

Yeah it's pretty common in games for many older CD-based consoles. Most famously games for the PC Engine (which was first with using the CD-ROM technology for games) often has an audio track of humorous dialog of the game characters telling you that the following data track can harm your CD-player. It's also done with games for Mega CD and even Saturn and PS1. I specifically remember my Akumajou Dracula X: Gekka no Yasoukyoku PS1 disc having a funny message like this reminding you about the same message in the PC Engine prequel.

[segaloco]

Well all things considered...this all makes it sound like sabotage, like some engineer devising a way to mess with future historians and literally *nobody* in the process thinking but wait aren't there technical problems here? Our species knack for trying to solve problems after they happen rather than forethought and proactivity is only going to keep getting us in these messes...when does it finally start to bug anyone enough to change?

[stan423321]

The CD format is widely known to be unlike current standards for designing storage media in some aspects, but that's not preservation sabotage. Having a restrictive DRM built into the format could be sabotage, including some subtle details independent implementations are bound to get wrong could also be sabotage. This is just someone making things up with relatively new technology and not guessing your preferences in advance, while having other priorities, like making a basic audio player with functionality cuts include as little RAM as possible or something.

The CD image formats are a bit of a mess because from perspective of modern professionals they're at best developer tools. And developer tools have always been allowed to have rough edges. So any improvements come from amateur environment.

[segaloco]

Adequate preservation isn't some petty preference that I'm trying to hoist on others but okay. I swear, I just want quality in a thing that matters *to other people besides me* and its still treated like I'm the one who is unreasonable.

Well conclusion I have come to is optical media preservation still sucks and still seems to not bother anyone besides me...I'll give it another decade and try again.

[Joe]

Adequate preservation isn't some petty preference that I'm trying to hoist on others but okay. I swear, I just want quality in a thing that matters *to other people besides me* and its still treated like I'm the one who is unreasonable.

Trust me, you are far from the only person who is upset about this.

Well conclusion I have come to is optical media preservation still sucks and still seems to not bother anyone besides me...I'll give it another decade and try again.

It's not that it doesn't bother anyone else, it's that the engineering hurdles are higher. Taking the floppy disk example again, floppy drives outright give you the waveform from reading the disk on one of the interface pins. As long as your drive is close enough to correct for the disk you're trying to read, you can record the signal coming from that pin and decode it into a perfect copy of your disk. If you want to do the same with a CD, you need to either find an appropriate test point and attach a bargain-bin Domesday Duplicator or modify your drive's firmware.

Incidentally, the modified drive firmware angle may be your best bet, as some drives allow uploading temporary firmware patches. A popular "backup" application uses temporary firmware patches to bypass Blu-ray copy protection.

[segaloco]

I guess I'm just bothered by the inaction then. I'm bothered by how many decades of this and nobody has tried to solve it better? When I run into a gap in my projects, I fill those gaps, rather than sitting around waiting for someone to hand me the tool I need. I *could* do that with optical media too, but the problem then becomes can anyone use my stuff since I've had to get *so* custom with it? With my disassemblies, tools that are missing, gaps that are present, they're easy to fill because I just spin some code and include it with the project.

With this optical media matter, does that mean I have to distribute special tools with all my images to support my custom format I've had to invent because nobody else would? That sounds like I'd just be inventing another .cdi or .mdf/.mds, etc. some obscure format with precisely one extraction tool and no support. In the end it particularly sucks because I don't need any of this for my personal use, I can get by just fine with my strategies for my own work, but the second I try to go out there with something, that's when it all falls apart. That's why, for instance, despite how much I would *LOVE* one, I haven't written an AT&T-style assembler for 6502. If I do all my work targeting my assembler, that shuts everyone else out unless they want to install the 10th 6502 assembler running on their computer. I only want one 6502 assembler, who am I to give someone yet another CD image tool they have to install with the 10 other ones to handle the other bizarre exotic formats that apparently also didn't solve the problem trying to be solved adequately.

It's soul crushing knowing that a community solution would have wide acceptance, and all the community solutions are so bad, and that I could try and write a sophisticated solution, but nobody would use it...

[stan423321]

To be clear, I don't think it's unreasonable to want the data to be preservable. I do, however, think that claiming "sabotage" on the entirety of the data not corresponding neatly to a single binary blob is a big stretch, especially when you bring up INES as a relatively positive comparison. The only thing it really has over cue/bin is being a single file; you can tar the cue and the bin together to get that improvement.

INES mapper field is a big hack for which most of the world agreed on a more or less imaginary giant list of particular numbers corresponding to particular additional circuits, working roughly as expected. Most of the time that's enough to emulate the games, but sometimes it is not. Now obviously I am not saying INES should include the entire space of PCBs that can be put into a NES and work, since NES emulators would need to become Verilog simulators or something at this point. But it's obviously representing only a subset of what the cartridge can be, sometimes missing nuances (like MMC3 variants). The thing is, this subset is useful.

No one to my knowledge made a CD shaped device with a coprocessor that would supply different data based on access patterns or something, so that is thankfully not a thing we need to consider. (I am sure, in comparison, it is totally doable with casette tapes of various kinds, and probably cased floppies.) But there is a standard data sanitisation step done in hardware. Historically, professionals would probably be very much against allowing wider access to the rawer data because of the copy protection schemes this would help reveal. Nowadays, there aren't that many CD professionals, and most of their corpo inheritors are presumably against widespread preservation for obvious monetary reasons. So the standards would indeed need to be community based.

The community approach doesn't lack motivation, but comes with its own problems. Firstly, CDs are kinda a varied thing. People with actual important data stored on CD-ROMs will likely be fine with just an archive of the listed files, or an ISO image if they're very peculiar about some meta details. People who listen to typical music will be happy with tracks ripped to individual files. Cue/bin covers most of the remaining complications that don't involve downright disregard to the specs. So for many people, the currently covered subset is alright, just like a SNES ROM image without coprocessor firmware attached is good enough for 99% of high-level SNES emulation.

The people who really want something rawer are basically CD game enthusiasts that don't like software cracks, and preservationists that just want to preserve all the possible details without a particular need for a use case for all of them. This is not a particularly good environment for developing a more precise image format or means of acquiring more precise dumps, since the latter group is smaller, but when it comes to discussing things like new image format without considering the rest of userbase, it may feel wrong.

Secondly, the hardware sanitisation step is a real pain to bypass for amateurs, when a big corpo could just order a rawer drive.

Things someone should probably correct me on:

I think the second pain point is why redump people have a certain fixation with Plextor drives; maybe they figured out how to bypass some of the processing involved, or it is just more deterministic between the Plextor drives than between the entire ecosystem. But I don't they bypassed everything.

I believe someone relatively respected in the emulation community one day wrote something to the effect of "cue bin doesn't save nearly enough data, here's a rough spec for a raw format that has data before ALL error correction", and then no one figured out how to write a dumper for that format that didn't just make the details up.

[segaloco]

Something that would help my pessimism would be if the cuesheet format was better enforced. If I can boil my frustration with bin/cue down to something even more pertinent than having a whole second file is the fact that the cuesheet is so easily corruptible and poorly implemented all over the place. No I don't have an example I can discuss at length right now, but I wouldn't be upset about it if it wasn't the truth. Any time I see a bin/cue image these days my mind immediately floods with oh god what am I going to have to change in that cuesheet for, as an example, bchunk to actually break it up properly? I've had countless bin/cues where for some reason, with the full track listing, bchunk chokes on them, so I have to hand-edit the cuesheet to isolate the single ISO data track characteristics. Then, if I'm lucky, the iso gets pooped out. I have even had dodgy luck with that, to the point that I actually wrote my own quick and dirty ISO ripper that simply assumes the first track in a bin/cue *is* an ISO and simply discards all the control snippets around it. I've had ISOs I could only extract from bins that way, because contorting the cuesheet into every adjustment I could think of still didn't work.

And these weren't backwater cuesheets either, these were the "good" dumps from some archival effort. That's just my experience with bchunk, I only even know what I do about cuesheets from over a decade and a half earlier, playing with Sega CD games in emulators, I remember even then I often had to tinker with cuesheets to get things right. I don't know how to better express that if I have to hand edit a file format for each program that consumes it, it probably isn't a very good file format....

[stan423321]

Yeah, you don't need to convince me about cuesheets compatibility varying. I don't know if there is no formal spec, if there is one but lacking in features so someone made extensions to it, if there is one with good support for all the bells and whistles but many programs only supporting the simplest sheets because that's what discs do, or if something else is happening.

I see this as one of disadvantages of plain text formatted files. The temptation to make human conversions of files a part of the loading procedure exists, and certain informal expectations may appear or disappear depending on the reader.

[segaloco]

Its kinda the same story as IPS with me. Nobody will fess up where the spec came from, people are just expected to trust the process. Something is *made* at some point and comes into an existence at some specific moment by human hands. That ubiquitous, decades old formats like this have no standard backing does not help the situation any. I really don't feel like I'm the one doing something wrong here by wanting this technology to be consistent and actually serve people...

[Pokun]

I think Redump recommends Plextor drives just because they are high-end drives with a long history of testing and are confirmed to be able to dump lead-in stuff. It's to make sure all dumps follows the same standard procedure.

The IPS format is pretty much as old as the emulator scene which means the early '90s. Probably not many of the few people who were around back then are still here, I'd bet it even predates Koitsu joining the scene.
It's a very simple format made to serve its original purpose and it just works. Its flaws have been addressed by formats like xdelta and beat (BPS) which can be used whenever you need to do things IPS can't or you just want some extra security.

[segaloco]

I only mention IPS as an example of another long-standing format with a lot of custom extensions and variations that have been grafted on over the years. I found myself obsessed for a little while with trying to track down the origin of the IPS format, and apparently it is unknown, speculated to have possibly derived from something in the Amiga sphere of influence, but inconclusive. In any case, this just then becomes a lamentation in general on a lack of standardization.

I still feel there are people who could make decisions to the contrary of our current state of affairs that truly did not do so out of greed and malice, rather than ignorance of how something would affect others rather than just their profits. That is not what our efforts are going to be remembered for. People recall products much easier than recall how much they cost or that company's annual reports. And yet its those dollars that drive the day to day while the quality and/or utility can only be appreciated through the rose-tinted lenses we so often look to the past in.

To spin things I guess, would an optical format built preservation-first be worthwhile? Something that is much easier to simply use for binary data, rather than bastardizing some format meant for a completely different kind of reproduction of the data?

Tapes have been exploited for computer data storage in more than one fashion. You have the huge magtape reels of the past, often seen in period pieces from the 50s-70s. Those have their particular patterns and marks used on the tape to indicate records, end of tape, and so on. But then you also have things like data recorders to tape, transcoders that convert data into voiceband noise, basically modems, writing that audible data to tape instead. And there are all sorts of other encoding schemes, with the common thread being a linear function of some fluctuation in a detector. For the CD, could a different angle have lied in the same technique, simply using the adjustable physical properties of the hardware to do something completely different than restricting oneself to the confines of where audio grooves read?

I suppose that would kinda already be what some of these exotic modifications are, except they were often done in tandem with augmentation of the hardware to be unique in their tolerances.

Maybe my joke I've told myself about micrographs of a disc surface isn't such a joke...that eliminates the laser characteristics from the question entirely...