State of Optical Preservation?

[Joe]

I've had countless bin/cues where for some reason, with the full track listing, bchunk chokes on them, so I have to hand-edit the cuesheet to isolate the single ISO data track characteristics.

Why are you extracting an ISO when an ISO can't store Mode 2 Form 2 sectors? (And bchunk silently ignores this problem.)

Maybe my joke I've told myself about micrographs of a disc surface isn't such a joke...that eliminates the laser characteristics from the question entirely...

I've been tempted, but I don't want to spend a bunch of money on a microscope only to discover it can't focus through a CD.

[segaloco]

I've had countless bin/cues where for some reason, with the full track listing, bchunk chokes on them, so I have to hand-edit the cuesheet to isolate the single ISO data track characteristics.

Why are you extracting an ISO when an ISO can't store Mode 2 Form 2 sectors? (And bchunk silently ignores this problem.)

Not always ISOs, just a common use case e.g. pulling the ISO out of a Sega CD bin/cue to then pull the bootstrap out of the low sector. I probably should've said track, not ISO, thats just one type of track.

[Joe]

e.g. pulling the ISO out of a Sega CD bin/cue to then pull the bootstrap out of the low sector.

I feel like this might be a bad example? The bootstrap should always be at the same location in the bin, and the data track should always be MODE1/2352, so you should be able to extract it directly from the bin without using the cuesheet.

[segaloco]

Should? Yes. Can I? Not without either grabbing some external thing or writing my own tool. A simple dd off of the logical header isn't feasible at this stage, its still intermingled with the control data that isn't used in this particular case.

In my mind its like having to interpret a TCP packet vs. an HTTP packet. One is sliced up in the other. However, both have very clearly defined interfaces and wide, consistent support.

However, I can't treat a bin like a device file of a disc. There is a whole second file to deal with and the experience simply cannot be performed the same way as a physical drive. By definition a piece of physical media and a dump of that physical media should be identical. That means on UNIX-y systems you shouldn't have to fuss. Yet with CDs out of so many other formats, it is nothing but fuss. I can extract tape archives from 40 years ago on a standard GNU/Linux setup but I can't reasonably expect anything comparable for the optical medium? It just seems backwards, like the situation would've improved with each successive thing rather than been set back.

But oh well its established that bin/cue is probably still the way to go but it just feels so incredibly clunky and unintuitive compared to other ways of doing things. Comes off as "ehh, good enough" rather than an intentional solution...

[Joe]

However, I can't treat a bin like a device file of a disc. There is a whole second file to deal with and the experience simply cannot be performed the same way as a physical drive.

With a physical drive, you need special ioctls and/or raw SCSI commands to read the TOC and access the different tracks on the disc. With cue/bin, they're just normal files.

I can extract tape archives from 40 years ago on a standard GNU/Linux setup but I can't reasonably expect anything comparable for the optical medium?

The driving force behind those tools is mostly piracy, and all the pirates use Windows. Compare bchunk (a quick hack with comments in the code showing that the author doesn't really understand CDs) to CDMage (all-in-one CD image parser, converter, extractor, error-corrector, whatever).

But oh well its established that bin/cue is probably still the way to go but it just feels so incredibly clunky and unintuitive compared to other ways of doing things. Comes off as "ehh, good enough" rather than an intentional solution...

The CDRWIN developers came up with cuesheets as a plaintext representation of the SCSI commands responsible for reading from or writing to a CD, and using a single binary file to hold all of the tracks sidesteps most of the complexity (and nonstandard extensions) in cuesheets. I'd still enjoy hearing about a better solution, though, even if it's never going to catch on.

[segaloco]

In my mind the better solution is this: there is a groove, that starts at one physical place on the surface and ends on another physical place on the surface. That groove is then made up of either a marked or unmarked portion per sample in the waveform.

What I want is a file where the first sample is array member zero, the second sample is array member one, etc. until the last sample which is then array member n where n is the number of unique value sample points in the linear array of data that CD represents.

Whatever *that* is, that is what I want. If there is enough physical information on the surface of a disc to describe that disc *in totality* to a laser, why can't that information possibly live as one file that people can then target with things like filesystem drivers for mounting. Part of the goal would be to use "mount" to inspect images, not the mountain of arcane CD tools out there. That I can't take a thing that is supposed to be a backup of a physical media I can mount in UNIX, and yet the prominent file formats for backup I *can't* mount in UNIX, that is a *direct* contradiction of "everything is a file" because the disc when accessed via /dev and the disc when accessed from all these garbage formats *behaves differently*.

The Windows world may be fine with having 30 different tools to do the same thing, but the point of UNIX is to avoid that complexity and have one best way that you don't have to keep re-learning over and over and over again. I love how daily I get to be reminded how Microsoft ruined computing...

[stan423321]

Dude, this is seriously getting unhealthy. POS*X people historcally didn't care about weird CD archival, so you're going to blame... Microsoft... for not forcing everyone to use a minimalistic monoculture of tools with no alternatives. Having multiple good options is an advantage! On modern Linux you have sed and awk and sh and perl; having one option would be something like just awk. Imagine forgetting about someone writing python or bash or zsh, because that would be a different program for handling Turing complete uncompiled programs, slightly different domain be damned, and you gotta use awk, because awk is the one best way to code, and POS*X defended itself from Microsoft infecting it with options. Am I strawmanning here? I genuinely don't know.

---

With that out of the way, let's explain why your raw laser idea for image format is not something that got popular in any known public spaces, through contrived analogy.

Imagine instead of CD drives in players and computers, you have magical demon summoninators that scan arcane parchments. Specifically, most of them have a more or less separate part that moves sheets in front of a simple camera and convert them into fixed resolution JPEGs before the main devices using them to decide what demon to summon (and presumably how) if any; we're going to call this auxilliary subsystem a "scanner". Compatibility of summoninators with scanners and parchments varies aside from very basic color parchments, but most scanners use very similar cables for control and reception of either color or grayscale JPEG encoding in a fixed resolution.

Some people want to preserve all those parchments in an operational state, for a variety of reasons. "Preservation" attempts with intentionaly unusable results shall henceforth be ignored.

The people we will call "cheapskates" decide to rip out a scanner from a summoninator and connect it with the standard cable to a microcomputer, then save the JPEGs from logical positions over the sheet, and also fetch some sheet sizing data from the scanner part which moves the sheet, after it turns out the latter is needed for some of scans to work.

You instead consider that a single higher resolution lossless picture is what can be perceived from the sheet, so that's what you need to preserve. Your first problem is that there is no publically available scanner that just does this; you need to make your own for a demonload of money or seriously hack up an existing one, instead of just using the cable. Your second problem is determining the exact required scan resolution, unless you're hacking up an existing scanner, in which case there's a low hanging hardware limit. A very consistent one amongst current scanners, but future ones may vary of course.

Your third problem is that you do want the scans to be usable. So what you need to do for summoninators you can get your hands on at the moment is either: grab the standard scanner cable, connect an external programmable thing, simulate sheet camera displacement, and then apply the parchment JPEG encoding, which means there is zero perceptible difference compared to the cheapskate scans; or hack the firmware to use your simulator instead of the cable connecting to external hardware, and keep doing the rest of the above, which again means no apparent difference. Or, for people who insist on using paper copies instead of relatively expensive and invasive scanner emulators, you need a really high quality printer that will bring absolutely average results, which means no visible difference once again.

Your fourth problem would be that people complained your scan files were big, but let's say, you just wisely start trying to construct a quality scanner for reasonable price, while quietly observing what's happening with cheapskate endeavours. There is a lot of bad scans that you think your approach would defeat without a problem. And the cheapskates take their time with cleaning them up. But eventually they do clean them up while your scanner's not yet ready. Accordingly, original parchment makers notice that some cheapskates distribute bootleg parchments or teach people how yo scan things.

Eventually, Sony DemonStrat1on summoninator comes to the town, and DemonStrat1on-only parchments surprise you a bit because the cheapskate scans look like they should work but they don't. It eventually turns out there is an ultraviolet lamp used for finding a positioning crosshair in known scanners, and DemonStrat1on's built-in scanner has a special command to reuse it and look for a watermark. The scanning resolution for visible colours is the same as it always was. While you're trying to figure out how to scan or print this nonsense in breaks from regular high quality scanner development, some cheapskates hijack the cable to say a valid watermark is everywhere and continue operating.

Eventually things get ugly. There are parchments that get slightly different JPEG artifacts depending on exact positioning movement sequence (they fold slightly) and rely on it. That, your scanning could handle, but you need a very good model of sheet movement for it to work. In fact, some summoninators don't work with original parchments for those. Cheapskates decide to draw over this nonsense and get a mostly working demon instead of thinking about it. They do the same thing with DemonStrat1on parchments that get weird results with too many watermarks.

Then DemonVsDemon summoninators happen and render your unfinished scanner obsolete. Your planned picture resolution is alright, but there are some implementation details that are a bit different. Cheapskates just take a look at the new cable and get scanning quickly, this time in JPEG 2000, since the cable outputs that now.

---

This may sound like nonsense I'm making up to make your argument sound bad, but this is really close to my understanding of how CD images work and how you would want them to work instead.

The operating systems don't talk to discs or lasers, they talk to the entire drive. Paradoxically, this means part of the problem is that there is one "best" way for all the CD error correctiom decoding: the drive does it. This doesn't work nicely with images, but hey, that's one best way's conditions.

[segaloco]

But what does that do for preservation? What does any of this do for preservation? Where are any of these supposedly clever, inventive engineering hacks serving literally anyone but their inventors? There are more people than them in the world. Arguably there are infinitely more people in the future than these self-serving engineers. Where do they get off?

Edit: And in case you're missing the point, I'm on about this because I care about OTHERS abilities to also work easily with optical media preservation. Yes I myself can jump through all these stupid hoops, I know how to, unfortunately, because of how BAD the state of optical preservation is. With these barriers to entry, how can others feel encouraged to get involved if they have to learn ALL OF THIS to interact with preservation of media. It's gatekeeping at its finest and *only* serves to prevent people from doing things. Build bridges, not walls, geeze.

[Joe]

Whatever *that* is, that is what I want.

That's a raw EFM capture. I guarantee you're not the only one who wants it. It'll be three times as big as bin/cue, though.

Part of the goal would be to use "mount" to inspect images, not the mountain of arcane CD tools out there.

...Um, what exactly do you expect to see when you mount a raw EFM capture?

[segaloco]

Whatever I'd see when interacting with the /dev entries created from the CD on a system that supports it. Sure, there'd be the matter of porting filesystem drivers from other platforms for CD filesystems and arrangements not supported in, say, the Linux or FreeBSD kernels, but that makes this an *option* as opposed to the current, patchwork state of affairs. That is the ideal, for me, of any filesystem backup mechanism, is that the resulting file can be presented identically in a sane environment like a UNIX system and be indistinguishable from what the drive that created the backup was fully capable of sticking in the /dev directory. That would be simple and intuitive in my eyes, because it would solve the goal of having an easily accessible backup mechanism. The current state of affairs is far too specialized and has higher barriers to entry than most other representations of filesystems I've dealt with on UNIX.

[Joe]

Whatever I'd see when interacting with the /dev entries created from the CD on a system that supports it.

You could do this with existing formats. I don't see how raw EFM would help.

Although actually, I'm not sure if you could do this at all, with or without raw EFM. How do you replicate needing a special ioctl to read the TOC? How do you replicate ordinary reads returning I/O errors outside of Mode 1 and Mode 2 Form 1 sectors?

[stan423321]

But what does that do for preservation? What does any of this do for preservation? Where are any of these supposedly clever, inventive engineering hacks serving literally anyone but their inventors? There are more people than them in the world. Arguably there are infinitely more people in the future than these self-serving engineers. Where do they get off?

Edit: And in case you're missing the point, I'm on about this because I care about OTHERS abilities to also work easily with optical media preservation. Yes I myself can jump through all these stupid hoops, I know how to, unfortunately, because of how BAD the state of optical preservation is. With these barriers to entry, how can others feel encouraged to get involved if they have to learn ALL OF THIS to interact with preservation of media. It's gatekeeping at its finest and *only* serves to prevent people from doing things. Build bridges, not walls, geeze.

I think you're running into the problem that "preservation" in computing context is a word like "freedom" and "justice", for which we all have a very rough common general idea, but the details may vary wildly and even put the results in opposition - for a relatively flamewar-free example, "freedom to drive a car everywhere" and "freedom to not own and maintain a car to get everywhere" are both interpretations of "freedom of transportation".

You are going to need to be a little more specific when you say "what does any of this do with preservation", because for me it is blatantly obvious. Disc drives with discs work like this, to preserve the operation of stuff using disc drives when they break we also need to act like this, and that naturally leads to file formats laid out like this.

Other people can just accept that's how the formats are and use them, and they often do. I used zip files for a decade before having any idea what a Huffman tree was, general computer programs a decade before knowing how to program anything, and I still watch movies without a burning need to figure out video codecs and multimedia containers. You're trying to get into the details of a CD, which is great, but that's on you.

Whatever I'd see when interacting with the /dev entries created from the CD on a system that supports it. Sure, there'd be the matter of porting filesystem drivers from other platforms for CD filesystems and arrangements not supported in, say, the Linux or FreeBSD kernels, but that makes this an *option* as opposed to the current, patchwork state of affairs. That is the ideal, for me, of any filesystem backup mechanism, is that the resulting file can be presented identically in a sane environment like a UNIX system and be indistinguishable from what the drive that created the backup was fully capable of sticking in the /dev directory. That would be simple and intuitive in my eyes, because it would solve the goal of having an easily accessible backup mechanism. The current state of affairs is far too specialized and has higher barriers to entry than most other representations of filesystems I've dealt with on UNIX.

That's because all of this is not really just a filesystem, definitely not an idealized UNIX filesystem.

The filesystem part of CD-ROM is well defined and can be mounted and easily backed up. The error correction layers are lower level than that. The filesystem part relies, for example, on numbered sectors, while the lower level laser lacerations provide them.

The track system in the middle has certain similarities to a primitive file system, and future disc formats got rid of it, but it still carries certain lower level properties, especially the whole mode thing.

Similar stuff is happening in HDDs and SSDs of today. Some SSDs even have area modes of lower and higher density. They just facade it away from the rest of the computer while CDs allow us to take a look.

[segaloco]

Other people can just accept that's how the formats are and use them, and they often do.

Others ignorance is no reason to let the past slip through our fingers. I don't know how people can put such blind faith in systems like that. That's part of why I write so many of my own tools, I don't want someone else's stinky system that I can't audit for bugs, and fix them when I find them. That's why I publish my stuff open source, so others can find my bugs and fix them.

That so many people gleefully use computers every day without knowing how the sausages are made is absolutely horrifying to me. I would be completely paralyzed trying to use computers if I didn't know what I do about them. Sometimes I wonder how the uninformed do it...

[stan423321]

Is that negative reaction to ignorance specific to computers?

[segaloco]

Is that negative reaction to ignorance specific to computers?

Hardly, computers are just one iteration of the idea of a "system" of some kind that most people put zero effort into *actually* understanding. Instead, they all put their blind faith in these things that they don't understand, I have to come along for the ride despite knowing better. Take cellphones for instance, I've been the token "they're surveilling us" nutter amongst my friends for a long time. It's one of the many reasons I outright ditched the idea of a cellphone 5 years ago. Annoyingly, despite most folks I know understanding that I know a great deal about computers, they don't want to hear any of the technical explanations or proof, and instead keep choosing to live in the blissful ignorance that the bug in their pocket is their friend, that the companies make them care about us at all.

I feel the same way about cars, about the concept of hierarchical employment, heck, the idea of employment in general, the idea that there is this thing called a job you have to do to survive, because our survival has long been abstracted from what resources we can actually pursue and instead is represented by a series of arbitrary tokens produced en-masse by some unaccountable (voting is not accountability...) entity. I don't need to go on my "everything is fake and arbitrary" rant again because I think that is pretty well known, but the complete lack of critical thought by folks out there in the world taking everything at face value and questioning none of it makes me kinda sick when I then look at how much effort I put into understanding and questioning everything just to be able to go outside without wanting to scream. At any given time I am surrounded by hundreds of computers *THAT I KNOW ARE SPYING ON THEIR OWNERS* and I can't do anything to mitigate it, through actions or words. At any given time I am surrounded by plenty of people who every day make decisions that cause living things harm but add to the counter of magic economy tokens some faceless entity keeps track of, because that matters, not people's immediate needs. The fact that all of this is going on around me, that I know it, and that routinely the norms of our society have only served to ensure there isn't a way for me to do anything about, it gives me the creepy crawlies.

Think Ring doorbells. Most people getting a Ring doorbell don't do *any* research into the relationship that Ring has with law enforcement. It doesn't even cross their mind that by buying a product from organization A, they are automatically opted in to being an informant for organization B. In this case, A is a private company and B is a public entity, but what stops B from also being a private entity, what stops someone from sharing something with an "ad partner" that is then going to engage in dishonest and deceptive marketing attempts (if we're lucky) with that information.

Do you really want your personal information to belong to Black Rock? I don't. Of course this is just another example, but hopefully I'm getting the point across that societies collapse when they are built on constructs that none of the common folk understand nor can maintain. I don't like the idea that if someone struck the main power generation facility in a region, and everyone was at work, now suddenly our ability to rebuild is not only hampered by the physical destruction of the resource, but by the loss of life by those who actually knew how to do anything with the technology. So I learn everything I can, where I can, because I never know when some disaster might strike. Our current system of nobody ever being expected to know anything means I literally can't rely on anyone in that situation, by definition, because our society rewards being ignorant and aloof.