ZapFC Headerless Format

[FitzRoy]

I've spend the last few months on a format that would make virgin data playable by way of emulator-side mapper database. Nothing will ever become as popular as iNES 1.0 because it was the first to be good enough, which is how all standards are determined in emulation. But unlike iNES 2.0, I think this format actually has a chance to be used by some people. And that's all we need to get the games preserved, correct boards and all. Supplied in the zip file is a prepared database and an explanation of the format, why it was made, and how it works.

http://www.zapatabase.com/ZapFC_Format_Guide.zip

A full converted library does exist and that's all I'll say about that. This topic actually seems to come up every year or so, the last time by etabeta. But no one seemed to get to the point with why headers are so disliked by some. From the guide:

Code: Select all

This format allows virgin data to function by way of external database checksum lookup. The main problem with rom-side mapping like headers is its inability to effectuate revisions to the format. When an oversight is discovered to affect certain games, as it was with RAM quantity in iNES 1.0, you're offloading mapper update responsibility to users who won't do it with tools that don't exist.

The problem goes deeper than that. Distributed rom files are based on databases like GoodNES or No-Intro, so how they checksum files is a determining factor in whether the rom you download will play properly. GoodNES is 7 years out of date. And if you were to download the latest No-Intro set, you'd find that many of the roms have wrong or missing headers. No-Intro ignores headers in validation, probably because iNES 2.0 files would otherwise go unrecognized. But by doing so, they greenlight files that won't work in emulators. It's a double-edged sword with headers. You can either ensure data integrity or mapping integrity, but never quite both.

The ZapFC format places no such burden on users because mapping is 100% emulator-side. As long as the user has his roms in this format, game issues stemming from missing or inaccurate headers are no longer a possibility.

It's also a great format for dumpers. We would no longer have to use special tools with offset magic to obtain and compare checksums of just the CHR or PRG sections.

I'll also address the "unlicensed" issue since I know it will get brought up.

Code: Select all

My stance on database inclusion is licensed, released games only. I exclude unlicensed games for the same reason that emulator authors don't bother emulating the endless streams of unlicensed add-ons and clone systems: they are a maintenance nightmare and I don't think we should be making inclusion decisions based on subjective quality appraisals. It's like Project Gutenberg trying to catalog everything from self-published books to the notes you took in math class.

We all know that it's not possible to include infinitely creatable material, and licensed games will suffer for it. The database is there if you want to extend it, but that quicksand will sink you like it did Cowering. The solution is to either (a) continue using headered frankenroms for unlicensed games or (b) give option on failed database lookup to add custom db entry with a mapping selector. 

Then there's something neither iNES nor an internal db can accommodate, which is people who want to make not only their own games, but their own hardware. Since this is pretty niche, and would be a rom-side format prone to revisions, this should be its own format. Though I doubt if anyone truly cares enough to spend months preparing one.

As for betas of licensed games, I am more sympathetic and could add them in the future. But betas are one-of-a-kind and nearly impossible to verify. How do we know a beta was unmodified and properly dumped?

Anyway, there aren't too many active NES emulators left. Not sure if anyone will support it, let alone see things the way I've come to see them.

[kyuusaku]

iNES successor proposals are like * everyone (here) (probably) has one.



*SMB/DH

[Wkter]

A database with all the official games is a great idea, as we can guarantee they will at least have the correct information supplied, however, I don't think the format you're proposing is the way to go. We don't need a format exclusively for licensed games. It would be better to find a universal format that supports both with headers (Unlicensed and homebrew) and without (Licensed games that use a database).

Personally, I don't generally like databases and cheksums (No database = no play), but I support the idea to at least have it available for those who want to use it.

While I like the idea of zip and separate PRG and CHR files because it makes it easier to edit them, I don't see any reason to have them separate just for the sake of having them separate. If your file format doesn't allow for them to be edited, I don't see any reason to keep them separate. Also, you must keep in mind that not every platform may be able to unzip and generate the checksum within a reasonable time due to some crazy restrictions (Like slow processing power or very little memory).

A new file format must also be truly portable for it to be used, thus I would like to see a hybrid of both headers and database (If the database isn't available, it can fall back to the header with the ROM).

However, unlike some people here, I support moving to a new file format, but it must be able to have enough information to be run as a standalone ROM, and it must be portable enough to support slow and memory restricted platforms. The impression I get from your suggested file format it that is lacks some these things I'd like to see in a file format.

[LocalH]

Perhaps one should look to other systems that have specific setups for a given ROM/disk image (VIC-20 comes to mind as some programs only run in certain RAM configurations, and fail if you have all possible expansion RAM installed) and draw some inspiration from there? I'm more of a C64-head so I don't really know how the VIC-20 emulators implement this (or if they just fob it off on the user). The NES is unique in this regard for it's widespread use of external memory mappers, so I'm sort of torn in the middle. On the one hand, since most header settings are unique to a ROM (or class of ROMs that use the same board), I can see the argument for keeping everything with the ROM. On the other hand, you make some valid points for not storing the header information with the ROM.

Unfortunately, I truly fear it may be too late to even make a dent regarding this issue. iNES 1.0 is so widespread that Nintendo use it in their own products (same is true for the other VC-emulated systems, AFAIK most if not every one of them uses an emulator "scene" standard to store the actual ROM data).

[tokumaru]

I hate databases...

[3gengames]

What's so bad about a 20 byte header that then offers offline play without anything else? How this would be a good idea? Yeah it's interesting and may work okay (What emulator has an internal database of checksums again?) but it won't be any better then the iNES header and will be worse for other situations like homebrew, what is supposed to happen in between compiles and revisions and such?


It's not a very good idea at all. iNES is the standard for a reason, it's the best way to do it. Why are there attempts to even change it?



Mind=BLOWN

[tokumaru]

These solutions oriented to "software preservation" (i.e. piracy) are a joke. It's been over a decade since the last licensed NES game was released and people are still looking for ways to properly catalog them.

I think there's no point in coming up with solutions that solve only half of the problems. If you are going through the trouble of creating a new way to describe ROMs, please create something that will work for 100% of the cases, including homebrew.

I would very much appreciate a format that allowed for hardware customizations, such as describing the functions of individual chips and specifying how everything is connected on the board, so that we wouldn't be restricted to built-in mappers. This would also solve a very big problem in the homebrew scene, because nobody wants to program games for mappers that don't exist and nobody wants to create a mapper for which no games exist, and maybe we'd finally have interesting and easy to produce homebrew mappers.

[Wkter]

I would very much appreciate a format that allowed for hardware customizations, such as describing the functions of individual chips and specifying how everything is connected on the board, so that we wouldn't be restricted to built-in mappers.

This would require more than a new format, it would require a new way of writing emulators (I'm under the impression that emulators just load the ROM into memory and assigns pointers to the current bank). It also wouldn't be backwards compatible. But this is a very interesting idea. Making it possible to "wire" the chips together as you'd like is possible, but I'm unsure how you would specify the function of the chips. Do you mean making new chips, or selecting from the already existing chips? Care to explain (Possibly in a new thread)?

[tepples]

I exclude unlicensed games for the same reason that emulator authors don't bother emulating the endless streams of unlicensed add-ons and clone systems

An emulator that can't run Tetяis, Klax, Spiritual Warfare, and the Battle Kid demo is an emulator that doesn't run notable commercial games for the NES. So emulators would have to support iNES and NES 2.0 anyway.

Then there's something neither iNES nor an internal db can accommodate, which is people who want to make not only their own games, but their own hardware. Since this is pretty niche

It happens to be the niche chosen by users of this web site.

and would be a rom-side format prone to revisions, this should be its own format. Though I doubt if anyone truly cares enough to spend months preparing one.

Kevin Horton cared enough, and I fully support his NES 2.0 format. It accommodates homebrew fairly well: it expands the "mapper" field to 12 bits and introduces a 4-bit "variant" field. It also expands the range of PRG ROM, CHR ROM, and save sizes that can be represented. Besides, most notable homebrew games appear to be on mapper 0 (NROM), 3 (CNROM), 7 (AOROM), 2 (UOROM), or 1 (S*ROM), which are the mappers supported by retrousb.com ReproPak kits.

While I like the idea of zip and separate PRG and CHR files because it makes it easier to edit them, I don't see any reason to have them separate just for the sake of having them separate.

That's what zip is for. Wrappers using the PKZIP format, such as jar, smzip, odt, and MAME zip files, allow the ROM sections to be simultaneously together and separate.

Also, you must keep in mind that not every platform may be able to unzip and generate the checksum within a reasonable time due to some crazy restrictions

Only Game Boy Advance really has a problem with that. Nintendo DS can checksum a whole 2.5 MB DS Download Play client binary in order to check its digital signature.

Making it possible to "wire" the chips together as you'd like is possible

The true way to represent a custom mapper would be to create a netlist in schematic capture or in some sort of HDL, and then simulate that netlist for every memory access. That probably won't run in real time on a smartphone or an Atom laptop.

[Wkter]

Only Game Boy Advance really has a problem with that.

Exactly the system I had in mind.

The true way to represent a custom mapper would be to create a netlist in schematic capture or in some sort of HDL, and then simulate that netlist for every memory access. That probably won't run in real time on a smartphone or an Atom laptop.

Why stop there? Let's make the custom mappers like this.
But seriously though, it should be feasible to emulate the wiring of the cartridge by simply adding a few more pointers and emulating the chips standalone. It would of course slow down the emulation to the point where it's unplayable by the really slow systems. But we're getting off-topic here.

[3gengames]

Maybe if emulators could instead all make mappers custom mappers, except iNES come defualt, so then people could inject their C code into the mapper file that they decide to make their mapper and then the emulator would compile it when the time comes and then could run it? I don't know enough about C to know if it is possible, but it's an idea, right? That'd allow not-too-much hacking to emulators, correct? And it'd allow for "custom" hardware. Win win!

[clueless]

Going out on a limb here....

Wireshark, tcpdump (linux, bsd), snoop (solaris) all do the same thing. They sniff network packets. You can tell it to filter the traffic and only show (or record to a file) specific packets. This is expressed in a language called "bpf", for "Berkley packet filter". BPF is compiled into a byte-code that is interpreted by libpcap (the library that tcpdump and wireshark use - idk what its called in Solaris). It is possible that some network stacks will handle BPF in kernel mode (to avoid the overhead of kernel/user mode switching). High-end network gear offloads the packet filtering to hardware (asic or fpga or dedicated co-processor on the NIC).

I mention this because the fundamental idea is that a high-level language, like BPF, is turned into an efficient byte code ran inside a VM (or JIT compiled to native code) (btw, bpf is NOT Turing complete).

Maybe someone with a great deal of brains could design a "mapper language". Not as low-level as a net-list. Maybe it implements a logic-gate level abstraction, maybe a higher level abstraction, like "if/then/else" and some integer array for maintaining state. The iNES format can be extended to flag the existence of this "mapper implementation", and it would be appended to the ROM after the (optional) char-rom segment.

I lack the time right now to go into greater detail, but I can envision how to implement MMC1 and NROM this way (in fact, NROM is just a null statement, kind of trivial).

I like this approach (I did my senior "capstone" college credit work on a compiler / language translator), so it appeals to me. Unfortunately, I lack the time to devote to making a reference implementation right now.

Any thoughts?

http://en.wikipedia.org/wiki/Berkeley_Packet_Filter
http://www.tcpdump.org/papers/bpf-usenix93.pdf


(edit, added content)

Wow. I never actually finished typing up my thought. Emulators could be adapted to implement the mapper lang -> byte code conversion, and byte-code VM. The VM would be receive the same input info as a real cart would (addr bus, data bus, ctrl signals). It would be capable of implementing a small state engine that would determine what gets mapped when, and when to fire IRQs. This layer within the emulator would handle all of the reads and writes to the cart. It would need access to the RAM allocated in the EMU that holds the contents of the RAM and ROM chips on the cart.

This might seem like a crazy idea, and indeed the lang -> byte_code "compiler" would be a bit of work (maybe ~1000 lines of C code?), but a fairly efficient VM can be constructed to handle this. A modern PC running an emulator should have NO problems maintaining a proper frame-rate.

I would also like to see an emulator that exposes, via its debugger, the internal state of its current (or future) mapper implementation. For example, when debugging a MMC1 game in FCEUX, it would be nice if I could access to pop-up window (debug -> cart mapper) that would show what pieces of ROM are currently switched in, and the current state of the MMC1 latch

Damn, I wish I had time to work on this. Maybe later this summer.

[FitzRoy]

How this would be a good idea? Yeah it's interesting and may work okay (What emulator has an internal database of checksums again?)

None? Why do you think I'm proposing it? What emulator guarantees correct mappings for downloaded games? None. You offloaded that responsibility to rom site morons who could care less about mapper integrity. They don't even care about rom integrity. If we could supply roms with emulators, we would, but we can't. Because most distributors are morons who know nothing about the files they're distributing. GoodNES has hundreds of bad dumps and mappings. I would know, I converted as much as I could from that set when I started mine.

but it won't be any better then the iNES header and will be worse for other situations like homebrew, what is supposed to happen in between compiles and revisions and such?

If that was true, I wouldn't have spent the time that I did. You are as clueless about the integrity of your files as the average user. How many ghost bugs have emulator authors chased down due to bad or missing headers? How many will future ones chase down? Emulator-side mapping is easy to implement and an instantaneous guarantee for everyone, user and author alike. And all I hear is complaining about having to use a different format for unlicensed games. Really? Wallow in it, then.

While I like the idea of zip and separate PRG and CHR files because it makes it easier to edit them, I don't see any reason to have them separate just for the sake of having them separate.

Let's say you try to verify your files against a trusted source, and you get a checksum mismatch. Is it because of a bad mapper, a bad chr, or a bad prg?

Let's say you dump a new game. Once you combine it and compare its checksum to the existing dump, and the checksum is different. Is that because the chr is different, the prg is different, or both?

Combining everything into a headered frankenfile makes it hard to find the reasons for differences, because any one of the portions could be to blame. As an archivist or a user just trying to verify his files, that's what's annoying about it.

[3gengames]

How this would be a good idea? Yeah it's interesting and may work okay (What emulator has an internal database of checksums again?)

None?

Nope, there's an emulator that does this already. I don't recall which one though. Too late?


I have yet to download a bad ROM from any site, so why is this non-issue being used as the best reason for it? Ughhh...the yearly lets change iNES to something else thread, yay.

[Zepper]