A note to all NES-emulator authors: ROM header wishlist

[oRBIT2002]

I have a wish for everyone that's coding a NES-emulator..
I know some emulators uses checksums for detecting certain ROMs and to automatically adjust the experience. STOP doing this, for several reasons:
1) I don't like it. Personally I think it feels like "cheating" in some way..
2) It's bad for people doing homebrew/ROM-hacking (like myself). Try hacking VS-ROMs and play them in FCEUX and see what happens. Nothing, they won't work properly because checksums doesn't match the builtin database and you don't get correct VS-functionality.

What do you guys think about this? Am I the only one upset about this stuff?

[Near]

This could be avoided if every emulator author agreed upon a way to denote all information about the game, and if every user subsequently updated their sets to reflect said format.

But that's not possible even in a scene with two emulators available for a given system, let alone over 9,000.

It's like herding cats.

[infiniteneslives]

It's like herding cats.

'nuff said...

[lidnariq]

While NES2.0 is basically not deployed at all (except for in the things that I myself have released), at least Nestopia does use that header to select the correct PPU for Vs System games.

[oRBIT2002]

I guess it's very practical for gamers that doesn't need to bother about the settings for "odd" ROMs.. But I still think the options should be there. FCEUX is a pretty big source for my frustration. How do you develop something for VS-Zapper? The only way is that you start a VS-Zapper game, inject your own code (copy'n' paste to RAM), or hack your own ROM to match a CRC-value already in the VS-Zapper database.
I think Nintendulator behave in a similar way..

[Quietust]

I guess it's very practical for gamers that doesn't need to bother about the settings for "odd" ROMs.. But I still think the options should be there. FCEUX is a pretty big source for my frustration. How do you develop something for VS-Zapper? The only way is that you start a VS-Zapper game, inject your own code (copy'n' paste to RAM), or hack your own ROM to match a CRC-value already in the VS-Zapper database.
I think Nintendulator behave in a similar way..

You think incorrectly - the only place Nintendulator has CRC checks is in its MMC5 mapper code for automatic detection of SRAM size, and that only happens if it can't find NES 2.0 header data.

For Vs System games, the only stuff it autoselects is the PPU (2C04 variants for palette, 2C05 variants for $2000/$2001 swap and the special bits read back from $2002) by looking at NES 2.0 header data; "Zapper" and "Vs Zapper" are completely different controllers that you have to select and configure separately.

[MottZilla]

For MMC6 (Startropics) you can look for a write to the WRAM register to set certain bits which other games won't. I think Startropics 1 and 2 both write #$20 to $A001 during startup which is related to SRAM block enabling. Regular MMC3 games don't set those bits. It's possible some do. But if they don't then that's one way to detect those games.

[natt]

1) I don't like it. Personally I think it feels like "cheating" in some way..

When you plug in a game cartridge to a real NES, the cartridge has significant amounts of "information" on it besides the bytes of the ROM image. That information includes the exact functioning specifications of every single mapper chip, plus how they're all connected, and more. A real NES never has to guess what variety of mapper chip is on your particular board, or whether there's a resistor at R5. It knows. Why should emulators not be allowed to know the same things?

As far as modified ROMs go, there have been some efforts made in Bizhawk to help out with those sorts of things. If there's something in particular that you need help with for a particular project, I'll see what I can do.

[MottZilla]

He specifically doesn't like using a checksum to ID a ROM image. Emulators ideally should be told about special cases in the header. The problem is that iNES has been around forever and expanded apon. Many people still have and distribute roms with bad or older headers. So it's up to the emulator to just fail or try to do additional checks to allow things to successfully load.

iNES 2.0 probably nails a number of special cases but the problem is the games that need this info already exist with older iNES headers that don't mention the additional information. So unless the user updates their header, it won't work. In theory though if you are a hacker, you should update the header and then the checksum will pass for the older header roms, and your hacked rom will specifically tell the emulator what to do. So it all works if the emulator supports the newer iNES header and acts on it.

There's also nothing wrong with a checksum/hash to detect certain games. However you'll have problems with hacks if you are hashing *everything*. Instead you could pick a portion of rom to hash that is unlikely to change in a hack, but still unique.

[tokumaru]

A real NES never has to guess what variety of mapper chip is on your particular board, or whether there's a resistor at R5. It knows.

Wrong. The NES doesn't know shit, as far as it's concerned all carts have 32KB of PRG and 8KB of CHR! It's the cart itself that does all the magic, and since the cart (board, mapper, wiring, etc.) is missing from the emulator + ROM combination, the emulator has to get that information from somewhere in order to properly simulate the cart. Using the contents of the ROM to know what cart to simulate is a poor way of solving the problem, for the reasons oRBIT2002 mentioned.

[Near]

For MMC6 (Startropics) you can look for a write to the WRAM register to set certain bits which other games won't. I think Startropics 1 and 2 both write #$20 to $A001 during startup which is related to SRAM block enabling. Regular MMC3 games don't set those bits. It's possible some do. But if they don't then that's one way to detect those games.

I thought about this approach for the Game Boy Advance (no information on whether the memory is SRAM, EEPROM or Flash, nor what size.) You can usually tell what it is based on what the game tries to do (barring copy protections.)

The bigger problem I faced was with things like save states and auto-rewind snapshots. Dynamically changing the amount of RAM the cartridge supports means you have to be extra careful when serializing and unserializing data. A bit heavy-handed to just always dump 128KB of "any RAM" into all save states ...

[natt]

A real NES never has to guess what variety of mapper chip is on your particular board, or whether there's a resistor at R5. It knows.

Wrong. The NES doesn't know shit, as far as it's concerned all carts have 32KB of PRG and 8KB of CHR! It's the cart itself that does all the magic, and since the cart (board, mapper, wiring, etc.) is missing from the emulator + ROM combination, the emulator has to get that information from somewhere in order to properly simulate the cart. Using the contents of the ROM to know what cart to simulate is a poor way of solving the problem, for the reasons oRBIT2002 mentioned.

Yay, let's argue semantics instead of fixing the problem. The emulated system, by which I mean the combination of the deck and the inserted cartridge, does not do any guessing, any heuristics, any fuzzy logic. While it's true that the NES deck itself does not know anything about the internals of the cart, all relevant internals are exposed by the sequence of data reads from the PPU and CPU buses, as well as cart interrupt timing. There's no uncertainty.

While it's true that hashing doesn't match what the console does, hashing does mach certain facts of life: Whenever a particular PRG ROM+CHR ROM combo was encountered on manufactured cartridges, it was always on a particular board (or one from a short list of particular board.) The most correct solution would be to put information in the ROM header, I agree; but that's not done, so pointless to talk about it from the point of view of what emulators should implement.

[etabeta]

but we don't emulate "the combination of the deck and the inserted cartridge", we emulate the deck and we try to handle the best we can what comes in the cart

this discussion is meaningful for me given that, even if in MESS we have a database of known good dumps, we also keep supporting iNES/iNES2.0/UNIF to allow homebrew software to run

the problem is that either we establish some sort of 'central authority' which assigns in a meaningful way the mappers and keeps doing this until the firms in Taiwan and HK quit producing new carts, or there is no solution

just look at the 5-6 known cases of mapper #s which have been assigned to 2 incompatible boards...

[natt]

but we don't emulate "the combination of the deck and the inserted cartridge", we emulate the deck and we try to handle the best we can what comes in the cart

All that's saying is that we make a best effort which doesn't always succeed. The intent and direction of effort is to emulate the combination of the deck and the inserted cartridge.

this discussion is meaningful for me given that, even if in MESS we have a database of known good dumps, we also keep supporting iNES/iNES2.0/UNIF to allow homebrew software to run

the problem is that either we establish some sort of 'central authority' which assigns in a meaningful way the mappers and keeps doing this until the firms in Taiwan and HK quit producing new carts, or there is no solution

just look at the 5-6 known cases of mapper #s which have been assigned to 2 incompatible boards...

I think UNIF is a bit better than iNES in this respect, just from the freedom of board choice; any string of reasonable length and content is allowed, so inadvertent collisions are much less likely than when you only have a few numbers to work with. Of course, neither UNIF or iNES 2.0 is used much at all in the wild, so...

[Near]

The most correct solution would be to put information in the ROM header, I agree

But therein lies the cat herding problem.

Who decides what information is worthy of going into the header, and how it's marked up?

Maybe I want to know what controllers a game is compatible with, so I can auto-select them. Maybe everyone assumes 8KB PRG is enough for Galaxian, and then a 4KB PRG ROM is discovered later on. You never know.

To me, the problem is trying to fit the whole game into a single file, and compressing all of this rich metadata about the PCB into this 16-byte header that has to work around a legacy mid-'90s header format.

I'd rather separate the ROM and the description of the PCB itself. And then allow that description to be stored in an extensible format (JSON, XML, ...) so that when something new comes up, we can support it.

From there, I'd probably keep a database of every commercial game internally, so that all commercial games can just load and play, regardless of whether there's an iNES 1.0 or 2.0 header, or not. And then require the external description to load homebrew. If it has to be one file, store the ROM and the description both inside a ZIP archive.

But now you have another problem. All the Javascript emulators are going to want to use JSON. MESS is going to want to use its own proprietary XML format. Nestopia has its own XML format. I use BML myself.

Even if we agreed to store external board descriptions in an extensible format, and even if we all agreed on a specific markup language, and even if we all somehow agreed on what tags to make and what to name them, we'd still run into issues where certain parsers had bugs, so a file that doesn't end in a line feed works on emulator A, but not emulator B.

Nobody who has already made homebrew would go back to make this board layout, and few people making new homebrew would want to make layouts for the dozen different emulators.

So I've pretty much come to the conclusion that there is no solution to this problem.

Support iNES 1.0, support iNES 2.0, support as many external board markups as you can, support an internal database, cross your fingers and hope for the best.