Reservation of mapper numbers

[tepples]

How I've been pronouncing nicknames of two regulars in the NES emulation scene:

• zeromus aka zeromuis: Zero Mouse (Danish, Latin, Norwegian, Old English, Faroese, Icelandic, Afrikaans, Dutch) (also the final boss from Final Fantasy IV)
• CaH4e3: Sanchez (Russian)

Anyway, it appears CaH4e3 and others dumping Asian games have been assigning iNES mapper numbers that conflict. This problem in fact dates back to fwNES, where there was at one time a separate space of "fwNES mapper numbers". So in #nesdev, zeromus recently proposed a way to clean up mapper support. We traced this to the lack of a central registry for mapper numbers.

Someone said he prefers board names over mapper numbers. But there is no one-to-one or even one-to-many relationship between mappers and boards. SxROM encompasses several functionally related boards, and boards like UNROM can be configured for two different mappers based on which 7400 series chip is in one position. Some boards don't even have recognizable names. (See UNIF#Shortcomings for the details.)

I define a mapper number as "a series of specifications which cause you to be able to emulate a ROM" that may correspond to 1 board or a dozen boards.

Until now, the best practice on the wiki has been to update the Mapper page based on Disch's docs at RomHacking.net and based on the source code of FCEUX, Nintendulator, and Nestopia. This resulted in, for example, the allocation of mapper 28 to a multicart mapper whose design I led. But some dumpers, such as CaH4e3, have gone off on their own and assigned mapper numbers proprietary to their own emulators.

The proposed solution involves a bit of pre-allocation of the NES 2.0 mapper space. I propose to lay it out in "planes", roughly like Unicode when it expanded past UCS-2:

• Plane 0 (0-255): Basic Multilingual Plane, the current mess
• Plane 1 (256-511): Supplementary Multilingual Plane (mostly for new homebrew mappers)
• Plane 2 (512-767): Supplementary Ideographic Plane (Chinese dumps)
• Plane 15: Private use area (not for publicly distributed dumps)

It also involves a series of numbered memos posted on the wiki and signed by several major emulator developers. These memos describe enough of the known behavior of one or more mappers to emulate one or more specific games, and this may involve defining new mapper numbers or revising the expected behavior of existing mappers.

A memo would have at least these:

• Titles of games affected by the memo
• New mapper numbers (if any)
• Summary of changes to existing mappers' behavior (if any)
• Links to specific revisions of wiki pages about the affected mappers

It was first proposed that these would be sequential, but because some memos might take longer to develop, that might not be the best way. How do OpenGL extensions and IETF RFCs get their numbers?

[zzo38]

Your idea of something similar to Unicode planes seems good to me. And yes it is good not to conflict please.

Reserve one mapper number (just one, no matter how many mappers I invent) for my use to add extra data to the end of the file.

[zeromus]

Unlike GL extensions or RFCs, we are desiring to go in order and have the participating emulators skip as few memos as possible. I'm not sure anymore how much wisdom we can glean from that.

Perhaps we can make a forum topic for a memo proposal, and do the discussion in there. This is similar to GL extensions which get named by the designers. Then on the wiki we can keep a list of in-progress memo deliberations, as well as a list of 'adopted' memos which would be assigned a number at the moment when they get adopted. Keeping in mind that 'adoption' will typically consist of the emulators participating at the time as saying 'OK, done.'

Maybe we can make a wiki page for the memo with a name like MEMO-MAPPER28 or MEMO-MMC3SUBMAPPERS and its understood that it should be a collaborative draft, which will get moved (or pasted) into a MEMO-XXX when it is adopted.

Also, I should note that some memos will be burdensome to implement, and so if an emulator hasnt got around to implementing it yet, it ought to write so on the memo wiki page, so that the emulator can brag "compatibility through memo-109 (excepting exceptions)".

I think a candidate for memo-001 would be:
Emulator MUST Support NES 2.0 headers (link to specification)
Future work MUST conform to (insert tepples mapper plane namespacing system)
Future development SHOULD conform to zeromus's definition of a mapper number (forestalls debates about what constitutes mappers, pegging it as nothing more than an engineering and organizational decision of convenience instead of some kind of factual description of physical hardware. To a certain degree, for me at least, its helpful to have as a mission statement.)

We should probably go ahead and make mapper28 a memo, cos its new, and to get the ball rolling.

Should we use a memo to define MMC3 variants as submapper values? I'm just throwing out some random ideas for prospective memo-writers to play around with.

Finally: Do we have someone willing to enlist as the nestopia manager for this project? I'm not sure what the most cutting edge nestopia is or where its getting developed, but someone needs to claim themself as boss of it and prove it by speaking for it in this venue we're proposing. This should probably get done for virtuanesEX as well, since a lot of cutting edge work on asian mappers gets done in there. We may need to make an open source project host for that, since its been managed in the medieval way so far, as far as I know.

[tepples]

We should probably go ahead and make mapper28 a memo, cos its new, and to get the ball rolling.

That and all the subsets of mapper 28 that have their own mapper numbers: NROM (0), CNROM (3), UNROM/UOROM (2), UNROM/UOROM (180), AOROM (7), and BNROM (34, CHR <= 8192 bytes). This way we have something old and something new for the first memo.


Then we'd just need something borrowed and something blue.

[zeromus]

I dont understand. Are you proposing to formally define those in the memo, or do they have some relationship to mapper28, like, they can be subsumed into mapper28? Or are we defining mapper28 in terms of those others depending on chr/prgsizes and register configuration, and so the definition of those others needs to be part of the mapper28 definition?

I'm not sure formally defining ancient mappers in memos is a great way to start. There are a lot of those. I figured we'd focus on mappers whose definitions were being revised as more information was acquired.

[tepples]

All I'm saying is to define them in the memo as an example of how an existing, well-known mapper might be documented. It's to establish a baseline: if you have this memo, you have these mappers.

[zzo38]

All I'm saying is to define them in the memo as an example of how an existing, well-known mapper might be documented. It's to establish a baseline: if you have this memo, you have these mappers.

I agree.

[zeromus]

here's some other information about virtuanesEx, and a good test case to discuss. here's the scenario:

some clowns dump a rom, and figure out how to emulate it.
they upload the roms to Tszone-fc.ys168.com (KT-1019 and KT-1020 in the first folder)
they also upload virtuanesEX to this site, in the last folder. These archives contain virtuanesex.exe which is closed source as well as a mapper004.cpp which contains the necessary hacks in order to make those game runs, and which was presumably used to build that virtuanesex.exe

( this url http://code.google.com/p/myvirtuanes/ contains some old open source virtuaNES, plus a bunch of updated mappers, administered by an individual named tensai.wang )

It seems that these two games are basically stock MMC3, with an extra register R at $5000 which selects extra prg bank numbers R*16.

So here are my questions:

1. We still need to find someone to manage virtuanesExTurbo or something, opensource, and consolidate mapper changes into it? can we adopt myvirtuanes on googlecode? Apparently local user byemu may be associated with that project.

2. Should this kind of change be done as
a. A modification to the MMC3 specifications, stating 'when KT-1019 or K-1020 are present, then register $5000 exists' (shouldnt use a submapper for this, since we'd run out quickly when it comes to certain mappers which have 100 hacked variants)
b. A definition of a new mapper, call it MMC3-KT, defined as 'MMC3 plus register $5000'

Option A is quick and easy... My definition of "mapper" was designed to leave open the possibility of choosing option A. But, let's think about how that gets implemented in practice: A CRC check runs to detect specific roms which trigger m004 hack conditions logic. Since I think part of our mandate is to try and forge a future where every rom could be correctly headered without crc checks, I think we're obliged to choose option B. Otherwise we're just supporting games while leaving a mess.

[zeromus]

Here's another kind of test case to discuss.

There are a number of roms which cah4e3 has dumped which are implemented in fceumm/fceux sources only via UNIF. Due to some emulator codebases only supporting .nes file extensions, or maybe some other reasons, they are actually unif files renamed to .nes, with board names invented by cah4e3. Additionally, some large dumps have been made as UNIF since iNES 1.0 couldnt support adequate sizes.

An example of this is "4-in-1 Benshieng (BS-7090)" Which can be found in cah4e3's "2007 year dumps" using the board name BMC-BS-5

1. We should clarify that UNIF format is deprecated, and should be treated as a barely adequate container for describing the PRG and CHR roms present for purposes of hashing to create the CRC for game IDing purposes. We should also specify the hashing approach to be used.

2. This is a straightforward example of a new mapper to be defined, in the chinese plane.

[tepples]

2. Should this kind of change be done as
a. A modification to the MMC3 specifications, stating 'when KT-1019 or K-1020 are present, then register $5000 exists' (shouldnt use a submapper for this, since we'd run out quickly when it comes to certain mappers which have 100 hacked variants)
b. A definition of a new mapper, call it MMC3-KT, defined as 'MMC3 plus register $5000'

That'd probably be a new mapper, just as the various MMC1- and MMC3-based multicarts with an extra register (e.g. NES-EVENT and the Nintendo World Cup/Super Spike V'Ball mapper) are new mappers. SUROM/SXROM doesn't need a new mapper because the 512 KiB size alone identifies that PRG ROM A18 is connected to the mapper's CHR A16 output. Look at all the MMC3-like mappers already assigned.

[byemu]

I think we need define a new format replace ines,unif is a better format,but this unif is still not inlcude much information for user(For example:vram size)!

[tepples]

NES 2.0 includes size of both battery-backed and non-battery-backed CHR RAM.

[Near]

These are just general musings. You'll no doubt want to stick with what you have for existing games, and come up with some new extension system for the billions of bootleg carts out there.

...

It has occurred to me that there are an infinite number of ways to describe the contents of a PCB.

Board IDs are wonderful, but many can be functionally identical, and some boards have no names.
Mappers are an attempt to classify sets of criteria into numeric values. But suffer from trying to pack slightly differing behaviors to fit into such a small iNES 1.0 space of 256 possible values (like the variant pinouts of VRC boards using the same mapper ID.) Also a problem was that there was never a true governing body to declare official values. Instead you have a free-for-all of dumpers making up their own values, with nobody to say who gets to decide what becomes official. Hence you get conflicts.

But mappers do hold one promising thing in being something that doesn't require a (possibly non-existent) textual board description, and they represent the most terse possible expression of a board's contents.

The first thing I would decide is whether or not we want a mapper ID to describe an exact hardware design, or an exact specification necessary for emulation. For instance, one board may substitute a 74LS139 for some other chip, yet the code to emulate the board would be 100% identical. Does that deserve two IDs, or just one?

I think if it were up to me, I'd have a unique ID for absolutely any variation. ROM size, RAM size, pinout modifications, etc. The only time two games would have the same ID would be if you could swap the ROM/RAM chips between the two, pin-for-pin, and they'd both play perfectly. You should be able to punch in an ID and get an exact image of the board, all of its chips, and its wiring. If an emulator's mapper code could emulate 30 different IDs with the same code, it could bind them all: if(mapper == x || mapper == y || mapper == z) useMapperTypeN();

After that, I would make a linear mapper namespace. Each ID represents one mapper. Each emulator could implement its own ID -> board description lookup, however they wanted: internal code, internal database, external database, whatever. The difference from the existing iNES 1.0 mapper IDs would be that we would never use the same ID for two boards that require any differences in how to emulate it.

Finally, one could nitpick the idea of submapper bits for things like pinout / size variations. This would help prevent fragmentation of the ID#s (eg we have two VRC4 pinouts, and give them 0x2016 and 0x2017, then we find some bootleg cart with a modified VRC pinout, but now it has to be 0x3147.) But it increases the complexity of a database layout. So I would instead make sure I described every possible board used by every Nintendo-licensed cart first. And then not worry about non-sequential ordering for similar bootleg hardware. That stuff will continue to turn up until the heat death of the universe anyway.

It would be easy to shoehorn it into iNES 2.0: just use mapper ID 0x8000+ for this. Other fun solutions would be to put the ID into the filename: "Duck Hunt [8a16].nes", or to have a CRC32->ID database off to the side that the official maintainer of this list would always keep up to date, etc. And these let you keep iNES 1.0 / 2.0 or whatever for infinite backward compatibility.

Diversion over, back to the real world then =)