NES 2.0 submappers

[rainwarrior]

Mostly I'm trying to understand your view on resolving the chicken-and-egg issue inherent in creating a new mapper.

If anyone wants to implement a new mapper, they should do it themselves first. Hack an emulator or otherwise create an implementation that they can use to develop with, and finally publish a ROM that requires it before asking for a mapper allocation.

If someone wants to directly approach maintainers of emulators and ask them to pre-emptively implement something for them, that implementer can make an evaluation about whether it's worthwhile (just like they can evaluate any part of the spec and implement it or not). It is up to the emulator author whether they would do that kind of thing. This mostly has to do with what kind of faith they might have in the person making the request, I suppose.

My interest is that the allocation process should be clear-cut. Publish the software first, then allocate the mapper in the specification. The specification should follow existing material (i.e. this it not chicken vs. egg, the ROMs are the source and reason to implement).

People should not be adding things to the spec in lieu of asking someone for help in hacking an emulator for their needs, or developing some kind of relationship with emulator authors. Adding your wishlist mapper to the spec is a detriment to that spec until there is some ROM that an implementer can actually run with it.

[kevtris]

Guess I should weigh in. I created the whole submapper thing to eliminate the need for CRC checking and other hacks on the existing roms. There is no way in the world to ever come up with a perfect "100% all inclusive" format that could account for anything ever new that showed up. I attempted to make additions that would benefit emulator writers and for my FPGA NES.

In fact, the spec doc I wrote is how my FPGA NES implements it. And probably this is how it always will implement it. I noticed on the wiki that someone came up with a whole set of extended mapper numbers for things like homebrews and hacks; this is exactly the kind of thing I did NOT want to happen with extra mappers. This is just another willy-nilly mapper number mess like what exists currently with the 256 we already have.

I do not like hacks to determine what kind of mapper/ROM/etc a game is, including CRCs, ROM sizes, and other things which are not extensible to new stuff that might appear (be it more games, or homebrews, etc). If you rely on CRCs or ROM sizes and someone hacks a game to include more levels, etc. (or just some other kind of minor hack) it will no longer work because the ROM size changed, or the CRC changed.

So that was the rationale for it all along- disambiguating existing stuff and with support for all the newer stuff I had ever come across up to that point. The idea was only a few percent (say, 2-3% at most) of existing ROMs would need to be modified so they could be disambiguated. In fact, the header was designed so if the ROM was run on a "Regular" emulator it'd still run as it always did.

[tepples]

In fact, the spec doc I wrote is how my FPGA NES implements it. And probably this is how it always will implement it.

Given this forthcoming flash cart whose mappers are compiled from the same source, I guess we ought to defer to you to some extent. This is why I've tried to seek your input in #nesdev on submapper proposals.

I do not like hacks to determine what kind of mapper/ROM/etc a game is, including CRCs, ROM sizes, and other things which are not extensible to new stuff that might appear (be it more games, or homebrews, etc). If you rely on CRCs or ROM sizes and someone hacks a game to include more levels, etc. (or just some other kind of minor hack) it will no longer work because the ROM size changed, or the CRC changed.

CRC I agree with. But if the ROM size changed, then it's often supposed to go on a different board. For example, expanding a 256K SNROM game to SUROM means that at a minimum, 1. the reset code needs to be fixed carefully to clear CHR A16 if the game powers up in the second 256K, and 2. the game needs to not rely on CHR A16 disabling PRG RAM. This is why I suggested guessing SUROM, SOROM, and SXROM behavior based on memory sizes. Besides, even in "submapper 0", CHR A16 is a PRG RAM disable bit only if CHR size is 8K (SNROM), not larger (SLROM).

[rainwarrior]

CRC I agree with. But if the ROM size changed, then it's often supposed to go on a different board. For example, expanding a 256K SNROM game to SUROM means that at a minimum, 1. the reset code needs to be fixed carefully to clear CHR A16 if the game powers up in the second 256K, and 2. the game needs to not rely on CHR A16 disabling PRG RAM. This is why I suggested guessing SUROM, SOROM, and SXROM behavior based on memory sizes. Besides, even in "submapper 0", CHR A16 is a PRG RAM disable bit only if CHR size is 8K (SNROM), not larger (SLROM).

I think "guessing" is the wrong approach. If the header has a bad/inconsistent specification, it's just a bad ROM, and it's allowed to crash or do other stupid things. We don't need to be able to recover from bad headers; we just need to be able to handle the things that the old headers couldn't specify.

[rainwarrior]

I didn't spot this earlier because I misread the tables when I was trying to understand earlier, but now that I've looked at it again I'm very confused about the SUROM, SOROM, SXROM submappers. What is it that makes them mutually incompatible?

By the description here I don't see any inconsistency between how each of them uses the bits, i.e. they just become a disconnect/mirror if the connected RAM/ROM isn't large enough, just like we do with any mapper that has support for variable sizes.

Code: Select all

 43210
 +---+
 PSSxC

 4:P - A18 of 512k PRG ROM, disconnected if smaller
 3:S - A13 of 16k PRG RAM, disconnected if smaller
 2:S - A14 of 32k PRG RAM, disconnected if smaller
 1:x - unused
 0:C - 4k CHR banking control

What am I missing? Why is this 3 submappers and not 1? (This looks VERY different than the MMC6 situation, in which the bits of the register have different functions.)

[lidnariq]

I get (and am sympathetic to) your point: the 16 bits of mapper and submapper should represent a fixed ABI... but...

If the header is inconsistent, behaviour is unspecified. The implementer can choose how to handle this, it's not important for us to specify what happens in response to every possible header, only what happens in response to well-formed headers.

So, that's a vote for "crash". With bonus "possibly yielding an exploitable surface for arbitrary code execution when the emulated 6502 VM uses an unallocated pointer".

I'm not talking about "typical" usage here. The set of four choices I gave was not a request for which behavior we should recommend.

My entire point is the observation that permitting the header to contradict itself is asking for trouble: designing things to explicitly expose pointy bits is asking for people to hurt themselves.

[rainwarrior]

The implementer can choose how to handle this, it's not important for us to specify what happens...

So, that's a vote for "crash". With bonus "possibly yielding an exploitable surface for arbitrary code execution when the emulated 6502 VM uses an unallocated pointer".

No, it's not a vote for crash. When I say it should be "unspecified" this is a vote for it to be the implementer's choice how to handle an inconsistency.

If a crash or insecurity is not a problem for the implementer, it is a valid choice for them to make. If these things are a problem, they will figure out how to prevent it.

I'm not talking about "typical" usage here. The set of four choices I gave was not a request for which behavior we should recommend.

Those four choices are outside the scope of the specification. The spec isn't telling them whether it's executing on an NES, flash cart, VM, Mac, Windows, whatever. That's entirely irrelevant to the goal of being able to specify the stuff needed to run a ROM. That's their choice, not mine, and not yours.

designing things to explicitly expose pointy bits is asking for people to hurt themselves.

All of the pointy bits are put there by the implementer, not by this spec. This concern is outside the realm of what we are doing here.


Anyhow, this has gotten far too ideological. I think I have heard your argument and you have heard mine about whether MMC6 should have a submapper assignment. Argument has clearly been unconvincing for either of us. What form of authority would you suggest we defer to instead?

[tepples]

By the description here I don't see any inconsistency between how each of them uses the bits

SNROM is the only one with an actual behavior difference. CHR A16 (the "P" bit) can be one of three things:

1. A16 of 128K CHR ROM or RAM, disconnected if smaller
2. A18 of 512K PRG ROM, disconnected if smaller
3. Disable PRG RAM, disconnected if PRG ROM larger than 256K or PRG ROM larger than 8K

[rainwarrior]

Yet SNROM never received a submapper allocation. I presume this is because the PRG-RAM enable is redundant and unnecessary for compatibility? I've added notes on how the SXROM submapper is a superset of SUROM and SOROM to the wiki to make it clearer how they are related.

I noticed on the wiki that someone came up with a whole set of extended mapper numbers for things like homebrews and hacks; this is exactly the kind of thing I did NOT want to happen with extra mappers. This is just another willy-nilly mapper number mess like what exists currently with the 256 we already have.

None of the extended mappers have been allocated, so I'm not really sure what "mess" you're referring to?

Personally I've been advocating the stance that the software must come before an allocation. If there's no ROM that needs it to run, don't allocate it. This actually puts a pretty big barrier to new mapper allocation, because people aren't nearly as good at making software as they are coming up with ideas for new mappers. So... no allocations so far.

I guess we ought to defer to you to some extent. This is why I've tried to seek your input in #nesdev on submapper proposals.

Now that kevtris' assignments for MMC3 have been re-instated, the wiki version of the spec appears to be fully compatible with kevtris' original document, so I don't think we have a problem yet. I moved everything that was unimplementable to "proposals", which cut us down to what is mostly a subset of the kevtris version.

Here are the things we have added w.r.t. kevtris' proposal:

• 001:5 - fixed 32k PRG for MMC1. I haven't tested all the games yet, but it seems that in at least some cases simply initializing the PRG banks in NROM order at load is already sufficient for compatibility with submapper 0. This might even be true for all of the games?
• 068:1 - Sunsoft Dual Cartridge System for Nantettatte!! Baseball.
• 210:1 - Namco 175 specification to remove the need for heuristic detection.
• 210:2 - Namco 340 specification to remove the need for heuristic detection.

Here are the things we have removed:

• 001:3 - Deprecated because this is already iNES 155.
• 004:2 - Not removed, but requirement for hard-wired mapping of MMC3 unclear. (Any games?)
• 016:1 - Bandai FCG.
• 016:2 - ? (No description.)
• 016:3 - ? (No description.)
• 019:1 - N106 Mindseeker.
• 034:1 - NINA-01. (Isn't Impossible Mission 2 is already compatible with 034:0?)
• 034:2 - Union Bond (Does anybody know what this means?)
• 065:1 - ? (Insufficient description.)
• 078:2 - Fixed vertical + WRAM. (Not needed for any game? NES 2.0 lets us specify WRAM already too?)
• 080:? - Description says they were moved to mapper 207.
• 083:1 - Cony with 2k CHR banking.
• 083:2 - Cony with other modifications.
• 166:? - Description says it is really mapper 27.
• 167:1 - ? (No description.)
• 188:1 - 256k PRG Karaoke ROMs.
• 245:1 - Flat CHR RAM.

So, many of these are insufficiently described to implement. Maybe some of them can be included. By the same criteria that I outlined for allocating mappers, I think we should treat these the same.
1. Explain what game/ROM needs a submapper to run.
2. Sufficiently describe the submapper implementation.
3. Let someone else verify that it is needed, and the implementation works.
4. Allocate the submapper, and keep an eye on backward compatibility. (If kevtris already had it, don't reassign.)

As long as we do this going forward, I don't really see a conflict with kevtris' original spec. If we don't have a known use case for one of kevtris' suggestions, there's no reason to put it in the spec (it can't be implemented until we know what it is, right?). If anybody comes up with something that needs a submapper, explain what it is and we should add it. Pretty simple.

Anything kevtris pre-emptively added because they might be nice for homebrew can wait until that homebrew exists before we add it to the spec. The spec isn't preventing people from trying any new mapper idea they want, and it'll still be here if and when any software is made for that new mapper. The wiki is living and changing, and a central source for this, there's no need to try to guess the future here because we can easily add it when we get there.

[Great Hierophant]

Is the Quattro submapper necessary? According to the NES Cart DB, both the standalone cartridge release and the Aladdin Deck Enhancer release of Adventure and Sports use the same ROM.

Also, for the discrete logic mappers with bus conflicts like UNROM and CNROM, iNES 2.0 does not seem to be able to tell the emulator that this game needs not to have bus conflicts.

[rainwarrior]

Also, for the discrete logic mappers with bus conflicts like UNROM and CNROM, iNES 2.0 does not seem to be able to tell the emulator that this game needs not to have bus conflicts.

The default implementation is not to have bus conflicts.

If there are games that require bus conflicts to be able to run correctly, please list them.

Bootgod's CRCs for the quattro games being the same is interesting. Does anyone have the relevant ROMs to elaborate on the problem?

[lidnariq]

001:5 - fixed 32k PRG for MMC1. I haven't tested all the games yet, but it seems that in at least some cases simply initializing the PRG banks in NROM order at load is already sufficient for compatibility with submapper 0. This might even be true for all of the games?

Requiring specific power-on values that don't come from the hardware is asking for problems.
It's vital to understand that the original hardware MUST be the authoritative version, and adding hacks like this to fake the right behavior will result in romhacks, translations, and homebrew that can't run on the hardware that they're theoretically designed for.
This is the same reason why MMC3 must be capped to 512 KiB—not because we can't make something that looks like an MMC3 that supports more, but because the original hardware must be the reference implementation.

Relatedly:

Also, for the discrete logic mappers with bus conflicts like UNROM and CNROM, iNES 2.0 does not seem to be able to tell the emulator that this game needs not to have bus conflicts.

The default implementation is not to have bus conflicts.

That is patently false. The default implementation DOES have bus conflicts, both in hardware, and in more recent emulators that implement these bus conflicts to keep people from hurting themselves on derivative works.

If there are games that require bus conflicts to be able to run correctly, please list them.

There's at least one instance documented on the wiki page for mapper 003. (as distinct from the page for CNROM)

Anyway, this is the same point. It is absolutely vital that these emulators emulate the hardware as it existed, not an abstracted idealized version of the hardware. Otherwise, there'd be nothing wrong with creating things that only ever worked on Nesticle.

034:2 - Union Bond (Does anybody know what this means?)

Union Bond (a developer)'s mapper seems to be oversize BNROM with PRG-RAM. I don't see why it needs a submapper.

080:? - Description says they were moved to mapper 207.

Has to be referring to Fudou Myouou Den. I have no idea why it's plural.

083:1 - Cony with 2k CHR banking.
083:2 - Cony with other modifications.

I'm pretty certain those things weren't in kevtris's page when the information was copied to the wiki.

----------

No, it's not a vote for crash. When I say it should be "unspecified" this is a vote for it to be the implementer's choice how to handle an inconsistency.

But that's exactly what abstention is, here. Of those four, any implementor can ONLY (consciously or subconsciously) pick one of those four results. And the "crash" choice is the most naïve choice, the one that would happen from what happens if a person doesn't think about it and just makes code that says "Oh, ok, 8KiB of PRG-RAM is marked, allocate 8KiB of RAM" and "Oh, ok, SXROM, I'll index into a 32KiB array".

If you vote to allow contradiction, and don't warn in giant blinking letters HEY THERE COULD BE A PROBLEM HERE YOU SHOULD KEEP IT FROM HAPPENING BECAUSE SOMEONE WANTED TO ALLOW THE FORMAT TO CONTRADICT ITSELF someone WILL implement it poorly. When, not if.

So, should we explicitly add that warning label to the submapper page? (Is it fair to say that you're arguing that that is that an acceptable cost for your clean 16-bit-UID-as-ABI-version ?)

If a crash or insecurity is not a problem for the implementer, it is a valid choice for them to make. If these things are a problem, they will figure out how to prevent it.

People don't intentionally write bugs (bug bounties notwithstanding). People don't intentionally write vulnerabilities (bribes and the underhanded C contest notwithstanding).

And people usually assume that their not-explicitly-security-based software will not be used in a location where such a vulnerability could be exploited. (e.g. the libtiff and libpng vulnerabilities)

All of the pointy bits are put there by the implementer, not by this spec. This concern is outside the realm of what we are doing here.

Specifications have sharp edges. Implementations have bugs. Sharp edges in specifications cause bugs in implementations. A vote to explicitly allow the format to contradict itself is a vote to create bugs later on.

What form of authority would you suggest we defer to instead?

Evidently not kevtris, since you've realized many of his submappers don't even mark a different ABI, nevermind my argument of whether the header should be allowed to contradict itself.

[Great Hierophant]

The chip markings on the Aladdin Deck Enhancer carts match the ones on the standalone carts, so I would guess that the Aladdin Deck Enhancer Quattro carts may scramble the game selections, but otherwise should have no effect.

As far as bus conflicts go, Donkey Kong Original Edition, the official Nintendo hack of Donkey Kong that adds the Cement Factory level into the game and was included with certain Wii consoles, requires bus conflicts to be disabled.

[rainwarrior]

Requiring specific power-on values that don't come from the hardware is asking for problems.
It's vital to understand that the original hardware MUST be the authoritative version, and adding hacks like this to fake the right behavior will result in romhacks, translations, and homebrew that can't run on the hardware that they're theoretically designed for.
This is the same reason why MMC3 must be capped to 512 KiB—not because we can't make something that looks like an MMC3 that supports more, but because the original hardware must be the reference implementation.

Well, there's different format goals. iNES was a compatibility format, lumping compatible boards together where it's convenient to implement them together. Actually, I think this kind of stuff is a good reason to use UNIF. UNIF's goal was accuracy by being very specific about the hardware that it wants emulated.

Specifying a power-on value for 32k MMC1 isn't really asking for problems at all in any practical case. There's only two obscure games that don't, and it doesn't break compatibility with them, or MMC1 games of any other size. It breaks accuracy with them in exactly the same way that emulators break accuracy with most games by initializing RAM, or a myriad of other things. This seems rather trivial and not worth specifying a submapper for. There doesn't seem to be an existing problem with running the 32k MMC1 games and the original iNES format; nobody is doing a CRC check to support them, as far as I can tell, so it seems unlikely that anyone would want to adopt that submapper to solve a problem they don't have.

iNES' goal of compatibility rather than accuracy is actually one of its biggest strengths. It's a big reason why it was more practical to implement than UNIF. Maybe a hundred mapper implementations instead of a thousand board implementations?

I'll put it another way, if somebody needs thorough validation tools for homebrewing, they should go ahead and build those things, but iNES is a terrible place to try and stuff all those needs. It began life as a compatibility format, and it would be a colossal mess to try and turn it into something that can specify every hardware quirk. You're proposing to change a format that was born for one purpose into something completely different. NES 2.0 is a compatibility patch for iNES, not a total change of direction.

The default implementation is not to have bus conflicts.

That is patently false. The default implementation DOES have bus conflicts, both in hardware, and in more recent emulators that implement these bus conflicts to keep people from hurting themselves on derivative works.

Sorry, I misstepped there. I lost track of which mappers we're talking about. For boards that have both bus-conflict and non-bus-conflict versions, no bus conflict has been the default, has it not? If all boards for a mapper had bus conflicts, then yes that is the proper default (and it should be specified in our definition/description of that mapper-- nothing to do with NES 2.0, the iNES 1 version of the mapper should do it).

There's at least one instance documented on the wiki page for mapper 003. (as distinct from the page for CNROM)

So for mapper 3 the question should be the opposite. Are there any mapper 3 games that require no bus conflicts? If yes, we should propose the submapper for this. If not, we don't need it yet.

Union Bond (a developer)'s mapper seems to be oversize BNROM with PRG-RAM. I don't see why it needs a submapper.

Hmm, well if it writes values to the last 3 bytes of PRG-RAM it would conflict with the NINA-01 + BxROM mapper 34 implementation, so that could require the submapper. Anyhow, until somebody can come up with the ROM that needs it I don't think we should worry about it.

[rainwarrior]

Okay, so in response to this change allocating 3 submappers for VRC2/VRC4:

• 023: 8 - this one I don't understand the description of. Is this a compatibility issue for "World Hero"?
• 023: 15 - the mapper articles claim that this is already compatible with the existing PRG-RAM implementation, i.e. it's not a compatibility issue.
• 025: 15 - the description claims that the one extant ROM is already compatible with submapper 0. This is future proofing against a romhack that wants to add an IRQ? I strongly suggest that we should not do this.

Both of 023:15 / 025:15 are clearly invalid by the criteria I think we should be using for submapper allocation, and 023: 8 has a very opaque description so I will need further clarification to understand what it is. The ROMs listed are said to be compatible already, so they're clearly not a valid test case to verify a correct submapper implementation.

This is really digging at what I was just saying about format goals re: compatibility or accuracy. If it's not currently incompatible, it really doesn't need a submapper. We should be solving existing CRC and heuristic problems, or possibly accomodating existing romhacks. We should NOT be future proofing against potential romhacks. It's a waste of our time. (Make the romhack first! We need a test case before it's worth implementing.)

Again, I think emulation accuracy tools for homebrewers and hackers are a wonderful thing to have and I would love to use them, but I think this is a completely different goal than NES 2.0, and should not be shoehorned into it. It seems that lidnariq and I have somewhat mutually exclusive goals here, and some consensus from others on how to proceed would be very helpful I think. I do not wish to engage in an edit war, as that will be simultaneously detrimental to both lidnariq's goals and my own.