byuu's XML format

Discuss emulation of the Nintendo Entertainment System and Famicom.

Moderator: Moderators

User avatar
infiniteneslives
Posts: 2102
Joined: Mon Apr 04, 2011 11:49 am
Location: WhereverIparkIt, USA
Contact:

Post by infiniteneslives »

Just my two cents:

I don't see how one could expect their format to be accepted if it doesn't support unlicensed games especially homebrew. You'll never gain support from the majority of people here. Gaining acceptance from nesdev would be your first step to widespread use I imagine.

Don't forget hardware requires the use hardware description. If you make things difficult/impossible for PowerPak, NESDEV1, Infinitenes, or fpga NES's nesdev will likely never adopt your format.
Near
Founder of higan project
Posts: 1553
Joined: Mon Mar 27, 2006 5:23 pm

Post by Near »

> I don't see how one could expect their format to be accepted if it doesn't support unlicensed games especially homebrew.

No one except people who already have my SNES emulator, and don't feel like downloading a separate NES emulator, are going to use my NES emulation anyway. So you're talking about maybe two people ever being affected by my decision on homebrew here.

That said, the point of the XML spec is to describe every officially released board. If a homebrew game stays within hardware that was commercially licensed, then all it needs is a manifest file to work. You can usually generate them from iNES 1.0 headers, unless they are like the VRC pinouts or something that's not specified, then you'll have to make the manifest by hand to fill in the missing info.

If the homebrew starts getting crazy with custom mappers and non-linear mapping, then well ... too bad. If we have no limitation, then we have to account for any theoretical situation, which is untenable. iNES can't do that, iNES 2.0 can't do that either.

Further, I am not going to waste my time emulating bootleg mappers for shoddy PS1 to NES Chinese ports, sorry. If someone else wants to do that, and add the appropriate markup IDs to the XML format to run those carts, great! Be my guest. The nice thing about using XML is that it's infinitely extensible. So you can add stuff when you need to without breaking older stuff. So for that, you can do <crazybootlegchip><!-- describe segmented PRG, extra circuit boards attached to the initial board, whatever you want --></crazybootlegchip>, and leave the base spec we're discussing alone. Put all the PRG and CHR inside <crazybootlegchip> if you have to, whatever.

> If you make things difficult/impossible for PowerPak, NESDEV1, Infinitenes, or fpga NES's nesdev will likely never adopt your format.

I've been watching discussions on formats for a long time, it's a predictable pattern.
Guy comes up with format, most people throw a fit because iNES 1.0 is just fine, and they don't want to support anything more than their iNES+DB that's already working for them.
Guy tries to get more people on board, gathers suggestions, keeps adding to spec.
Eventually you end up with the complexity of UNIF, and nobody uses it.
Or maybe you streamline it to be fully backward-compatible like iNES 2.0, and nobody uses it.
Either way, most people will still give you grief about it being a new spec, no matter what you do.

Point being, iNES 1.0 is as eternal as CR-LF text files and IPS patches.
I'm not worried about it. The whole world can complain, my emulator will use XML, and come with tools to convert iNES to it. Don't like it and won't use the emulator as a result? Great, no loss to me.
Call it rude, arrogant, inconsiderate, whatever. This is the only way to create a new spec that will ever be used by anyone else: an emu author has to make it, implement it, and require it.
tepples
Posts: 22345
Joined: Sun Sep 19, 2004 11:12 pm
Location: NE Indiana, USA (NTSC)
Contact:

Post by tepples »

byuu wrote:If a homebrew game stays within hardware that was commercially licensed, then all it needs is a manifest file to work.
Thank you. I can get behind a spec that supports at least the mappers currently available to homebrew, which for now means the common discretes supported by the ReproPak (NROM, CNROM, BNROM, GNROM, AOROM, UNROM, and flip-UNROM used by Crazy Climber), their oversize extensions as commonly interpreted by existing emulators, and MMC1. I guess some of my rage just spilled over from the "everything worthwhile to play is licensed by Nintendo; your righteous efforts in developing an original unlicensed game are like filthy rags" tone of early ZapFC.
Further, I am not going to waste my time emulating bootleg mappers for shoddy PS1 to NES Chinese ports, sorry.
I see your point about these. Someone else who cares about obscure mappers, such as CaH4e3, can supply a patch to support such a board. But some of these Chinese demakes, such as FFVII, run on mappers that don't provide much functionality beyond an oversize extension of BNROM anyway, so an xor.gz/ups/bps patch to turn such a game into a proper mapper 34 binary might be one solution until someone else implements a patch to support such a board.

Likewise, someone else who cares could write a tool to convert Pasofami manifests to XML manifests.

And technically, LJ65 might be considered a PS1 to NES demake, albeit not Chinese and using NROM-128 instead of a Chinese bootleg mapper. The series on which "Bottom rotation" is based began with a game on a PS1-based Capcom ZN-2 board.
Or maybe you streamline it to be fully backward-compatible like iNES 2.0, and nobody uses it.
As I see it, part of the problem with lack of adoption of NES 2.0 appears to have come from the fact that Nestopia hasn't had an update in four years.
The whole world can complain, my emulator will use XML, and come with tools to convert iNES to it.
I can support that, just as I supported headerless .sfc files. My sticking point is making sure not to reject the following ROM images:
  • Original homebrew programs that use a licensed board, which you have stated you will accept.
  • Unlicensed games that your emulator's users are likely to have actually owned, such as those by Camerica and Color Dreams. I currently own a copy of Bee 52, and at various times, I have owned a copy of Bible Adventures, The King of Kings, and Exodus.
User avatar
infiniteneslives
Posts: 2102
Joined: Mon Apr 04, 2011 11:49 am
Location: WhereverIparkIt, USA
Contact:

Post by infiniteneslives »

Thanks for that explanation byuu.

With your goals and purpose I agree this format is very fitting. I can definitely relate and support your decision to not support every crazy pirate mapper out there. And the fact a manefest file will allow a homebrew to work solves any rational complaints against your emu mapper support.
Nessie
Posts: 134
Joined: Mon Sep 20, 2004 11:13 am
Location: Sweden
Contact:

Post by Nessie »

byuu wrote:I've been watching discussions on formats for a long time, it's a predictable pattern.
Guy comes up with format, most people throw a fit because iNES 1.0 is just fine, and they don't want to support anything more than their iNES+DB that's already working for them.
Guy tries to get more people on board, gathers suggestions, keeps adding to spec.
Eventually you end up with the complexity of UNIF, and nobody uses it.
Or maybe you streamline it to be fully backward-compatible like iNES 2.0, and nobody uses it.
Either way, most people will still give you grief about it being a new spec, no matter what you do.
I believe xkcd properly describes the situation with NES ROM formats:
Image
3gengames
Formerly 65024U
Posts: 2281
Joined: Sat Mar 27, 2010 12:57 pm

Post by 3gengames »

Too bad there's one standard now and forever, iNES. In 10 years, iNES 2.0 will be there and this won't be anywhere. Deal with it and move on from this garbage.
User avatar
rainwarrior
Posts: 8062
Joined: Sun Jan 22, 2012 12:03 pm
Location: Canada
Contact:

Post by rainwarrior »

This thread is amazing.

I think a comprehensive database for correcting ROM headers is really cool.

I'm not sure what the point is for a new format though. The database solves the problem of bad/inadequate headers; what problem does the new format solve?

The other problem to consider is that if you want to proliferate a new format you must make ROMs and distribute them. Presumably a big reason you want a database instead of just fixing the headers is that you don't want to open yourself up to the liability that distributing ROMs does.

I mean, you can easily make a corrected set of iNES ROMs of all licensed games (barring the few licensed games iNES is not good enough for). The real problem is getting it out there. With a new format you have the same liability problem that comes with distributing ROMs, combined with the format-inertia that has already been discussed.

Anyhow, good luck. I hope you maintain your database project, because I think that part is worthwhile. If you think you can get a new ROM format going, well, go ahead and try, but I think you have yet to present a compelling problem that this new format will solve that the database alone doesn't.
User avatar
kyuusaku
Posts: 1665
Joined: Mon Sep 27, 2004 2:13 pm

Post by kyuusaku »

FitzRoy, I'm sorry if I've attacked your character. You are one stubborn, self-promoting dude (or dudette) though :D (aren't we all?)
FitzRoy wrote:You pretty much flat out said that board serials could imply everything.
Not everything. Boards do decide which chips they accept in their layout, I never meant to imply that boards can imply when a chip location is populated, and with what. My argument is that there isn't a need for chips to be listed (unless everything was for completeness, or netlist emulation), and instead argue that the IC population are attributes of the board.

Sometimes you wouldn't even be able to list a chip when for example the name was scratched off. Also since many boards don't have IC designators this makes it a little tricky to pull off elegantly because you're listing ICs without context. What if a board may accept one, two or zero '139 all affecting mapping (they have pullups on their inputs)? When you just list some chips it may be difficult or impossible for the emulator to decide on the appropriate logic, much less put together the logic from the description. Or maybe the cart uses programmable logic (a few licensed FC games do) and it's extremely difficult to know if the logic is revised or not, it's difficult to even denote two revisions of the same chip without going back to arbitrary attributes.

Maybe I'm using the wrong paradigm, perhaps the board should enumerate each set of PCB pads and what's in them. That is a LOT of info for emulators with hardcoded logic though.

In my opinion the best formats won't have these kind of personal touches, but if they're necessary given software constraints, don't beat around the bush and make up complex solutions that still fall short, might as well take the easy, simple solution and make it elegant.

---

RE non-linear mapping: not only do SNES games do this, but a few Neo Geo games do it for graphics protection and lots of arcade games have several ROMs containing various types of tile layouts all on the same bus. Sometimes they can't be addressed linearly because the video logic pre-decodes the ROMs by their purpose. Sometimes instead of wasting resources on creating a virtual address space each ROM's address lines may come directly from its respective system. I don't have any specific instances of it but ROMs have been known to have their address lines swapped from their JEDEC pinout.
There are no jumper pins on which an actual jumper shunt would go
Solder jumpers are jumpers too.
But the mapping code within the emulator isn't written with a specific size in mind. It basically expects a range of validities and will function according to what you give it.
I'm not sure I understand. Since you are letting people input arbitrary byte-aligned ROMs there's ambiguity on how to interpret non-2^n values. Also if only 2^n values or a small series of them are valid, then it'd probably be better to not use a byte count at all instead opt for the industry-preferred kilobits, megabits etc.
> This I have to hear.
I am genuinely interested, who here wouldn't be?
An NES would be a very unlikely candidate to emulate the NES.
A NES doesn't have to, but since it can why not allow it to use an alternative (like 16-bit checksums) at the user's own risk? I wasn't thinking so much about the feasibility of hashing one file, more like the time to audit a ROM set on tomorrow's computing devices which may or may not have crypto-acceleration which may or may not be accessible under 3 layers of virtual machine to user programs.
then one could leave the hashes for a validation tool.
Oh, right. I guess I'm still fuzzy about how the format is meant to be used.

After the emulator opens the ROM set, does it directly look up the game's name in the database or does it directly hash the program and character file if applicable and search for a match? Or does the ROM set now include the XML?

Edit edit: or is the XML meant to be used with a visual ROM loader?
Last edited by kyuusaku on Mon May 21, 2012 2:07 pm, edited 2 times in total.
Near
Founder of higan project
Posts: 1553
Joined: Mon Mar 27, 2006 5:23 pm

Post by Near »

> Unlicensed games that your emulator's users are likely to have actually owned, such as those by Camerica and Color Dreams. I currently own a copy of Bee 52, and at various times, I have owned a copy of Bible Adventures, The King of Kings, and Exodus.

In that case, I really hope Bible Adventures uses some non-standard hardware. Religious proselytization in a children's video game is disgusting :P

> In 10 years, iNES 2.0 will be there

Hahahahahahah, keep dreaming. I'll bet you all the money in my 401K that won't happen. Even No-Intro has zero interest in it.

Moving on from iNES requires more than 10% of people to agree on anything. Judging from this thread, that'll be one cold day in hell.

You must remove support for iNES 1.0 first, and nobody here is idealistic enough to do that. Casual gamers will never upgrade their ROMs if you continue to support them in 1.0.

> and this won't be anywhere. Deal with it and move on from this garbage.

http://www.youtube.com/watch?v=XgRLG1tl9DE#t=47s
Last edited by Near on Mon May 21, 2012 1:34 pm, edited 1 time in total.
User avatar
FitzRoy
Posts: 142
Joined: Wed Oct 22, 2008 9:27 pm
Contact:

Post by FitzRoy »

byuu wrote:Imagine a translation released with an included board file (or manifest as I call it.) It has "hash=(32 characters)". How does your emulator know how to match it ... is that SHA256, or SHA3? There's no guaranteeing future hashing algorithms won't use the same number of bits, and no guaranteeing old-style data won't be loaded in.
Basically, the game would fail to load, which is what would and should happen. If a sha256 hash keeps working, then why would anyone switch to sha3?

But let's say you legacy supported everything and just pretended that people updated. You could write a version tag into every manifest file. If version 11 adds sha3 support, then the emulator would know by the version tag what hash type the manifest contains.

The SNES loves to do this, which is why the SNES has the map mode+offset+size attributes.
*If* any NES games do this, then it's most likely a detail of the mapper. Mappers are kind of a recurring trend on the NES.
If the SNES did this, then you're referring to 1YAON boards and stuff where the program is multi-chip, but could have or was released single chip on a 1A0N. If the 1YA0N board behaves differently than a 1A0N board, but the behavior is superfluous in output to the 1A0N variant, then I see no reason to release the rom in both variations and give them each their own mapping entry. The destination is the same even if the journey was different, it's redundant.
Last edited by FitzRoy on Mon May 21, 2012 1:35 pm, edited 1 time in total.
3gengames
Formerly 65024U
Posts: 2281
Joined: Sat Mar 27, 2010 12:57 pm

Post by 3gengames »

who the hell says you have to remove support? that's why there's a 2.0 flag, to tell you when to use the extensions. You shouldn't need to change anything, people just have to write newer emulators. And oh yeah, like THIS is going to take over iNES. That'd be a colder day in hell.

Image
Near
Founder of higan project
Posts: 1553
Joined: Mon Mar 27, 2006 5:23 pm

Post by Near »

> Basically, the game would fail to load, which is what would and should happen. If a sha256 hash keeps working, then why would anyone switch to sha3?

I am not catering to people who won't upgrade. Life is too short to convince people like those in this very thread. I'd have as much luck convincing people to switch religions and political parties.

Whatever, I tried. It's your format, so go with hash if you feel it is best.

> If the SNES did this, then you're referring to 1YAON boards and stuff where the program is multi-chip

Dude, if there's one thing where anyone should trust what I say, it's SNES stuff.

All games do this stuff. Banks $00-3f and $80-bf reserve A15=0 for I/O, RAM, and other uses. You can't map a 100% linear ROM in there unless its exactly 32KB.

It's not a big deal. I just use the manifest to tell the emulator where to mirror ROM to.

> And oh yeah, like THIS is going to take over iNES.

No one ever said it was going to.

I would actually prefer pessimists to stay away from my format entirely. I'd prefer all the casual gamers use your emulator instead (assuming you have one. But yeah, besides FitzRoy, who here doesn't?) Better you deal with them than me.

I wish I wasn't lumped into this thread in the first place. I'd rather discuss my format on my board.

Yeah, let's do that. If anyone would like to continue the discussion on my spec, post at board.byuu.org. This is the ZapFC thread, so discuss ZapFC here only please.

Image
User avatar
FitzRoy
Posts: 142
Joined: Wed Oct 22, 2008 9:27 pm
Contact:

Post by FitzRoy »

byuu wrote:Dude, if there's one thing where anyone should trust what I say, it's SNES stuff.

All games do this stuff. Banks $00-3f and $80-bf reserve A15=0 for I/O, RAM, and other uses. You can't map a 100% linear ROM in there unless its exactly 32KB.

It's not a big deal. I just use the manifest to tell the emulator where to mirror ROM to.
I do, I just am not understanding programmer speak for what's going on here. When you were working with me on NES db markup, at no point did you say that it was necessary to define hex ranges and stuff in there. I know that you do that with the SNES manifests, but is it really necessary or is it a function of not having and calling "[board].cpp" like you do with NES?
Near
Founder of higan project
Posts: 1553
Joined: Mon Mar 27, 2006 5:23 pm

Post by Near »

Are you sure you want to discuss this here? It'll detract from your NES format discussion. But well, think of it like this:

On the NES, you can have the MMC1 or MMC3, and these chips control what ROM data is mapped to what bus address range (eg it controls what the CPU reads from certain areas.)

Their biggest use is that the NES is limited to 32KB ROM space. So you write to these chips to ask it to swap some (or all) of that space with other ROM data.

The only analogues on the SNES are the SPC7110, S-DD1 and the BS-X town cartridge. Possibly the SA-1 if you stretch the definition. And true to form, when those are described in my markup, they just say "these bus addresses can be remapped by these chips."

When it comes to all other SNES carts, they can easily map up to ~10-12MB into the existing address space without a bank mapper. So in all honesty, the bank remapping is pretty stupid. There was no SNES game that ever required that functionality. It just made SO and FEoEZ harder to develop.

So most SNES games had their own mapping chips. The most popular was the MAD-1, but it was overkill for the simplicity of games on eg 1A0N boards. Those just used simple logic gate chips like the 74LS series. What these chips do is translate SNES bus addresses to ROM and/or RAM chip addresses. It selects the appropriate chip for an address, and transforms the address to be linear for that individual chip.

So in a very real way, we could replace all of the <map address=... offset=... size=...> stuff with <mapper type="MAD-1" pinout="..."/>, and reproduce the map that way as well. But that's a lot like saying "we can eliminate half the board types on the NES", a lot of these boards are just simple 74LS logic chips once again. Only they're a tiny bit more advanced, since they have primitive bank mapping by way of being writable (they are decade counters, usually.)

So when we say "NES-CNROM", we are really saying "board with a 74LS decade counter connected to these pins of the ROM address line." And when we say "S-DD1", we are saying something similar about that mapper+ROM chip. And when you talk about a0=x,a1=y pinout on VRC-n boards, what you're really doing is describing the only pins that change between officially released titles. The real VRC chip probably has something like 20 - 80 different pins on it. It's a huge jump to allow arbitrary connection of those pins to support any possible configuration (even non-sensical ones like connecting +5V to bus audio left channel, for instance.)

Now it would be nice to come up with an implementation that allowed us to wire every pin between every chip in XML, but it would be an absolute nightmare to emulate such a beast, and performance would drop dramatically. Think about that Pong circuit simulator I used to talk about, or Visual 6502. And I think we both know that's not an option at this point anymore.

My SNES XML maps build a 16MB mapping table, so that when you read an address, all it has to do is a memory fetch to determine what type of memory is there, then call a function pointer form a table indexed by that type. That function can mask the address, and return the underlying memory. Even this is already painful, but it's very linear.

Trying to do a chip-level simulation would be far more complicated. And I'd be surprised if we didn't have NES carts with more than one 74LS chip.

There is also the fact that with the SNES, I'm more interested in homebrew development there. For instance, there would be no way to do the FEoEZ expansion nor run neviksti's Star Ocean 96mbit patch, if not for the XML remapping. And we don't even really have all the details of the pinouts for all logic mappers on the SNES side, so we couldn't simulate that even if we wanted to.

Technically, the SNES XML format I use can do everything SNES-used 74LS and MAD-1 chips can do, and much more. And it results in faster emulation. But it does make the spec more complex. The same, however, cannot be easily done on the NES, because unlike the SNES, the NES almost always needs to bank-swap in more ROM data. You can't describe something like that with just a plain list of range+offset+length entries. The underlying content changes dynamically.

This is basically the big problem I have with MAME/MESS. You can't mash every system ever into the same structure. It just makes them all overly complicated and confusing. Instead, you should take into account the realities of each system, and make something that works best for it. And I'll say right now, I don't currently have the expertise to do this for the NES, like I do for the SNES. So my spec is very much going to be evolving for the next several years as I learn more.

To be quite honest, I don't even think it's appropriate for emulators to share an XML game mapping. It's highly dependent upon how the emulator was developed. It really should all be done internally with a database for games. But the problem there is you just can't support homebrew that way, so at some point you -have- to describe something for board formats. And then you get people jumping down your throat thinking you're out to kill iNES 1.0, which is nonsense.

As such, worrying about a hypothetical NES cart with non-linear mapping of two PROMs is kind of silly. You're going to need a logic chip to decode the bus address and select the appropriate chip anyway. So we specify that logic chip in the markup, and the emulator will know what to do with it.
tepples
Posts: 22345
Joined: Sun Sep 19, 2004 11:12 pm
Location: NE Indiana, USA (NTSC)
Contact:

Post by tepples »

byuu wrote:Are you sure you want to discuss this here? It'll detract from your NES format discussion.
It's probably a good idea to split at the revival point, seeing how little of this relates directly to FitzRoy's proposal.
Post Reply