a new alternative to iNES

[etabeta]

Even if I've spent quite some time, in the past year, to improve compatibility in the NES emulation of MESS (by adding new mappers, by adding support for UNIF files and by implementing preliminary iNES 2.0 support), I have never been really happy with the current NES formats.
Being my background mainly related to MAME, I've always been under the impression that gluing together the different chip contents + an 'arbitrary' header was not the best solution. Especially, given the large number of ROMs floating around with the wrong header (while working on pirate mappers, I have found tons of chinese roms generically assigned to Mapper 4 or 0, which in fact required different features to work), and the generic lack of consensus on some mapper numbers for obscure chinese carts.

This is why, starting from today, MESS publicly supports loading zipfiles containing separate PRG and CHR files, in addition to iNES and UNIF!
The matching between the ROM files and the correspondent boards is obtained through a .xml database which gets read when carts are loaded and which is public for anyone to use (as long as new findings about obscure boards are made public so that anyone can use them): you can get a copy from this link (click on 'plain' and save the ~2.5MB xml file)

http://git.redump.net/cgit.cgi/mess/tree/hash/nes.xml

Before getting into the xml details, I would like to point out that only 3 things are required to add support for these files in any emu:
1. an xml parser, to read info from nes.xml about the board type, the memory sizes, the mirroring, the dispwitches, the peripherals, etc.
2. the capability to load separate files from inside a zipfile based on checksums or filenames (like MAME, Nebula and FBA do for arcade romsets)
3. a lookup table to pass from the board type [read in nes.xml] to the correct mapper emulation. NEStopia has a lookup table of this sort in src/core/boards/NstBoard.h, if a reference is needed

once the file is loaded in memory and the emulator has chosen the appropriate read/write handlers, emulation can go on as it would usually do after loading an iNES file to memory.

Now let's focus on the xml format. Let's examine a typical entry (in this case for Aerobic Studio)

Code: Select all

	<software name="train3as" supported="partial">
		<!-- Serial: FT-03 - Release Date: 1987-02-26 - Alternative Title: ファミリートレーナー③ エアロビスタジオ -->
		<description>Family Trainer 3: Aerobics Studio (Jpn)</description>
		<year>1987</year>
		<publisher>Bandai</publisher>
		<part name="cart" interface="nes_cart">
			<feature name="pcb" value="BANDAI-PT-554" />
			<feature name="mirroring" value="horizontal" />
			<feature name="peripherals" value="ftrainer" />
			<dataarea name="prg" size="32768">
				<rom name="aerob pr" size="32768" crc="f8da2506" sha1="c14075b1b7c98b684870e8d85b7ff2b462e3cd10" offset="00000" />
			</dataarea>
			<dataarea name="chr" size="32768">
				<rom name="aerob ch" size="32768" crc="ff1d2bfd" sha1="c41872d3c28230f3a907db9bf46568ca8a789e22" offset="00000" />
			</dataarea>
			<dataarea name="adpcm" size="262144">
				<rom name="m50805" status="nodump" offset="00000" />
			</dataarea>
		</part>
	</software>

Each game is in a <software> node with a "name" attribute (a shortname like MAME uses: the emulator will then look for the files inside a train3as.zip file) and a "supported" attribute (this is MESS specific, but other emus can strip it out with e.g. a perl script).

Then, we have the basic info which could be shown in the emu UI:
- <description> for the complete name of the game
- <year> & <publisher> for the corresponding info
the commented out fields (serial, release date & alt titles) might be re-added later, I'm not sure (this xml format is used for many systems in MESS, so some compromise had to be accepted)

Finally, <part> includes everything an emulator needs to know to emulate this specific piece of software (the "name" and "interface" attributes are internally used by MESS to know this is NES and not another console).

Ideally, an emulator could start any file of the xml list by simply parsing the part node:
- from <features>, you would find out: which "pcb" the game uses (in this case a Bandai CNROM variant containing an additional sound chip), so that you can choose the required mapper; if the game has hardwired "mirroring", horizontal in this case; if the game supports particular peripherals; etc. for some particular pcb there might be specific features, e.g. the pin settings for protected CNROM games, or the pin connections for Konami VRC-2, VRC-4 and VRC-6 games, or the chip revision for MMC1 and MMC3 (to handle different MMC3 IRQ or Mapper 1 vs. Mapper 155 emulation)...
- from <dataarea>, you would find out which chips were present on the pcb: in the example we have the expected "chr" and "prg", plus a "adpcm" area for the content of the audio chip, if we ever manage to dump it. Other possible 'dataarea' are "vram" (if present on the board), "wram" and "bwram" for PRG RAM with or without battery, "mapper_ram" and "mapper_bram" for mappers which have internal RAM with or without battery like MMC5, MMC6 & Taito X1-005. At start, an emu should simply load prg & chr, and take note of the other dataarea to setup VRAM & WRAM like it would do by parsing the iNES header
- from <dipswitch>, you would read if the cart had switches on it (this is needed for a few pirate multicarts)

Here are other three examples for Excitebike, Shin Satomi Hakkenden and Taito Grand Prix, respectively

Code: Select all

	<software name="excitbik">
		<!-- Serial: NES-EB (EEC/NOE/ESP) - Release Date: 1986 -->
		<description>Excitebike (Euro)</description>
		<year>1986</year>
		<publisher>Nintendo</publisher>
		<part name="cart" interface="nes_cart">
			<feature name="pcb" value="NES-NROM-128" />
			<feature name="mirroring" value="vertical" />
			<dataarea name="prg" size="32768">
				<rom name="pal-eb-0 prg" size="16384" crc="0b5667e9" sha1="8c37839c645d54184df6273e481527d4cfa312d2" offset="00000" />
				<rom size="16384" offset="4000" loadflag="reload" />
			</dataarea>
			<dataarea name="chr" size="8192">
				<rom name="hvc-eb-0 chr" size="8192" crc="e5f72401" sha1="a8bf028e1a62677e48e88cf421bb2a8051eb800c" offset="00000" />
			</dataarea>
		</part>
	</software>

	<software name="shinsato">
		<!-- Serial: TDF-91 - Release Date: 1989-12-08 - Alternative Title: æ–° é‡Œè¦‹å…«çŠ¬ä¼  光㠨闇㠮戦㠄 -->
		<description>Shin Satomi Hakkenden: Hikari to Yami no Tatakai (Jpn)</description>
		<year>1989</year>
		<publisher>Toei Animation</publisher>
		<part name="cart" interface="nes_cart">
			<feature name="pcb" value="HVC-SNROM" />
			<feature name="mmc1_type" value="MMC1B2" />
			<dataarea name="prg" size="262144">
				<rom name="tdf-91-0 prg" size="262144" crc="23e9c736" sha1="835a0f9a3ddd516cecbccf34fbe8f7632f297558" offset="00000" />
			</dataarea>
			<!-- 8k VRAM on cartridge -->
			<dataarea name="vram" size="8192">
			</dataarea>
			<!-- 8k WRAM on cartridge, battery backed up -->
			<dataarea name="bwram" size="8192">
			</dataarea>
		</part>
	</software>

	<software name="taitogp">
		<!-- Serial: TFC-TG-5500 (15) - Release Date: 1987-12-18 - Alternative Title: タイトーグランプリ æ „å…‰ã ¸ã ®ãƒ©ã‚¤ã‚»ãƒ³ã‚¹ -->
		<description>Taito Grand Prix: Eikou e no License (Jpn)</description>
		<year>1987</year>
		<publisher>Taito</publisher>
		<part name="cart" interface="nes_cart">
			<feature name="pcb" value="TAITO-X1-005" />
			<feature name="x1-pin17" value="CHR A17" />
			<feature name="x1-pin31" value="CIRAM A10" />
			<dataarea name="prg" size="131072">
				<rom name="pr0 x3-018" size="131072" crc="17627d4b" sha1="5a8680dce5f2f2ecb28ef592b7418b2371fe98f4" offset="00000" />
			</dataarea>
			<dataarea name="chr" size="131072">
				<rom name="ch x3-019" size="131072" crc="505aa340" sha1="8890e530e9b1485bfeb7e73669104edce8fa625f" offset="00000" />
			</dataarea>
			<!-- 128b Internal RAM (battery backed up) in the X1-005 chip -->
			<dataarea name="mapper_bram" size="128">
			</dataarea>
		</part>
	</software>

Notice that, as shown by Excitebike, the "prg" dataarea is always at least 32KB wide, with smaller roms reloaded/mirrored where necessary

A few more remarks:

Concerning accuracy of the info in the xml list, the first part of the list is very precise, being based on the latest version of Bootgod's database.
On the other hand, the second part is not so complete (yet): it contains files (around 2500) from various other collections (GoodNES, nointro and NonGoodNES), with pcb, mirroring and vram/wram settings taken from the headers. As such, there might be mistakes which I hope to fix little by little... Moreover, additional info are mostly missing (I will add them next, but I thought checksums had definitely priority over Manufacturers and Release Dates ) and some Alt sets might actually be bad dumps which got incorrectly labeled, and they will need further investigations to be sorted out.
However, I think the current list is not that bas as a starting point, given that 95% of the games shows something when loaded

Concerning included file, you might notice that the xml list does not contain hacks or translations: this is done in purpose, since in my opinion these should be made available as IPS patches (to be applied at loading time, after the separate prg & chr have been loaded into memory). On the other hand, dumps from pirate carts are included.

Finally, concerning the file availability (a not so trivial question, given that it's what basically killed the nice UNIF format), conversion from iNES & UNIF is pretty easy since it mainly consists of splitting PRG and CHR chunks from the existing files: MESS is already capable of dumping out the data when the user loads one of his files, if required, and rommanager programs like clrmame or romvault can use the xml list to create zip archives with the correct names from the split files. Also, there is a group of users actively splitting the files so that ready-to-use zipfiles might appear soon-ish through rom collector websites. I'm not sure this will be enough to gain some momentum,
but I hope I can find some more people not exactly happy with the iNES format and the arbitrary mapper numbers

Of course, I'd be interested in feedback and suggestions (as well as critics, of course) and I'm ready to answer to any question which might arise.

That's all folks. Thanks for reading. Cheers.

[tepples]

A bit of constructive criticism:

This is why, starting from today, MESS publicly supports loading zipfiles containing separate PRG and CHR files

In other words, "Now supports Pasofami split format!"

The matching between the ROM files and the correspondent boards is obtained through a .xml database

So it's almost Pasofami, just with the board spec in a central XML database instead of a .prm file. But then how can a homebrew game add itself to the database? Should a zipfile include its own nes.xml? Or did you intend it only for ROM images of games that have been released commercially or otherwise satisfy English Wikipedia's notability criteria?

Code: Select all

	<software name="train3as" supported="partial">
		<!-- Serial: FT-03 - Release Date: 1987-02-26 - Alternative Title: ファミリートレーナー③ エアロビスタジオ -->

Shouldn't text in XML be in UTF-8 rather than Shit-JIS?

Code: Select all

	<software name="excitbik">

Who registers software names and publisher names?

Code: Select all

		<description>Taito Grand Prix: Eikou e no License (Jpn)</description>

"no License" sounds like something Tengen or Color Dreams might do

Concerning included file, you might notice that the xml list does not contain hacks or translations: this is done in purpose, since in my opinion these should be made available as IPS patches (to be applied at loading time, after the separate prg & chr have been loaded into memory).

If you want to handle them like MAME handles clones, a zipfile with an IPS in it would have the main ROM as its parent. But then IPS isn't so good at handling disassemble-patch-reassemble hacks such as the newer SMB1 hacks based on Doppelganger's SMBDis. Those would be better served by xdelta than by IPS, as KaioShin of Romhacking.net suggested to me.

[3gengames]

Well I definently think this is cool but I kinda think that it's a bad thing. iNES is the standard for a reason, so it can be standardized. I don't think MESS should be trying to change it, it should conform to it IMO. I also believe that maybe it should have this data in it if needed but just solve it via checksum and ROM size list, we shouldn't need a whole new zip file with multiple files and such when with iNES we just have one file that if need be, we could find what game it is through a database inside the emulator/program (Bootgod's site is amazing) Well thats my opinion but others can definently go off to a new standard it won't matter to my FCEUX and NESASM3

[MottZilla]

You're not likely to succeed in replacing iNES with a far more complicated format. iNES has some short falls but overall works well. Beats implementing some XML parser and other such things.

[etabeta]

thanks a lot for the comments, tepples

In other words, "Now supports Pasofami split format!"

I was not sure Pasofami format was exactly like this (no files seems to be still available), but then yes: I have resurrected the separate PRG/CHR thing

But then how can a homebrew game add itself to the database? Should a zipfile include its own nes.xml? Or did you intend it only for ROM images of games that have been released commercially or otherwise satisfy English Wikipedia's notability criteria?

well, so far I tried to list all and only the carts released for the real hardware (both licensed and unlicensed).

one of the reasons we added xml lists to MESS (for several systems) is that we received a lot of bug reports due to bad dumps. In the NES case we also had reports due to corrupted headers, or headerless roms (which got spread out with early nointro datfiles). this xml list offers a list of best-dump-to-knowledge to check bugs against, and therefore I was initially concerned with released games for the most part

eventually, I'd be interested to add homebrew programs which run on the real hardware, but I simply haven't got around to it yet.

shipping a single-set xml file with the software is another possibility as well, I really haven't spent too much time on this yet.

Shouldn't text in XML be in UTF-8 rather than Shit-JIS?

to say the truth, I directly extracted the Japanese titles from bootgod's db, without checking the encoding. I will check the issue (especially if we decide to uncomment the alternative titles)

Who registers software names and publisher names?

so far I've been the only one working on the list (that's why additional info are often missing: for start, I devoted a lot more attention to the checksums than to titles/manufacturer/release date), but now that it's public any contribution and help would be very welcome.

we also plan to eventually create some sort of PHP interface in our wiki to make list improvements a community effort, but we're still working on it

If you want to handle them like MAME handles clones, a zipfile with an IPS in it would have the main ROM as its parent. But then IPS isn't so good at handling disassemble-patch-reassemble hacks such as the newer SMB1 hacks based on Doppelganger's SMBDis. Those would be better served by xdelta than by IPS, as KaioShin of Romhacking.net suggested to me.

well, if these hacks get released as iNES files rather than patches, then they might be as well released as separate PRG+CHR with an additional xml file, like homebrew...

however, this is definitely one of the reasons we would never drop immediately iNES even if this format would meet immediate consensus and success: too many nice hombrew programs and interesting ASM hacks are only available in iNES format and sometimes an IPS (or UPS) patch is not enough

[etabeta]

Thanks for the replies to 65024U and MottZilla as well. I hope you won't get offended if I merged the counter-replies to your critics in a single post

Well I definently think this is cool but I kinda think that it's a bad thing. iNES is the standard for a reason, so it can be standardized. I don't think MESS should be trying to change it, it should conform to it IMO.

MESS supports around 150 different iNES mappers and will continue to support the format, because too many users are only interested into playing and will simply keep using their old iNES files.

there is no big kill-iNES-and-impose-a-new-standard conspiracy behind this

just an(other) attempt to overcome some of the iNES limitations with an alternative approach

You're not likely to succeed in replacing iNES with a far more complicated format.

let me clarify this point again: I do not plan to replace iNES files.

that said, I do not think it is a very complicated format (except for the xml parsing): say that you have the parser, then you simply
1. read the xml, extracting pcb type and the info about PRG and CHR checksums and sizes
2. load from zip the files with the expected checksums
3. store the files in memory with prg followed by chr

now your emulator can treat the file like a iNES one! simply use the pcb type to choose mapper, divide prg size by 16k and you get prg_chunks, divide chr size by 8k and you get chr_chunks... then you're ready to start emulation.

of course, if you do not want to add an xml parser (despite libexpat being free), the format becomes complicate to support.

however, I'd like to stress once more that I'm just offering an alternative, not trying to evangelize other emu authors. this alternative seems to work pretty fine in MESS, but I agree that it might not fit other people needs.

Overall, I'm already happy you spent some time reading my posts

I also believe that maybe it should have this data in it if needed but just solve it via checksum and ROM size list, we shouldn't need a whole new zip file with multiple files and such when with iNES we just have one file that if need be, we could find what game it is through a database inside the emulator/program (Bootgod's site is amazing)

well, my xml list is generated from bootgod db and it just adds a lot of carts which haven't still be verified with bootgod's quality standards

also, MESS already have a series of crc hacks to handle multiple boards sharing the same mapper... I've just always thought that relying on checksums sort of defeat the point of using files with an header (see below)...

about the choice of separate files, I made the following reasoning:
1. let's say you want to use an internal crc list or an internal database like e.g. NEStopia does... well, doesn't this beat the whole point of having an header? if you have to identify boards based on the crc, then you could add to your database the mapper number as well, and the header would become pointless/unused

2. once the header is not so useful anymore, we might start thinking about headerless roms... but then why don't support the real content of the carts (like documented by bootgod) instead of files which glue together PRG and CHR? I know my MAME experience is probably the reason behind this second point, but if it has proven to be working for arcade games, why should it be so bad for NES?

iNES has some short falls but overall works well.

except for the need of submappers in many cases (mappers 32, 34, 71, 78, 113, 153, 185 & 242), except for not being able to natively support Galaxian with its 8K PRG, except for some Chinese dumpers assigning new mapper numbers without anyone tracking them (except maybe Cah4e3), except for the wrong/corrupted headers the users might have, except that even if you fix the headers or upgrade them to iNES2.0 next GoodNES might change them back or the users might never update the headers in their collections... etc... etc...

I know that iNES is not that bad (and MESS will keep supporting it, of course), but I was personally not satisfied with it

Well thats my opinion but others can definently go off to a new standard it won't matter to my FCEUX and NESASM3

If you're simply not interested in the thing (or you don't want to touch xml), I respect your opinion.
but if you see any design flaw which makes the format incompatible with your projects, please let me know: I'm always open to suggestions

[RetroRalph]

I think moving past INES is an important thing for the future. However I'm not sure some XML + rom thing is the best way forward. Firstly it doesn't help either emulator authors or end users. With no standardized compression/format for the actual ROM data we will still be left to support a myriad of formats for the data, or a format such as ZIP which is antiquidated. Who wants RAR, ZIP, 7ZIP, TAR, GZ, etc support in every single emulator? It's a waste.

Secondly, it seems XML is favoured because of the ability to 'be liquid' in the future when it comes to adding new features. Considering the games we are covering here are almost 30 years old, we pretty much know what they contain, the best solution is to have people spend the hundreds of hours necessary to write appropriate things for each game, and bundling data into appropriate containers. Not adding simple INES -> XML converters which do nothing except change the data into another form.

Any solution which doesn't involve some people spending the hundreds of hours going over each game isn't really a solution and will never take off. I know from my experience with RetroCopy and the new .GAME format, without people helping you with the databases there is pretty much no point in trying a new format.

As a developer I look at what you have done etabeta and go, what do I get out of it? If you can't offer developers a positive benefit they aren't going to take a new thing up. MESS isn't like MAME in that it can dictate formats as very few people actually use it. Without proper consultation of developers it's pretty much going to be like running on a treadmill any time you waste on such an endeavor.

[tepples]

eventually, I'd be interested to add homebrew programs which run on the real hardware

With PowerPak becoming more widespread, it's far more common for homebrew to work as advertised on an NES.

Because a homebrew game might be released early and often, a database (or section) designed for homebrew should probably not list outdated dumps. For example, the entry for LJ65 on pdroms.de lists only the latest version (0.41). But common ROM cataloging tools are designed for commercial games, and far more prototypes can be found for a homebrew game than for a commercial game. The database might end up with entries for versions 0.01, 0.02, 0.03, 0.04, ..., 0.45, 0.46 of a program, as if they were alternate dumps. It becomes complicated when cataloging tools such as GoodNES include a function to spit out NESMiss.txt, a report listing ROMs in the database that the tool could not find. The presence of outdated dumps in the default view of NESMiss.txt encourages people to try to collect a "compl33t GoodNES set!!!11" including every outdated prototype version ever released. That's why the copyright screens of very early versions of LJ65 (formerly Tetramino) included notices to the maintainer of GoodNES that they weren't yet ready for wide distribution, let alone for the database.

And while researching this reply, I found your post in a discussion on how NESMiss.txt expresses excluded ROM categories. (And is that the distributed.net cow head in Cowering's favicon?)

I hope you won't get offended if I merged the counter-replies to your critics in a single post

In fact, that's common practice on this and many other phpBB boards I've used. The problem with merging happens when a comment mixes replies to on-topic and off-topic posts, and a moderator wants to split off the OT discussion. Moderators can split a topic but not an individual post.

of course, if you do not want to add an xml parser (despite libexpat being free)

It's free as in speech, but not free as in memory. Some NES emulators run on platforms with 4 MB of RAM or even less: see nesDS and PocketNES.

if it has proven to be working for arcade games, why should it be so bad for NES?

Apart from "arcade consoles" such as PlayChoice, CPS, and Neo Geo, a given arcade system board supports only one or a few games, and they have far less of a homebrew scene than consoles. (There's Vantris, a falling block game for Vanguard hardware, and what else?) This makes adding a game less cumbersome, as the emulator developer has to add support to the emulator's code anyway.

except for the need of submappers in many cases (mappers 32, 34

I see 34 and think BNROM. The wiki page for 34 claims that BNROM and NINA-001 are easy to distinguish: NINA-001 has CHR ROM.

[etabeta]

As a developer I look at what you have done etabeta and go, what do I get out of it? If you can't offer developers a positive benefit they aren't going to take a new thing up.

well, it replaces the need of a mapper number with a string of the board name: there would be no risk of multiple boards assigned to the same mapper, or the need of changing number to the mapper at a later stage (quite difficult after you have released a rom with the wrong header). nor limits on the number of mappers would be present anymore.

also, from the point of view of emu authors, with the xml list VRC2, VRC4 and VRC6 code can be simplified a lot, based on the pin settings (which basically shifts which bits control chr/prg bankswitch)

similarly Taito X1-005 and Mapper 185 games can be setup more easily at start without a crc check, but simply parsing accurately the pin 'features' in the xml

again, the controller needed by some game can be retrieved by the xml and also the dipswitches that some pirate multicart have.

all these aspects could be handled by using a list of crc in the emulator source, but it's much more transparent when externalized.

also, it's easier when a new cart gets dumped: say you put in the source the dipswicthes for a 65-in-1 cart and then a 77-in-1 different cart get dumped with the same cart hw but different switch settings: if you use an internal crc list you have to release a new version of the emu with the new crc; in this way you just have to add the xml entry and the emulator would read the enw settings without even the need of recompiling it...

Secondly, it seems XML is favoured because of the ability to 'be liquid' in the future when it comes to adding new features. Considering the games we are covering here are almost 30 years old, we pretty much know what they contain, the best solution is to have people spend the hundreds of hours necessary to write appropriate things for each game, and bundling data into appropriate containers. Not adding simple INES -> XML converters which do nothing except change the data into another form.

except that a few years ago pirate carts with dispwitches started to get dumped. something completely unexpected when iNES got created...

except that a couple of years ago, bootgod proved that some boards (CNROM now assigned to mapper 185) have a copy protection involving particular pins on the board and that each setting would result in a different protection scheme.

there are continuously new findings about these games, despite their age: as such, we could expect some more extension to be needed in the future (even if for smaller and smaller tidbits)

Any solution which doesn't involve some people spending the hundreds of hours going over each game isn't really a solution and will never take off. I know from my experience with RetroCopy and the new .GAME format, without people helping you with the databases there is pretty much no point in trying a new format.

erm... did you take a look at the link in the original post? the database has already been created (by me, partially based on bootgod amazing website) and it lists ~3600 images... of course, I would appreciate *a lot* correction and external contributions, but the initial work has already been done.

also the format is simpler than .GAME, in the sense that is plain zip (no rar, no 7z, no gz... only zip) containing a few prg and chr files, expected to have the exact checksum listed in the provided xml database... and nothing else.

also it does not require tons of changes to support it (except for the xml parser, I know): you load the separate files one after the other and then you can treat it like a iNES file

MESS isn't like MAME in that it can dictate formats as very few people actually use it. Without proper consultation of developers it's pretty much going to be like running on a treadmill any time you waste on such an endeavor.

I'm not trying to dictate anything and I would really like to know where you did take this impression from...
I was unsatisfied with iNES and displeased that neither UNIF nor iNES2.0 were getting success. I looked for inspiration in NEStopia (with its emulation based on the board types and its internal xml db) and in bootgod's website, and I pushed their approach to the extreme consequences because it matched my needs, my background and MESS capabilities.

Once the list was ready, I thought the result could be of interested for other developers as well. I also added the controller and dipswitch settings, that MESS cannot even use at the moment, because they could be of use for some other (better) emulator. that's all.

I don't expect iNES to be dropped at all (and indeed MESS will always keep supporting it). hence if you don't like xml, feel free to ignore it.

[etabeta]

Because a homebrew game might be released early and often, a database (or section) designed for homebrew should probably not list outdated dumps. For example, the entry for LJ65 on pdroms.de lists only the latest version (0.41). But common ROM cataloging tools are designed for commercial games, and far more prototypes can be found for a homebrew game than for a commercial game. The database might end up with entries for versions 0.01, 0.02, 0.03, 0.04, ..., 0.45, 0.46 of a program, as if they were alternate dumps.

nope: outdated entries could be removed from the database, of course

It's free as in speech, but not free as in memory. Some NES emulators run on platforms with 4 MB of RAM or even less: see nesDS and PocketNES.

touche'. but I'm not really an expert about 'handheld' NES emu, so I'm going to do some research before suggesting anything about it.

Apart from "arcade consoles" such as PlayChoice, CPS, and Neo Geo, a given arcade system board supports only one or a few games, and they have far less of a homebrew scene than consoles. (There's Vantris, a falling block game for Vanguard hardware, and what else?) This makes adding a game less cumbersome, as the emulator developer has to add support to the emulator's code anyway.

the same holds true if you replace "arcade board" with "cart pcb/mapper". there are mappers which only support 1 or 2 games and "the emulator developer has to add support to the emulator's code anyway"

I see 34 and think BNROM. The wiki page for 34 claims that BNROM and NINA-001 are easy to distinguish: NINA-001 has CHR ROM.

ok, I took the wrong example. but it only means that instead of checking for the submapper but in iNES 2.0 I need to add to my emu

if CHRROM -> use NINA
else -> use BNROM

until we find an obscure NINA game which has only PRGROM and only writes to 0x7ffd

[RetroRalph]

all these aspects could be handled by using a list of crc in the emulator source, but it's much more transparent when externalized.

Agreed, I don't think INES is the best solution, but what you have offered is merely a minor improvement.

also, it's easier when a new cart gets dumped: say you put in the source the dipswicthes for a 65-in-1 cart and then a 77-in-1 different cart get dumped with the same cart hw but different switch settings: if you use an internal crc list you have to release a new version of the emu with the new crc; in this way you just have to add the xml entry and the emulator would read the enw settings without even the need of recompiling it...

Well this is taking the simple case isn't it? A "major" finding would still require new information to be added and every emulator to be recompiled. Simple things can already be handled with INES or pretty much any decent format. XML isn't some magic bullet, and it requires decent amounts of code to support (RetroCopy already supports it so it's not an issue from my perspective).

there are continuously new findings about these games, despite their age: as such, we could expect some more extension to be needed in the future (even if for smaller and smaller tidbits)

I'm not advising a 100% closed solution is possible at this point, however it seems to me most of what you have done is automation with a few tweaks on top. You can't take your few hours? of work and expect it to matter much when hundreds, likely thousands of hours is required.

You are merely fixing small issues with INES, you aren't helping to cover all cases needed by emulators going forward. Unless you have a complete solution that fixes all issues and improves on nearly every aspect of INES I can't see how you can champion a minor improvement that no emulator supports. For something that RetroCopy requires I don't see any real benefits for me to convert, and that to me is what matters. Until you cover all the cases that *I* need, which is a rather advanced setting, you won't have a good replacement for INES.

That said, I welcome any improvements you can do because my helpers will be taking everything they can when they add NES support to .GAME Or alternatively if you do want to get serious about it I welcome a complete solution, I'd prefer not to be the one that does all the work (or manages it rather).

[etabeta]

Agreed, I don't think INES is the best solution, but what you have offered is merely a minor improvement.

yet it is exactly the kind of minor improvements which has lead in the past to the proposal of UNIF and iNES2.0 formats...

what would you suggest as a major improvement?

Well this is taking the simple case isn't it? A "major" finding would still require new information to be added and every emulator to be recompiled.

of course I took a simple case. any 'revolutionary' new finding will always require major changes in the emulators and no well studied format could take care of the emulation issues for you

still, no matter how easy it might seems to you, iNES could not handle these easy cases as transparently as an external xml list, nor UNIF could (except if a new [DIPS] chunk gets added to the format)

You can't take your few hours? of work and expect it to matter much when hundreds, likely thousands of hours is required.

leaving aside the fact that
* I have spent 6 months on this before disclosing it yesterday (and you can also publicly review in the MESS svn the past 2 months of work on the emulator side, after the long time spent by myself and judge and micko on the xml side),
* I have had several discussions with bootgod about both the xml and several tiny pcb details since last January
* I have tested briefly *ALL* 3625 images to check obvious mis-assigned boards and/or mirroring settings

you should not forget the *years* which have been spent by other emu authors on matching mappers number to pcb boards...
something which resulted in the wonderful "board" files you can see in NEStopia source (namely src/core/board/NstBoard.c & .h) and which I was able to use the founding stone when I decided to build up my list

You are merely fixing small issues with INES, you aren't helping to cover all cases needed by emulators going forward. [snip]..[snip] Until you cover all the cases that *I* need, which is a rather advanced setting

you can consider them as small issues, but they cover most complaints other authors reported over the years.

please be so kind to give me some examples of *all* the cases you think you will encounter and I will happily take a look at them... until then, I still think a 1:1 representation of the cart hardware (as the separate files + xml <fearure> fields allow) is way more effective than .GAME

[RetroRalph]

yet it is exactly the kind of minor improvements which has lead in the past to the proposal of UNIF and iNES2.0 formats...

Both formats failed to gain any traction.

what would you suggest as a major improvement?

Standardized advanced compression (7ZIP), proper integrity checking, proper naming schemes including a global naming system for linking different systems. Multiple name to rom linking. Ease of use, to support .GAME requires almost no code for the emulator itself and is more simple than an XML parser to add. Details of game such as input devices, number of players, ports used, slots used, etc. .GAME has all these for supported systems. In essence, everything you need to know about a game for an emulator is there. You can argue "it's XML, we can add them", but the point is it's not there now is it. Comparing what you've done to say UNIF is like comparing FireFox 3.5 to 3.6.

still, no matter how easy it might seems to you, iNES could not handle these easy cases as transparently as an external xml list, nor UNIF could (except if a new [DIPS] chunk gets added to the format)

That isn't really the point I think. Is it easier to add a few CRC checks for the minority of games or add a complete new ROM format that does it? The answer is simple. Until you have some side benefits, all the things I mentioned for instance, who is going to convert to it? What is the reward for doing so? The chance a few undiscovered pirate boards may work without recompiling? The line will be long.

leaving aside the fact that
* I have spent 6 months on this before disclosing it yesterday (and you can also publicly review in the MESS svn the past 2 months of work on the emulator side, after the long time spent by myself and judge and micko on the xml side),
* I have had several discussions with bootgod about both the xml and several tiny pcb details since last January
* I have tested briefly *ALL* 3625 images to check obvious mis-assigned boards and/or mirroring settings

you should not forget the *years* which have been spent by other emu authors on matching mappers number to pcb boards...
something which resulted in the wonderful "board" files you can see in NEStopia source (namely src/core/board/NstBoard.c & .h) and which I was able to use the founding stone when I decided to build up my list

I am very thankful of those that have spent their time trying to improve the NES scene in any way possible, even your attempt here. I welcome change.

However if you spent 6 months coming up with what you did you haven't worked optimally in my opinion. In reality whatever work you've done is going to be mostly wasted because what you offer isn't a complete solution for emulators going forward. REsolving the problems of the 1990s isn't going to help those of us in 2010.

[etabeta]

Standardized advanced compression (7ZIP), proper integrity checking, proper naming schemes including a global naming system for linking different systems. Multiple name to rom linking. Ease of use, to support .GAME requires almost no code for the emulator itself and is more simple than an XML parser to add. Details of game such as input devices, number of players, ports used, slots used, etc. .GAME has all these for supported systems. In essence, everything you need to know about a game for an emulator is there. You can argue "it's XML, we can add them", but the point is it's not there now is it. Comparing what you've done to say UNIF is like comparing FireFox 3.5 to 3.6.

Most of the things you listed would probably belong more to a frontend than to an emulator, imho, and the one who don't (like the controller devices) are already listed above.
As such, personally I'm not interested to include the things you mentioned directly into the emulation.

That isn't really the point I think. Is it easier to add a few CRC checks for the minority of games or add a complete new ROM format that does it? The answer is simple. Until you have some side benefits, all the things I mentioned for instance, who is going to convert to it? What is the reward for doing so? The chance a few undiscovered pirate boards may work without recompiling? The line will be long.

didn't I say more than once that the multicart dump was just a stupid example which came easily to my mind? rather than list other small advantage I will repeat my (few) points

1. imho the list of crc is a gross hack and defeats from start the usage of headered files (call it NES, UNIF or GAME)
2. simplifying (as Marty first did, and as I was able to do as well) the emulation of Konami Boards is more than enough as a side effect for me.
3. accurately represent and describe the content of the carts is an added value, and cannot harm the documentation side of emulation

feel free to disagree. as I said I'm not forcing anyone to use the format I added to MESS.

However if you spent 6 months coming up with what you did you haven't worked optimally in my opinion. In reality whatever work you've done is going to be mostly wasted because what you offer isn't a complete solution for emulators going forward. REsolving the problems of the 1990s isn't going to help those of us in 2010.

I had a goal in mind and I reached it with the easier solution I could think of (a solution which makes wonder for ~9000 games in MAME).
this same solution would also allow people with the right instruments like bootgod to repair broken carts. is this enough for me? yes, in my eyes it is.

That said, I'm not interested in converting anyone to see separate PRG & CHR as a revolution.
They just represent an alternative approach to iNES. In particular, an approach capable to address most past complaints and all my needs in MESS. And an approach publicly available to anyone who wants to use it.

If no one wants to use it, ok, I'm still happy with the result I obtained for the reasons I listed above. In the end it's just an hobby

[blargg]

In other words, this is research/an experiment in another way of representing NES cartridges digitally. The results of this will help anyone in the future considering a similar approach. Maybe it won't pan out for some/all uses, but it will provide actual experience.