Sup NesDev, iNES is dead, how do you feel?

[dvdmth]

A database is useless for homebrew, translations, and hacks where the checksum will be changed. There must be a way for the end user to set the hardware instantly. No reason a prg/chr checksum database couldn't be used to FIX the bad headers of existing files tho.

Amen! I've been waiting forever for SOMEONE to write a database-driven ROM header fixer. The Mac version of Nestopia doesn't support soft IPS patching, and I HATE it when I apply a translation/hack to a ROM and watch it blow up because it has a bad header and Nestopia can no longer resolve it. A couple of years ago I wrote a small, hackish program that output to a text file the NstDatabase contents for a given .nes file, so that I could fix the header with a hex editor based on Nestopia's treatment of the ROM. All well and good, until Nestopia was updated and some fields in the database file were changed, causing my little program to fail (ugh!!).

The next version of Nestopia will have its database converted to the new XML format. I love this aspect of the XML proposal, because the format is more standardized and more easily used by other software, such as a ROM fixer. I don't care much for the zipfile container approach, but the XML database feature is very appealing to me. If my programming skills weren't so bad (it takes me forever to do a hackish job at something), I'd write a ROM fixing utility as soon as the new Nestopia version was out. I sincerely hope that someone - ANYONE - out there will FINALLY get a decent ROM tool developed for the NES!

[/rant]

[BootGod]

Just FYI, my dumping client does have the ability to fix ROM images, but it will only work with images known by the DB. Although, because it's doing everything online, it's not very practical to do batch operations.

I am planning on making a standalone utility that can load up the XML either from Nestopia or one generated by the cart database in order to easily and efficiently repair and convert between formats.

I'm not sure where this talk of replacing iNES came from, I can't speak for Marty, but I don't know that this format is aimed at replacing anything. My client and upcoming utility will be able to convert images into whatever format you want, your not going to be forced to convert to the XML based one.

Just a few comments on some points brought up:

Writing an XML parser is a pain. It certainly is, but luckily there are dozens of free libs out there that you can drop right in. I found a nice lightweight one, incorporated into a single class that allows to to easily read,create,modify,etc XML files.

I also thought INES2 was (and still is) a great idea. If anyone is ever up to completing and documenting it (mainly the sub mapper stuff), I'll be happy to add support for that as well. It's not as flexible as the XML format, but simply the fact that it is backwards compatible makes it a huge advantage.

Viewing ROMs in a hex editor is much easier to follow with this XML format since the ROMs are stored separately and individually, so the offsets don't get messed up.

As far as IPS patching goes, one would think soft-patching would usually be the preferred way to go about it and Nestopia will still be able to use patches in this manner. However, there certainly are cases where you'd want to do hard patching as well. I could add this ability to the ROM utility.

The flexibility of the XML format is quite an advantage and you can really define a lot of detail about the hardware if you want too, even though it's not always necessary.

It already does define some pin information in cases where it is necessary. For instance to distinguish between CNROM types (m3 & m185) and X1-005 (m80 & m207). It also defines things like sound samples for games that have a built-in chip like the D7756C. I plan on doing this for some other boards as well, especially to deal with all the Konami VRC variants.

Another thing I like about the new format is the ability to have ROM "sets", as in one zip can contain all the data for a game and any revisions or regional versions. Besides being more organized, it also has the bonus of being able to share ROMs, as quite often it is only the PRG ROM that changes.

This is also the main reason my client doesn't yet build images in this format though as my db doesn't currently have any link information between regions. A lot of this data will be able to be generated automatically based on catalog ID's and shared ROMs, but there is still plenty that will have to be done by hand.

[bunnyboy]

I'm not sure where this talk of replacing iNES came from, I can't speak for Marty, but I don't know that this format is aimed at replacing anything. My client and upcoming utility will be able to convert images into whatever format you want, your not going to be forced to convert to the XML based one.

Looks like it mostly came from X-or, and some of the people on the bannister board. I bet there would be a lot less controversy if it had just been presented as a way to add additional files to the current ines, instead of a complete replacement. Even Marty in one post said the .nes file should be left alone for backwards compatibility. Then there are people like R Belmont that say nesdev doesn't care about hardware, when the hardware info found here is more accurate than Nestopia...

Writing an XML parser is a pain. It certainly is, but luckily there are dozens of free libs out there that you can drop right in. I found a nice lightweight one, incorporated into a single class that allows to to easily read,create,modify,etc XML files.

Feel like writing one in 6502 for a device like the PowerPak? Handling all the different versions of zip compression wouldn't be fun either...

Viewing ROMs in a hex editor is much easier to follow with this XML format since the ROMs are stored separately and individually, so the offsets don't get messed up.

Viewing is easier (assuming you have already replaced ines, which may or may not be the goal) but editing is certainly not. Once you make a change, your CRC/MD5/SHA/etc based database is broken. Of course that may be the idea, to make sure edited roms aren't treated as the real ones. What would a checksum do if its not for determining hardware?

Another thing I like about the new format is the ability to have ROM "sets", as in one zip can contain all the data for a game and any revisions or regional versions. Besides being more organized, it also has the bonus of being able to share ROMs, as quite often it is only the PRG ROM that changes.

Other than the sharing of duplicate data, making sets can already be done. Would the duplicates be removed by zip anyways? Its just a normal zip file so you can toss in whatever you want, including manuals and boxes for preservation. ines in no way restricts what other files you bundle with it in a zip. This new format should in no way restrict the use of ines files instead of separate prg/chr files, since that would just be a simple entry in the xml file. With the tiny size of nes roms you could even have the nes and prg/chr files in the zip. Artwork will be FAR larger than many copies of the rom anyways.

[BootGod]

Writing an XML parser is a pain. It certainly is, but luckily there are dozens of free libs out there that you can drop right in. I found a nice lightweight one, incorporated into a single class that allows to to easily read,create,modify,etc XML files.

Feel like writing one in 6502 for a device like the PowerPak? Handling all the different versions of zip compression wouldn't be fun either...

Yeah you've got a point there, the PowerPak is definitely one situation where the XML format is not suitable at all. One could always use the conversion utility to write INES/INES2 files right to the memory card though.

Viewing ROMs in a hex editor is much easier to follow with this XML format since the ROMs are stored separately and individually, so the offsets don't get messed up.

Viewing is easier (assuming you have already replaced ines, which may or may not be the goal) but editing is certainly not. Once you make a change, your CRC/MD5/SHA/etc based database is broken. Of course that may be the idea, to make sure edited roms aren't treated as the real ones. What would a checksum do if its not for determining hardware?

In the ROM format aspect, the checksum doesn't really matter at all. Files inside the zip are defined in the XML by the ROM filenames, not by the checksum. So any patched ROMs will still load just fine in the format.

Another thing I like about the new format is the ability to have ROM "sets", as in one zip can contain all the data for a game and any revisions or regional versions. Besides being more organized, it also has the bonus of being able to share ROMs, as quite often it is only the PRG ROM that changes.

Other than the sharing of duplicate data, making sets can already be done. Would the duplicates be removed by zip anyways? Its just a normal zip file so you can toss in whatever you want, including manuals and boxes for preservation. ines in no way restricts what other files you bundle with it in a zip. This new format should in no way restrict the use of ines files instead of separate prg/chr files, since that would just be a simple entry in the xml file. With the tiny size of nes roms you could even have the nes and prg/chr files in the zip. Artwork will be FAR larger than many copies of the rom anyways.

The zip format doesn't do solid compression (each file is compressed individually) so no, zipping would not have the same effect as removing the duplicate files.

[tepples]

Writing an XML parser is a pain. It certainly is, but luckily there are dozens of free libs out there that you can drop right in.

Feel like writing one in 6502 for a device like the PowerPak?

Remember the Contiki web browser? It handles HTML, which is related to XML. So I know the NES is powerful enough to handle at least a useful subset of XML, even if not a full-blown DOM or SAX implementation. But I don't know how much RAM needs to be made writable for it to work.

Handling all the different versions of zip compression wouldn't be fun either...

Do popular PKZIP-compatible archivers ever use any of the PKZIP 1.x codecs? But the typical 32 KiB ring buffer size of Deflate might be the biggest pain on a machine with a small address space.

Would the duplicates be removed by zip anyways?

No. The PKZIP format, unlike some other archive formats, does not use solid archiving. A non-solid archive makes it easier for the client to decompress individual files, as the files don't depend on files earlier in the archive, but it doesn't exploit file-to-file redundancies. The tar.gz format uses solid archiving, but it still has Deflate's 32 KiB window, which is smaller than everything but the earliest NROM games. The only widely-used Free archive format I know of that has solid archiving and a window size on the same order of magnitude as an NES ROM is .7z used by 7-Zip.

[Dwedit]

Zip Store, then Zip maximum compression to get solid archiving with zip.
How is interpreting XML any different than a basic interpreter, like the one from the Apple ][?

[tepples]

Zip Store, then Zip maximum compression to get solid archiving with zip.

That would have the same drawbacks as tar.gz: the 32 KiB window limit.

How is interpreting XML any different than a basic interpreter, like the one from the Apple ][?

Differences include the following:

1. Applesoft BASIC used a 1-byte character encoding based on ASCII, while XML uses UTF-8 or UTF-16 that can address over a million unique code points.
2. Applesoft BASIC's parser doesn't need to load and interpret custom DTDs that define custom entities (&stuff;).
3. A program in Applesoft BASIC is a linked list of lines, each of which is a nul-terminated string of tokens and characters. An XML document can be an arbitrary tree. Processing trees directly in assembly language on an 8-bit CPU can get hairy.

We would need to define a custom subset of XML that includes what we need and excludes what we don't. At that point, we might as well just use INI format.

[Hyde]

Minor letdown. When I saw the subject, I thought it was supposed to mean Marat has quit.

[Dwedit]

String comparisons in UTF-8 are identical to string comparisons in 8-bit ASCII. No piece of a high character can ever appear in ASCII, and there is no way to find one extended character inside another. For everything except counting characters, you can ignore the fact that it's not ASCII.

[tepples]

String comparisons in UTF-8 are identical to string comparisons in 8-bit ASCII.

True, but XML files can be in UTF-16 too. And is "Mario's Time Machine" the same as "Mario's Time Machine" and "Mario's Time Machine" and "Mario's Time Machine"?

For everything except counting characters, you can ignore the fact that it's not ASCII.

Everything except counting and printing. Do we ignore the mainland Europe and French Canadian markets by printing any character in the title that's above U+007F as a '?' character?