bsnes and headers

[HJRodrigo]

Long ago, I was really huge on hating Cowering because he was including fan translations that got propagated to ROM sites. I joined the hate bandwagon with every other ROM hacker.

Interestingly enough this has a direct relationship to the issue of headers. If it were not for Cowering's work there would be many translations/hacks that I would have not have even tried, due to poor documentation. Authors of these patches often failed to specify whether the ROM images required headers or interleaving. I would just say screw it to these works, as I did not want to try multiple IPS/ROM combinations. Then I discovered that Cowering did the leg work for me, so I would just visit one of those sites that propagated these authors works and try out the games. This is of course a problem that need not occur this day an age with people adopting no intros ROM standards.

P.S. I belonged to a different type of hacker, one that viewed ROM sites and acknowledgement in goodtools as a good way of getting our crappy work out there . Only the big shots seemed bothered by having their work cataloged by goodtools. Many of us small time guys would actually submit our work for inclusion into goodtools .

[tepples]

My beef with Cowering was mostly when he misspelled my name in some GoodNES entries.

My name is Damian YERRICK. It's listed as such at the bottom of every page on my web site and on my contributor page on English Wikipedia. Yet I've seen "Yeppick" and "Terrick" on some of my works in GoodNES, possibly due to my attempts to stuff an half-uncial or insular font into 8x8 pixels. If you've wondered why I've used the same plain-jane Chicago font (see "Who's Cuter" in this sampler) for just about everything since then, now you know.

[Near]

Another point ... let me show how easy it is to have a complete collection of headerless ROMs, and still use them with your copier.

First, download and install kaijuu (http://byuu.org/programming/kaijuu)
Add a new rule: for files of type "*.sfc" execute: "ucon64 -swc {file}"; optionally make it the default.

If you set it to the default, you can double-click your files and play them on your SWC.
If you didn't, all you have to do is right-click the file, choose "Transfer to Copier" and enjoy.
No need to ever have two copies of each game.

kaijuu will also let you associate all of your SNES emulators, so you can easily play them.
But if you won't use it because I made it, you can add the action to HKEY_CLASSES_ROOT\*.sfc manually.

Linux file managers have kaijuu's functionality built-in, too. Thunar calls them custom actions, Dolphin and Nautilus have them as well.

...

So I create a two-click tool to remove all headers, a two-click tool to play games on copiers, a drop-in replacement for tools that only work with copier headers, give out my full emulator source code under an FSF/OSS license, and I'm the bad guy.

FuSoYa makes a closed source tool that only works with copier headers, because that's what he likes, and nobody has any problem with that, and he doesn't take any grief for it, and he's the good guy. Go figure.

This is of course a problem that need not occur this day an age with people adopting no intros ROM standards.

It's another point of stubbornness against change.

BPS is so vastly superior to IPS, and is in Snes9X, but nobody will use it for their works.

BPS verifies the checksum of the game before and after patching. It even verifies the patch itself is undamaged.
BPS offers metadata authorship. So you can embed your contact info, website, and even the readme itself, into the patch.
BPS is a delta format, so you can insert, move and delete data. Substantially smaller patches on non-ROM based systems.
BPS' delta encoding can also result in significantly smaller patches on certain ROM types (eg Mystic Ark is half the size this way.)
BPS has no limit on file size: not 16MB, not 4GB, not 100 Googol Exabytes. No limit, period.
Writing a BPS patch applier or linear patch creator is doable in ~20-30 lines of code; only ~30% more complex than IPS.
Writing a BPS delta patch creator is of course tough, but only one person needs to make that.
It also supports BPM, which allows you to patch entire folders. Allowing the same format to handle ROMs, CDs, PC games, etc.
The spec itself is many, many times smaller than Xdelta and bsdiff. It's possible for people to embed soft-patchers with this format.

All that's needed here is an incremental approach.
Start by allowing IPS and BPS in emulators, and offering all patches in both IPS and BPS on RHDN.
Continue by releasing new patches only in BPS format.
Eventually, start making soft-patchers offer to convert IPS to BPS.
Create compelling advantages for BPS: like emulators that can check a database when you load a game and find translations for you to download.
Next, stop offering IPS patches for download at all.
Finally, remove IPS support for emulator when its usage drops below 1%. Be willing to deal with being called a "religious crusader" for it.
Done =)

My beef with Cowering was mostly when he misspelled my name in some GoodNES entries.

Whoever indexed Der Langrisser attributed it to [D+byuu], which is unfortunate for the 20 other people who also helped out.

[HJRodrigo]

BPS seems great, but I am under the impression that the ppl at RHDN think you have abandoned it like was done with UPS. Release a minor update to beat and I will add a news update on RHDN, so everyone will know you are still backing this format. If you plan to release your newest hack with it (which I thought you mentioned would be a necessity for some reason) then this could also be mentioned in the news update as a selling point of how good the format is. Technically you could do the news update on RHDN yourself, but I believe you left the site and a log in is required to post news updates.

[Near]

We screwed up with UPS, yeah.
Focused too much on the niche feature of patch reversibility and didn't factor in its costs: substantially larger patches post-compression.
Wasn't going to stick with a bad format just to bolster public confidence.

But yes, BPS is a done deal. Extremely well vetted, tested against thousands of files. It's not going anywhere.

Newest version was posted in August, there's little to be done at the moment: http://byuu.org/programming/beat/

We just need someone who -really- understands delta encoding to help make a more effecient delta algorithm for files > 100MB.

I suppose if you guys want to push it there, great, I'm not going to. That would be like a hen in a foxhouse. May want to wait for the next Snes9X release so that a mainstream release supports it.

[Sik]

3) ZIP files are hard to handle when it comes to romhacking: if the ZIP contains several versions of the same code, you may choose the wrong ROM inside it when you open it from time to time. It's pretty more secure to unzip the required ROM and keep it as the "master source code".

Most emulators spazz out when there are multiple ROMs in a single ZIP so those ZIPs already would be unusable for starters.

4) I don't use bsnes for gaming, I use it for accuracy issues. My laptop even can't run bsnes faster than 35fps, so it's impossible to me playing games. But when it comes to test hombrew code, it is the best emulator, just like running the code on the SNES. From my point of view, bsnes is aimed to developers like me, and I don't need ZIP files or headers or anything else but 100% accuracy.

If you want 100% accuracy then ditch the emulator and use real hardware. Sorry, but I have learned that emulators shouldn't be trusted at all - even when they emulate things like cycle-level operations and such, they still won't emulate issues like the hardware crashing if you try to access it too quickly (or if they do, they won't emulate in what way this happens) and such. On systems with caches they also won't emulate the caches properly, which can cause issues depending on whether caches are flushed or not (or which parts of the cache). Also different revisions of the hardware have different issues and emulators rarely will emulate them.

Emulators are good to debug game logic - and to be fair, this is what you're going to do most of the time. They are also good to see if you're uploading the right data to memory before you can see anything. But if you need to see if some new code dealing with hardware is working, you have no option but to use real hardware to be sure (and even then you have to be careful - if you're using a flashcart that has a firmware instead of booting directly chances are the hardware initialization done by the firmware will later mess up debugging, I already know of one flashcart firmware that likes to clear RAM upon boot).

But the ROM hackers, holy shit do they ever hate change. Most are still using ZSNES, IPS, xkas v06, copier headers ...

Reminds me of how many Sonic hacks were still being made for Gens no matter what (not to mention how almost every single hack seems to have an address error somewhere). These days that kind of dropped out thanks to the EverDrive making it easy to use flashcarts with modern systems (so people want to play hacks on real hardware), but yeah.

(says the guy who still sticks to asm68k, but honestly this has more to do with about every single assembler having its own variant of the syntax making things completely incompatible with each other unless you want to drop the most useful features)

My beef with Cowering was mostly when he misspelled my name in some GoodNES entries.

LOL my biggest issue is how about every homebrew game seems to get labeled public domain -_-'

EDIT: also why is it that on the Mega Drive end headerless ROMs were the standard (with the only copier ever supported being Super Magic Drive, and only back when it was still somewhat popular) but this isn't the case with the SNES? Heck, Mega Drive ROMs even have .bin as their standard extension rather than a console-specific one (there's .gen, but nobody uses it).

[Near]

On systems with caches they also won't emulate the caches properly

I will agree that hardware is always preferred where possible. But bsnes is good enough for prototyping that you don't have to run every small change on hardware.

For cache, take the SuperFX. I emulate the 256-byte instruction cache, and the way it's setup as 16x16-byte stripes that fill any time a byte in that block is fetched. I emulate its ROM buffer cache, its RAM buffer cache, and its primary -and- secondary pixel buffer caches. And all that for a chip used by ten games.

No, it's not perfect. But it's an order of magnitude more precise than any other emulator. I'm really serious about getting things as close to perfect as possible.

Also different revisions of the hardware have different issues and emulators rarely will emulate them.

I do emulate the differences found between the CPU revisions 1 and 2, with exception of the DMA crash. I handle the different pattern to DRAM refresh, to HDMA init timing, etc. These things are -really- hard to find, it's needle in a haystack stuff, so certainly I've probably missed some things.

(says the guy who still sticks to asm68k, but honestly this has more to do with about every single assembler having its own variant of the syntax making things completely incompatible with each other unless you want to drop the most useful features)

A nice part of mine is a table-based syntax. So you can use any opcode syntax you want, you just have to write the table.
Nobody's done a 680x0 table yet, though.

EDIT: also why is it that on the Mega Drive end headerless ROMs were the standard (with the only copier ever supported being Super Magic Drive, and only back when it was still somewhat popular) but this isn't the case with the SNES?

Well, at least interleaving mostly didn't take off because only ZSNES tried to support it initially. Thankfully Snes9X was popular enough that their not doing so lead to ROMs being deinterleaved widescale.

The SMD file format is pretty annoying, but at least it's easily detectable thanks to that forced "SEGA" text.

I really admire the Genesis memory map, though. ROM starts from 0 and goes upward.

Here is SNES mapping, by comparison: http://byuu.org/snes/database/

There's about 50 boards that all do things slightly differently. The ROM header does not give you enough info to tell which board it uses.

[HJRodrigo]

I suppose if you guys want to push it there, great, I'm not going to. That would be like a hen in a foxhouse. May want to wait for the next Snes9X release so that a mainstream release supports it.

I considered waiting, but then I realized Snes9x may never get a new release (like ZSNES). My fiancée has an account in her nickname over there, so I just went ahead and got it submitted (It is currently pending acceptance). One day, I will make an account for myself .

I had to paraphrase/shorten some of the information you had written on your site and I combined it with some of what you wrote here for the sake of easy legibility and since I am no editor (or a native speaker) you should forgive any errors I may have committed in the process... You know what they say, "If you want something done right do it yourself" .

[tepples]

LOL my biggest issue is how about every homebrew game seems to get labeled public domain -_-'

When I wrote my interpretation of the GoodCodes for Pocket Heaven, I didn't mention public domain at all in my explanation of "(PD)". Instead, I wrote "(PD): Internet release authorized for free redistribution".

Heck, Mega Drive ROMs even have .bin as their standard extension rather than a console-specific one (there's .gen, but nobody uses it).

Early on, GBA also used .bin, but it fairly quickly switched to .gba for programs designed to run in ROM or .mb for programs designed to run in RAM. I prefer .gen for MD because it's distinct from any Atari product. At one time, I planned to make a launcher for .bin that would peek into the file, detect whether it's Atari 2600, Genesis, or GBA, and launch the appropriate emulator, like the .py launcher for Windows introduced in Python 3.3.

A nice part of mine is a table-based syntax. So you can use any opcode syntax you want, you just have to write the table.
Nobody's done a 680x0 table yet, though.

My sick vision of a 68000 table includes letter names for registers (A-H for D0-D7 and Z-S for A0-A7) and 6502-style load/store instructions as aliases for MOVEs in and out of registers. Imagine LDA.B, LDX.W, and STY.L. Then imagine 65816-style REP and SEP, but instead of emitting an instruction, they switch the default size between .B and .W for subsequent MOVEs without a dotted size in the same scope. Either that or ape the syntax of Thumb.

There's about 50 boards that all do things slightly differently. The ROM header does not give you enough info to tell which board it uses.

Then perhaps it's time to design our own copier header: jSNS. It'd be like iSNS but one better, as it would encode the missing info as a JSON object stored in a separate file: Super Mario World.sfc and Super Mario World.jsns.

[Near]

My sick vision of a 68000 table includes letter names for registers

Heh. I created a 6502 mnemonic set for the SPC700. You could tell 90% of the opcodes were "inspired" by 6502 anyway. They probably just redid it all last minute for legal reasons. mov x,#$00? Come on. ldx #$00.

nocash remakes every instruction set into x86 for some reason.

So there's definitely precedent if you want to make your own. Unlikely to catch on, but we usually do these things for our own benefit.

Then perhaps it's time to design our own copier header: jSNS. It'd be like iSNS but one better, as it would encode the missing info as a JSON object stored in a separate file: Super Mario World.sfc and Super Mario World.jsns.

That was part of the joke, yeah. For all the good iNES does, you could just make a 2-byte mapper ID and uniquely specify every tiny variant of every board out there. The same is true for the SNES. Drop the maker ID (SHVC-, MJSC-, MAXI-, EA-) as irrelevant, and then store the rest as the header. "1A3B-20" ... and an internal DB of the 50 boards. That's all you need for 100% perfect mapping.

My vision though is to leave the binary files as pure ROM contents, no user-supplied stuff that is due to be argued about for generations like iNES 2.0 and UNIF are, no compression archives, no existing containers like ZIP/TAR with all the stuff that doesn't matter to emulators. That gives emulator authors two choices.

Option 1: keep a database of SHA256={mapping layout}.
I'm making that database now at byuu.org/snes/database

Option 2: provide a manifest file that describes the PCB.
Just like your .jsns, I support this as well. Only instead of JSON, I went with something far simpler to parse. I call it BML.

Basically it's indent-level sensitive ala Python. You can do attributes like XML, but internally they're child nodes just like anything else.
You can assign with name=data, or name="data with spaces", or name:data with spaces and "quotes"
= runs to the next space
="..." runs to the second "
: runs to the next line feed (if it's a child node, it appends multiple lines)
And that's it. No comments for the same reason as JSON. No escape character/entities to decode. No Billion Laughs by entity expansions (XML) or linked nodes (YAML). Can be decoded with no memory allocation other than the child nodes (inserting zeroes into the loaded document, nothing ever expands.)

Here's an example:

Code: Select all

cartridge region=NTSC
  board type=2J5M revision=01
    rom name=program.rom size=0x180000
    ram name=save.ram size=0x8000
    map id=rom address=00-3f,80-bf:8000-ffff
    map id=rom address=40-7f,c0-ff:0000-ffff
    map id=ram address=10-1f,90-9f:6000-7fff
    map id=ram address=30-3f,b0-bf:6000-7fff
  information
    title:    Romance of the Three Kingdoms III: Dragon of Destiny
    name:     Romance of the Three Kingdoms III - Dragon of Destiny
    region:   NA
    revision: 1.0
    board:    SHVC-2J5M-01
    sha256:   88fa0a78ca98c6386e086c7fa9e81a2625bdecc55115dde6c6f4770b0670aa40
    description
      :This is a strategy game by Koei.
      :It's very confusing to try and play.
    configuration
      rom name=program.rom size=0x180000
      ram name=save.ram size=0x8000

Only cartridge/board is needed. cartridge/information is optional, and cartridge/information/configuration tells you how to split a ROM file into individual chunks. For the NES, it'd help you split PRG and CHR. For the SNES, it helps you split program from data ROMs, and coprocessor firmware ROMs.

Emulators would be free to support whatever they wanted in cartridge/information, since it's extensible.

Aside: title is for the native language, so it'd be Japanese letters if from Japan. name is always using Roman letters (A-Za-z), and avoids NTFS restricted characters like : and \, and is sort friendly ("Brainies, The" and not "The Brainies" like the title.)

So my long-term vision ...

Most people will just have their folder full of .sfc files. They will pick games and play them. I find them in my database of verified software, and load the mapping from there.

ROM hacks will be distributed as BPS patches, that have the manifest.bml embedded in the patch. The emulator will apply the patch, pull the manifest out, and use that mapping (some games change mapping rules.)

Users can also distribute ZIP containers that have SFC+BML files, and it'll use that mapping.

Developers can have folders with game.sfc+game.bml files, and it'll use that mapping.

Behind the scenes, I make game folders. It's a folder named game.sfc/, and inside you have all the files related to it. program.rom, save.ram, bsnes/state-1.bst, cheats.bml, box/front.png, etc.

The emulator cores only load game folders. The frontend makes them before sending them to the core, completely transparently to the user.

For people who -really- agree with my vision, they can use game folders directly as well. They're my favorite approach. No compression, no merged files at all. Anything emulator-specific in its own subfolder. A folder is equivalent to a real game. When you move it, you move everything related to that game with you. There is no need for the emulator to have separate path configurations for every file type.

[Sik]

Well, regarding iNES, I think the problem is that with NES games you can't do away with headers because there are multiple ROMs in the cartridge (two, more specifically) and then you have a myriad of mappers and no header in the ROM at all to retrieve them. A header providing all this information is pretty much a must (or you could take the MAME or ZOMG approach and make a ZIP file with multiple files inside, with the filename telling what each file is meant to contain).

Also this discussion reminds me about when once on IRC we were discussing about Chinese bootleg Mega Drive games and how every one of them seem to include copy protection. Somebody joked about that at this rate we'd need to include headers in the ROMs (since sooner or later it'll become impossible to distinguish between two schemes just going by how the program behaves). The problem is that of course this information is never present in the ROMs (in fact most of the time they don't even have valid data, period). An even worse problem is that the schemes seem to be unique to each game most of the time, so we wouldn't be talking about specifying which mapper to use in the header, but outright storing how to emulate them if we ever pretend to keep sanity around.

A nice part of mine is a table-based syntax. So you can use any opcode syntax you want, you just have to write the table.
Nobody's done a 680x0 table yet, though.

But how far can it reach? Because some things get really tricky. For example, each assembler has its own way to do local variables, and it isn't just different symbols but outright different semantics (e.g. just a local named label, a fancy named label that is actually part of the global scope, forward/backward nameless labels, etc.). Also there's how strings are parsed (e.g. using " and ' for any string vs. using " for integers and ' for dc.b text). Then there are assembler-specific features (e.g. asm68k has the filesize function, as well as the concept of sections though I never figured out how they actually work...). Also operator precedence (e.g. asm68k for some stupid reason will put binary operations above arithmetic operations - I just use redundant parenthesis to keep myself sane >_>). Etc.

Though even then it may be still useful, where is this tool? Maybe I could just go make a table and use that.

I really admire the Genesis memory map, though. ROM starts from 0 and goes upward.

Yeah, one thing about Sega's hardware is that it's a lot saner to program for in comparison to Nintendo stuff >_> That makes hardware abuse on the former much easier than on the latter.

[blargg]

Well, regarding iNES, I think the problem is that with NES games you can't do away with headers because there are multiple ROMs in the cartridge (two, more specifically) and then you have a myriad of mappers and no header in the ROM at all to retrieve them. A header providing all this information is pretty much a must [...]

Why wouldn't the same checksum approach work? Checksum the PRG+CHR file and use your database to get the mapper, CHR and PRG sizes, etc.

[tepples]

For people who -really- agree with my vision, they can use game folders directly as well. They're my favorite approach. No compression, no merged files at all. Anything emulator-specific in its own subfolder. A folder is equivalent to a real game.

In fact, this concept of an application bundle is so good that Apple uses it for applications on Mac OS X. The problem then becomes one of how to change the default intent when such a folder is double-clicked to "Open this game in an emulator" instead of "Open this folder in the file manager".

or you could take the MAME or ZOMG approach and make a ZIP file with multiple files inside, with the filename telling what each file is meant to contain

WSZ (Winamp skins), SMZIP (StepMania songs), JAR (Java applications), and APK (Android applications) also use the pattern of a bunch of files in a Zip archive. And when unzipped into a new folder, this becomes byuu's application bundle concept.

Think of it this way: Ultimately, iNES is an archive format analogous to tar.

Yeah, one thing about Sega's hardware is that it's a lot saner to program for in comparison to Nintendo stuff

I read that the N64 was a pain to program too. But after that, Nintendo started designing things more sanely. The Game Boy Advance, for example, is wonderfully more orthogonal than the Super NES, even though it is missing priority per tile, subtractive blending, and hardware audio resampling and mixing.

Why wouldn't the same checksum approach work?

Emulator authors would need to be notified of the checksum for each of the NES test ROMs that you have written, for one thing. A database of checksums is fine for the ROM images of the cartridges released during the Super NES's original commercial era, but not new productions in the homebrew era. Homebrew is why hacks might provide a new .bml, or why I eventually convinced the ZapFC guy to add an optional board descriptor file alongside the PRG and CHR files in fc.zip archives.

[blargg]

Why wouldn't the same checksum approach work?

Emulator authors would need to be notified of the checksum for each of the NES test ROMs that you have written, for one thing.

These issues apply to the SNES version of this too. Sik raised the objection that this wouldn't work on the NES because of multiple ROMs, and my point was that multiple merged ROMs in the file don't make it any less feasible for the NES than for the SNES. The same problems occur as on the SNES, with cartridges outside the commercially-released set from the console's life needing additional files with mapper information.

[Sik]

His point stands though, you're hardcoding the data into the emulators and whenever a new ROM comes out (could be homebrew or just a new dump, for all we know - there are still rare revisions of games out there that didn't get dumped) you need to update the emulator. Besides we're talking about keeping track of a database the emulator has no reason to know about - it should be part of the dump itself, not the emulator.

As for NES vs. SNES, with the latter you can still make a game that's just the ROM and that's it, with the former you have no option even for the simplest games, so the concept of raw binaries can't be used at all in that case regardless of situation.

EDIT: which reminds me, NES dumps not only are two ROMs, but also the direction in which the tilemap is mirrored. Fun stuff.