nesdev.org

The iSNS thread was a joke about rushing through a universal standard without fully understanding the problem domain first.
(UPS was a personal example of my own similar failure, in fact.)

When I listed the ROM and RAM multipliers as increments larger than the smallest examples, it was a jab at the issue with Galaxian.
Separating the RAM size into two fields mirrored the issue with "16 mappers should be enough for everybody."
Leaving dead space was just asking for "DiskDude!" and other people to continue extending the spec.

But to be fair, although I am against including mapping information in with ROM data directly (SNES mapping is -really- nuanced, it's unlikely anyone would agree on a format to use); I would prefer it over having no precise mapping information at all. That also won't happen because it requires changing existing ROMs.

So NES is stuck with (inadequate) headers because we can't change files, and SNES is stuck with (useless) headers because we can't change files, and the GBA is stuck with no headers because we can't change files.

Doesn't it get tiresome that we can never make things better for fear of change?

Ironically, we might be able to subvert copier headers to add mapping info. But that too will probably get people complaining.

I learned a long time ago that nothing you do will go without criticism. Sometimes you just have to ignore people.

I'm surprised nobody has mentioned yet that IPS has several variants that are incompatible with each other, and not always possible to distinguish other than for producing erroneous output. This just makes things even worse and even more of a reason to drop that format =P

tepples wrote:In fact, indexed BMP might even be as efficient over the wire as GIF due to HTTP Content-encoding: gzip.

Actually, 4-bit and 8-bit BMPs are compressed. I'm not making this up, BMP does indeed support compression (some form of RLE).

I'm just offering my bigger take on things because it's an interesting exercise and I think it's the only way to shed light on the topic. I encourage anyone not interested to just skip the rest.

I'm also highly, HIGHLY amused by how such an insignificant-seeming thing, 512 essentially padding bytes (no useful information for the main case we're talking about, just a nuisance to skip) at the beginning of the file, are able to weave a web between so many things at so many levels. I love it.

byuu wrote:This is not change for the sake of change. I showed you 19 different things that would be improved if they were gone. As minor or trivial as you take them to be, they're big deals for myself and many others.

The issue isn't whether we can snap our fingers and suddenly have no headers on any SNES ROMs on everyone's machines, all IPS patches everywhere made for headerless ROMs, all emulators and tools supporting headerless ROMs, and everyone's play-on-copier flow made to add headers appropriately. If we could, I think most people would agree that this power would allow us to fix countless problems with file formats, bad dumps, etc.

The argument here is about what we can do, and which of those things are respectful of users' unique setups and the larger ecosystem. It goes beyond technicalities to the intent of people choosing particular ways of handling things. If the intent is to get people to do something that one considers superior, end of discussion, then it won't be received well. This always shows up in the product; it has to, as the technical approach is chosen to implement this intent. This kind of thinking lacks humility and necessarily disregards that reality might differ or others might have different ideas and like to explore them. Yes, one isn't forced to use a piece of software, but there is still an element of control in being given the choice to be constrained or use something else.

Unless I'm misunderstanding, the code to handle skipping an optional 512-byte header from a padded ROM is trivial. Maintaining complex code one feels is unnecessary is clearly a burden, but if it's just a few lines and doesn't require revisiting, it's not a technical burden. If you supported properly-padded ROMs with a 512-byte header (that is, acted as if the file on disk had that removed), would that handle most of the headered-ROMs out there? I'm not asking about IPS patches, mind you, just the ROMs. If supporting this knocks out the need to remove headers on most ROMs, then you've eliminated a lot of unnecessary modifications of files all over the planet. Basically, you get the same effect as if you had removed them with regards to them working, acting the same as the unheadered ones (in particular, requiring IPS patches that assume no header), completely transparently and without a technical burden. All that's left are the unpadded, headered ROMs, and if they're relatively uncommon, they can be rejected without causing much inconvenience.

I do understand your side. You are pragmatic. You think that if software can work around an issue, it's better than inconveniencing the user.

I am idealistic. I think that if something can be done better, we should do it the better way.

No disagreement here. I think software exists as a tool to submit to the will of the user, to extend what the user can do. It should never be used as a form of control of the user.

My experience with idealism is that it never works in reality like it works in my model of reality. This is an opportunity to learn more about reality and perhaps actually make it work there too.

And what's considered a "better" way is by a certain set of what's considered valuable, and what's considered costly. There are costs to changing things and some have greater ones, perhaps much greater than the benefit it gives. I can't help but connect this to free markets and centrally-controled ones, where the former is decentralized and has no person on top of everything, who understands everything, since it's the entire entity that's constantly making local decisions to maximize value, and able to take into account every individual's unique situation in maximizing value. Central control has to constantly disregard everything outside its model, to throw away the useful information encoded into the problems when trying to implement plans. It's outside the paradigm of coming up with a master plan that everyone will follow.

others might have different ideas and like to explore them

I don't see how there is any benefit to having copier headers, except to not break things (in a way that is trivial to fix) that have worked in the past.

I explained how the floppy disk size, interleaving, and multiple header formats makes the headers unreliable anyway; so it's hard to imagine a legitimate scenario where one is seriously keeping one ROM set that can both be used in an emulator and on hardware. I will acknowledge that it's possible to make such an emulator, but it doesn't currently exist.

However, I would like to entertain hearing more about why people want the headers. If I have missed something, please let me know.

From my perspective, it's about making executive decisions. Life is such that people often want conflicting things, and you can't please everyone. For everyone who really wants copier headers, there's another two who are tired of the problems they bring. The needs of the many outweigh the needs of the few.

I am tasked with weighing which position holds more merit. Supporting both with and without is my implicit endorsement of copier headers' continued existence. Those who want to abolish copier headers will never be able to do so while I stand in the way. Example: FuSoYa has stated he won't support headerless ROMs with his tool unless Snes9X stops supporting them as well.

Yes, one isn't forced to use a piece of software, but there is still an element of control in being given the choice to be constrained or use something else.

True, and I have gone pretty far to give people that choice. Yet I seem to be required to provide compiled binaries with header support here for some reason.

Unless I'm misunderstanding, the code to handle skipping an optional 512-byte header from a padded ROM is trivial.

Yes, it is. The unpadded ROM count has dropped substantially from where it used to be back in the '90s.

If you supported properly-padded ROMs with a 512-byte header (that is, acted as if the file on disk had that removed), would that handle most of the headered-ROMs out there?

Yes, it would.

If supporting this knocks out the need to remove headers on most ROMs, then you've eliminated a lot of unnecessary modifications of files all over the planet.

But you are skipping the IPS patching issue. The usage of command-line tools like sha256sum to check the integrity of or compare a file. The ability to rely on a ZIP archive's CRC32 to identify a file by hash without decompressing it first. ROM hackers manually adding and subtracting 512 to their file offsets. These are all intertwined.

If I remove header support, I encourage tool authors and patch creators to agree on a consistent choice. No emulator enforces copier headers, so headerless is the default choice to work everywhere. If I leave them in anyway, the situation will remain chaotic forevermore.

I'm not holding a gun to peoples heads forcing them to remove the headers, but I am doing my part to advocate my position. Others are free to create their own software, debate people as I have in favor of the headers, etc. We make our case and let the chips fall where they do, keeping our own autonomy and choices our own.

(in particular, requiring IPS patches that assume no header)

The problem is that this would cause issues as well. There are hundreds of patches out there that will only work when applied to a ROM with a header. If I were to support IPS patches but only after the header were removed, I would have people stating that I am forcing people to change their already-working IPS patches to suit my arbitrary decisions. What gives me the right to dictate/change how IPS patches are created?

And if I only support a format like BPS that stores the checksum, I would be told I'm trying to impose a new standard and take away everyone's IPS patches entirely.

I think software exists as a tool to submit to the will of the user, to extend what the user can do. It should never be used as a form of control of the user.

The divide is that I honestly do not believe I am trying to hurt the user. I am trying to make things easier for everyone.

I recognize that trying to state how things should be done comes off as arrogant. But I have a decade of experience working with this. I understand the issues very well. Uninformed people do have a tendency to work against their own interests, eg "A little knowledge is a dangerous thing", and "What's the Matter With Kansas?"

Should we have kept PCX and IE6 bugs around forever, because a very vocal minority didn't want to change? Was DEP a mistake because it required a few software programs to be redesigned? Growing pains are a natural part of evolution.

I feel that as software authors, we have a certain responsibility to usher things in the right direction. I will admit that I do so in a very heavy-handed manner. Much nicer would be to allow headered ROMs and offer to convert them. Thus allowing choice but increasing the ratio of unheadered to headered ROMs. But like I said, I am not very pragmatic.

There are costs to changing things and some have greater ones, perhaps much greater than the benefit it gives.

I don't believe that is the case with removing copier headers, but if one were to convince me, I would reverse course and offer copier header support again.

I just want to say something...
If you're trying to rid the world of copier headers, that's fine, copier headers are stupid. Just don't try to make people switch to "Rom Folders" at the same time. Windows just isn't the same as Mac OS. People don't interact with a singular "Icon" that contains a hidden internal directory structure.
If you really want to pursue Rom Folders, at least throw a shiny "Desktop.ini" file in there, give the folder a cool icon, and provide something to double click on to launch the emulator. And provide a good readme file with the emulator explaining that things are different in this town.

I just downloaded Bsnes recently, read the homepage, and saw that it emulates the GBA. So I was thinking "Oh cool, it now emulates the GBA! Let's try PocketNES". Then I tried to open a file, and wondered why the file open screen wasn't showing any GBA files. I couldn't drag-drop GBA files onto the main window either.
Then I looked for a readme file, but there was no readme file bundled with the emulator explaining Rom Folders or how to open files.
I couldn't even drag-drop a ROM file onto the EXE.
I thought that the program was extremely buggy, and the code for the custom-style rom menu must be broken or something. Which is really strange for an emulator with such a good reputation for accuracy.
Finally, I used the "purifier" tool to load a game, and that worked.
I was wondering "What the hell, GBA files have no copier headers on them. Why do they need to be purified?"

Later I read the documentation on the main website that explains what Rom Folders are, and realized why I couldn't open anything. Before reading this (and after testing out the emulator), I had no clue that the emulator worked with Rom Folders.

Needing to use a separate program just to try out an emulator is not user-friendly. It reminds me of the MAME for MS-DOS days, where you needed to run a Mame Frontend just to use Mame.

Putting the save file inside a Rom Folder is also a bad idea, because then it doesn't work on read-only media, network shares, or folders with restricted write permissions. If I was trying to use these, I'd be forced to make a ton of symbolic links, and that would be a pain.

-----

On a completely different note, PocketNES doesn't work very well in the emulator. The graphics blending is incorrect, and the Green channel is completely missing in the backgrounds. And the sound is missing too. But the GBA emulation looks like it's still in the early stages, so bugs are expected.

NES emulation looks really good now, for the mappers that are supported. Smooth and glitch-free. But I noticed that the purifier named any unsupported mapper as "NROM" in the manifest file.

Thanks for checking it out and leaving feedback.

> Just don't try to make people switch to "Rom Folders" at the same time.

Don't worry, bsnes v091 was a bit heavy-handed because it was the first release of a new UI.

higan v092 will let you load individual SFC images.

> Windows just isn't the same as Mac OS. People don't interact with a singular "Icon" that contains a hidden internal directory structure.

That is true, but I do want to stress how seamlessly I've made that to indeed be the case with kaijuu. And Windows users are used to the paradigm as well, double-clicking the CD-drive executes autorun.inf, and you have to right-click to browse a CD's contents.

> If you really want to pursue Rom Folders, at least throw a shiny "Desktop.ini" file in there, give the folder a cool icon, and provide something to double click on to launch the emulator

That's exactly what kaijuu does. And it also works on XP, where desktop.ini wasn't very powerful yet.

> I was wondering "What the hell, GBA files have no copier headers on them. Why do they need to be purified?"

The main issue with GBA is not being able to detect the save RAM type at launch. You can use heuristics of running code to guess, most of the time, but then you can't know things like save state format and size until after the code has run for a while.

> Needing to use a separate program just to try out an emulator is not user-friendly.

Integration is poor, certainly. If you drop a .gba file onto bsnes.exe, it will shell purify and play it. But I haven't yet been able to make a DLL that hooks into the GUI to add a standard file dialog interface to purify.

It's still planned, though. Long ago with the Qt GUI this was really easy, and I had a separate DLL that did stuff like SMC, ZIP, 7z, copier header, etc loading. I liked that approach a lot: it kept the code I hated out of my core. People could just delete the DLL if they agreed with me and wanted to do things my way. And I didn't have to maintain code I didn't like.

> Putting the save file inside a Rom Folder is also a bad idea, because then it doesn't work on read-only media, network shares, or folders with restricted write permissions.

The way the system is intended to work ... you have your ROM file that is your archival image. Put it in a read-only place, whatever.

To play in higan/bsnes, somehow that gets copied into somewhere (defaults to your home folder, can be pointed anywhere) and that is where all files related to that game are now stored. You can keep opening the game with the archival image if you want.

This is going to be important for some things. FlashROM and EEPROM in some instances are rewritable. So the idea is that internally, I'm going to use game folders, but the user won't have to care about it at all. It'll be behind the scenes.

If you're upset about the duplication of the image, I'm sorry, but I'm just not going to worry about it with today's multi-TB hard drives.

> On a completely different note, PocketNES doesn't work very well in the emulator.

It's not a very great GBA emulator yet.

> NES emulation looks really good now, for the mappers that are supported.

Only because NES is so easy, and I had help from Ryphecha. But thanks =)

The last system I want to support is FDS. Then it'll just be about refining what's there. Anything else (like NDS) will have to be other people writing the cores, or they won't happen.

Ultrabooks have 128-256 GB SSDs, not multi-TB hard drives. When higan starts supporting disk based consoles or someone else's DS core, the limitation of one emulated memory card per multi-hundred-MB application bundle will start to hurt.

byuu wrote:The main issue with GBA is not being able to detect the save RAM type at launch. You can use heuristics of running code to guess, most of the time, but then you can't know things like save state format and size until after the code has run for a while.

I doubt that'd be reliable at all, though. There are games that will fail to run if you give them a larger amount of RAM (most likely to deter flashcarts), and they will make the emulator think there should be more RAM than there really is. Heuristics will fail miserably with those.

Even 128GB is enough to store 32 complete SNES ROM sets. Or buy a $6 SD card / USB drive.

And the thing is, it only eats the space on games you actually play. I'll bet you a million dollars that 99% of people do not actually play even half of complete SNES game sets. Let's face it, 90% of SNES games suck royally. For every Zelda 3 there are twenty <Sports Genre> <Year> and Bass Fishin' games.

NDS could become an issue, sure. Those games are huge.

> Heuristics will fail miserably with those.

Yeah, I hate heuristics. I want an internal database for every commercial GBA game, and a manifest board for everything else. I am amazed there's not a simple one pre-made, but I'll make one. Eventually. SNES database comes first.

byuu wrote:I suppose if you guys want to push it there, great, I'm not going to.

A news update about beat has been posted, so the last step of pushing it is complete . There have already been ten downloads, so hopefully we will begin to see people leaving feedback about their experience with beat/BPS.

HJRodrigo wrote:

byuu wrote:I suppose if you guys want to push it there, great, I'm not going to.

A news update about beat has been posted, so the last step of pushing it is complete . There have already been ten downloads, so hopefully we will begin to see people leaving feedback about their experience with beat/BPS.

I like the post, but it doesn't mention one of the most important (IMO) features, and the one most relevant to this discussion, file checksums

EDIT: I see the actual project page has it, that works

qwertymodo wrote: I like the post, but it doesn't mention one of the most important (IMO) features, and the one most relevant to this discussion, file checksums

EDIT: I see the actual project page has it, that works

BPS is just so jam packed with positive features that it is a shame that all of them can't be mentioned in the news update

The response in the forums has been fairly positive and the download count has gone up to 33, so maybe down the road we will actually see some hacks getting released as BPS files.

The strongest point will also be the hardest. This is a solid format that's finished.

The whole thing is ISC licensed (with the caveat that extending the format is strictly forbidden unless you change the patch signature/extension), and aside from the optional metadata, is absolutely set in stone. There won't be a BPSv2, or a new ?PS format in a few months.

At some point I'll probably make a general advisory format for metadata. Like a "translation scene / ROM hack scene info template" or something.

People are naturally going to be weary because UPS failed. I screwed up there, but I learned from the mistakes and avoided them with BPS. The BPS spec has been around for over a year now and much more extensively tested than UPS. Now, BPM is a bit new, but I've done a few tests there and everything seemed okay ...

Also probably worth noting that currently the delta encoder sucks. It's O(n^2) time. So you can't do files >100MB or so in a reasonable amount of time (if ever, people have left 700MB files running for days without completing.) This is temporary, although it'll probably be a long while before we can solve it. Very, very tough problem to compare all possible differences in two files without it being quadratic. But all the older ROM-based systems work just fine with the linear coder.

What's important is that the format is just fine. Patch appliers will just work and remain fast. We just need to make a faster patch creator.

Also be sure to mention that Snes9X v1.54 will support it for soft-patching if you haven't.

I hear "BPS" and think of the intro to Yoshi's Cookie for Super NES. It used to mean "Bullet-Proof Software" but apparently changed its name to "Blue Planet Software" in 1996 when Henk Rogers moved it to Hawaii in the process of establishing The Tetris Company. That and Pipe Dream, inspiration for half the hacking minigames out there.

Someone needs to make a Yoshi's Cookie hack that just changes the B on the logo screen to an I, then distribute it exclusively as a .bps patch.

byuu wrote: What's important is that the format is just fine. Patch appliers will just work and remain fast. We just need to make a faster patch creator.

Also be sure to mention that Snes9X v1.54 will support it for soft-patching if you haven't.

It has been mentioned in the forum. beat has sparked the interest of quite a few people in the forums. Download count has gone up to 59, so the odds of seeing BPS patches has gone up slightly .

Of note, somebody even offered to do a Nimrod version.