bsnes and headers

[MottZilla]

You're right -- we're not in the 90s. I'm sorry that I can't go back in time and change all the demos that were done by all the demo groups, properly padding their size. I'm not asking people to do that either

Wouldn't the thing to do be to have people that distribute ROMs update what they are distributing so that eventually most people have ROMs without various useless headers? Most ROMs have a SMC or SWC header which contains no useful information really. Except for the copier. And if you are using a copier (I have several) you can't trust the file extension anyway and you need to have a program like UCON64 convert the ROM for your copier anyway.

Maybe someone does need to go and take every homebrew/demo rom that has a header or odd ROM size and cut the header and pad the ROM. You are right that it's crazy to ask *everyone* to do that for every ROM but if just someone does it and passes it along, eventually the problem is gone.

I think it's a reasonable suggestion to try to get rid of the copier headers on images just because they are useless to emulators and flash carts and copiers as well because you can't trust a file actually has a *valid* header for your particular copier either.

[Zonomi]

ROM files sent to Nintendo for approval don't have headers.

You're right they don't -- and do you know how that information is provided to Nintendo? I do: on paper. Do you want proof of my statement? I'm happy to show you the pages of the official SNES developers manual ("SNES Software Submission Requirements") which contain 2 pages of fields/data one has to fill out when submitting a ROM to Nintendo for review. Would you like to see it or will you just trust me?

That's why Byuu's approach would be the most accurate one (xml file for mapping information).
The analogy with web browser is accurate : why keeping an old, inaccurate and deprecated thing, useful only to 0.0001 % of rom users (and no, I'm not exaggerating).

Good fucking lord people! The topic at hand is whether or not ***REMOVING THE HEADERS FROM THE ROM ITSELF (MODIFYING THE FILE)*** is justified, or if they can just be ignored by software. This isn't about "why keep using them?" or other philosophical/opinion-based shit. I was told there are technical reasons why they should be removed from the file (vs. just ignored) and I have yet to see those reasons. This is not difficult to understand -- I'm sure all of us here are technical and intelligent and can understand this.

Programs can ignore those headers -- that's completely 100% legitimate and I couldn't care less if a program does or doesn't do that. What I'm against is requiring people to mass-modify all their existing ROM images to comply with an opinion, specifically when there's no technical reason the headers can't just be ignored/skipped over.

Capicé?

Only one word : accuracy.
How do you know if the header is valid or not ? And what about games that can't be dumped using a copier ? Do you add header ?
Byuu's work has always be for more accuracy, to be closer to the "real hardware". He doesn't force people to use his tools. But sometimes, to make people forget their old habits, you have to force them. And he even offers utilities to keep headers...

Headers are useless. Can they be ignored by software ? Sure, they can. So can be an overdumped ROM. Or any unuseful data at the end of the rom.
Majority of people doesn't need them. So why keep them ?

Anyway, all this is taking us nowhere. NoIntro people are redumping roms, sometimes without even using copiers. So, in a few years, most of, if not all, the roms will be dumped by anything but copier.

[Near]

I wrote up a very lengthy reply to each point here: http://pastebin.com/y9RcR0J7

But really, there is no point in having a discussion when you have clearly made up your mind. You reject every problem I point out as having a workaround. That does not dismiss that it's still a problem. We can live in a world where we do not need workarounds. You claim that because I'm an emulator author I have no say in the file format my emulator uses. That's complete nonsense.

I'll summarize things, since we're at a standstill. Here is the bottom line, for the fence sitters:

Less than 1% of people own copiers that require headers to work. Of those people, distributed ROMs (which make up 99.9% of how ROMs are obtained) are already going to have to be modified to run on their specific copiers. And of those modified ROMs, at least 30% will not work on both their emulator and copier with the same image. And if they use a floppy-based copier, it's closer to 70%. And even if they copied every SNES game in every region for every revision, they'd only eat up five extra cents worth of disk storage space. Although this duplication is trivial to avoid with a simple batch file. But the truth is, copiers are obsolete. Computers don't even have floppy drives or parallel ports anymore. Most stores don't even sell floppy disks anymore. In ten years, almost nobody will be using them at all. $69 or less will get you a flash cart that is vastly superior in every possible way.

There is the pragmatic approach of supporting copier headers because they are out there. This approach means we will have to support them forever. Even though they have not been for sale in decades, and will eventually be impossible to obtain. This approach continues the problems with patching, with hashing, with verification, with conflicting tools, and everything else I've said. Yes, there are workarounds. And they are annoying, and they are unnecessary. This approach has, does, and will continue to cause small amounts of pain points for everyone. Whether you acknowledge or trivialize it or not. Patching is still a problem in 2012.

blargg's point on browsers was an interesting one. Sure, they support JPEG, GIF and PNG. But do they still support PCX, TIFF and BMP? Should they support PCX until the heat death of the universe? What about all those people with Allegro graphics files?

There is the idealist approach of standardizing on exactly one format. This approach will cause a lot of pain for everyone, yes, this is true. But it's a one-time cost. We do it like taking off a band-aid, and it's done forever.

I truly do understand your pragmatic appeal, and I've done everything possible to ease the transition. I wrote a two-click, ten second batch convertor for all your ROMs. I wrote a launcher that forces header-only tools to also work with headerless ROMs. I've made a superior patching format to fix IPS' problems. I've spent dozens of hours writing posts like this to educate people. And so on.

And there is the compromose approach of slowly pulling off the band-aid. Allow emulators to load headered ROMs, but offer to remove them, then require they be removed, convert IPS patches, try and get RHDN to update their archives to all be based on headerless images, and so forth. This makes it a little less painful, but it'll take many times as long to finish, and we'll have to deal with threads like this a lot more. This is likely how things are going to end up, because we can't get everyone to agree.

Which is exactly the point of all this. Nobody's ever going to agree on whether or not to use or require headers or no headers. The pain points will continue forever until people finally say enough is enough and put an end to it.

The NES scene may be hopelessly stuck with iNES 1.0 forever, but I will spend the rest of my life trying to improve the SNES scene. I am against headers, and not only do I advocate that, I write the software and tools to make it happen. Perhaps you should do more than post tirades on a BBS if you want to make your vision a reality.

...

So, here's the bottom line. Support is growing every day. I have 40,000+ users who have all removed their copier headers already. No-Intro and Zapatabase have abandoned them completely. 80% of the ROMs distributed on the internet today have no headers at all. Every emulator and every known tool (thanks to my launcher) can currently work without headers.

Whether ROM hackers, patch makers and tool authors like it or not, SNES games are more likely to not have copier headers today. And that trend is only accelerating. If they choose to enforce them, they will be the ones causing the most inconvenience to their userbase.

There will always be those who will fight tooth and nail to the very end. But rest assured, unless you start taking these arguments seriously, you're going to lose this battle. In many ways, you already have.

...

And you know what? Even if I fail, so what? At least I tried, and advocated for something I believed in.

The people who give a shit about the issues I've brought up agreed and moved over. If people really want to cling to headers for their dear lives despite how illogical it is? So fucking what. Let them, I don't really care. They get what they deserve with all the problems they bring.

[HJRodrigo]

I wrote up a very lengthy reply to each point here: http://pastebin.com/y9RcR0J7

I am amazed you took the time to do that, but you need to get back to coding. I need my Far East of Eden Zero fix and you are the only dealer in town -j/k

[koitsu]

I appreciate the lengthy response and the time you spent writing it, byuu. I especially appreciated the part about the UFO format for mode21/hirom images -- I had absolutely no idea they were interleaved to that degree. The UFO copier was one of the harder-to-find units, by which time most of us back in the 90s had already saved up or gotten deals on SWC units (I had to do both -- I got mine from one of the guys at Anthrax in exchange for some code, but I also had to haul wood and hay for a summer to afford it even after the discount). I did have the "pleasure" of using a MGH (Multi Game Hunter) unit a friend of mine had -- you know, the dual Genesis/SNES copier -- which had its own file formats too (can't remember which ones it used). That thing actually fried (as in burnt up) after about a year.

I am in full agreement that you and I will probably never agree on this topic. I'm fine with that, as I'm sure you are.

You're absolutely correct that it's your emulator, your code, you can do what you want, etc. -- my point this entire time has been that there is nothing concrete that from a pure technical standpoint removing the copier headers solves a problem (sans what tepples just mentioned -- yup, in full admission that one is 100% justified!). It may make things a lot easier (keep reading please), but what I was told was that there were technical facts that justified modifying the ROM images vs. skipping/ignoring the headers via software (front-end, emulator, etc).

What you've listed off are lots of excellent points justifying your opinion/philosophy on the subject, based on experience as an emulator author, as to why you feel they should be removed -- in fact, it might surprise you to know I sympathise with many of them and can understand how they cause complications for you, or annoy you. I get it, I really do! But those are opinions/philosophies/personal reasons, not something that says to me as an engineer "there's no other way to deal with this, we have to remove the headers from all the ROMs. It's the only choice for everyone". There's a huge difference between that and "well, there might be some annoyance, but supporting all, or at least some, of the existing formats means we don't have to ask everyone on the planet to change something that's been out there for 20 years".

I hope we can at least come to terms in the sense that we see the standpoint one another takes, even if we don't agree.

One final point (or answer to your question I guess) -- you asked who the peers were whom share my opinion that the ROMs should just be left alone (with copier headers) as you'd like to talk to them. I reached out to all of them this afternoon. Of the off-forum four (4), two (2) of those stated very rudely that they don't want to talk to you as they have tried communicating with you in the past (about other things emulation-oriented) -- and as one put it "you shit down his neck". One of the 2 presented a single-paragraph viewpoint of you which one of the others said represented his opinion as well, and the former was quite happy to have me post his opinion on his behalf. So here it is:

Byuu is just a self-important spergelord. He's a talented coder, he wrote C++ code that didn't make me want to commit self-harm when I read it. But I wouldn't concern myself with his fits of whimsy. I imagine this is him holding breath and whining over header files in ROMs and him threatening to go home unless people undertake effort for his sake.

All I got from the 3rd: "Haha. No." What's interesting is that this individual actually uses one of your older emulation libraries in a private project of his own, so I'm not sure what the history is there, nor am I going to ask.

I can't imagine you would want to sit down and waste your time talking to people who feel this strongly about you. I know I wouldn't, at least not when the conversation would absolutely be destructive vs. constructive, you know?

The one who's on the fence does not care to involve himself in this matter at all. Quoting him: "arent you and him [byuu] done beating a dead horse already? it doesnt matter how i see headers if i did id post on that bbs").

As for the anonymous forum PM person, I'm sorry but that's going to remain that way -- not only because I'm a site admin, but also because that person has already been through enough drama in their life and I'm doing it out of respect. I hope you can understand that.

[tepples]

You claim that because I'm an emulator author I have no say in the file format my emulator uses.

You don't need a say. It's the front-end's job to make sense of headers, which may involve stripping them to a temporary folder before launching the emulator. If a PC based front-end wants to support the bsnes engine and ROMs with copier headers, it's just four reads for the common cases.

blargg's point on browsers was an interesting one. Sure, they support JPEG, GIF and PNG. But do they still support PCX, TIFF and BMP?

PCX I'm not sure about, but some browsers support a subset of TIFF, perhaps through a plug-in, because USPTO.gov demands it. Others support BMP because IE supports BMP, and BMP isn't really that much more code, especially compared to libpng. In fact, indexed BMP might even be as efficient over the wire as GIF due to HTTP Content-encoding: gzip. This analogy might work if you're familiar with the old compress tool that was popular on UNIX before Unisys started flexing its patents: .bmp.gz is to .gif as .gz is to .Z. As far as the web is concerned, the big gain of PNG over BMP is support for scanline differencing and support for alpha from day one (apart from old IE).

What about all those people with Allegro graphics files?

The Allegro library can save_bitmap() in the BMP format, and browsers can read BMP.

The NES scene may be hopelessly stuck with iNES 1.0 forever

Or at least until someone figures out how to push a firmware update for NES 2.0 support on the PowerPak. There's 8 KiB of repeated CHR data in the PowerPak firmware.

I have 40,000+ users who have all removed their copier headers already.

By what factor does this exceed the number of people still using their floppy or parallel copiers?

[Near]

I appreciate the lengthy response and the time you spent writing it, byuu.

Not at all what I expected to hear in response to that, but I'll take it! :D

The UFO copier was one of the harder-to-find units

For me it was the only one. If you'll believe it, I have never been able to get an SWC DX2. Even with all my connections, nobody would ever sell me one. eBay of course deletes copier auctions. Mind, I don't want one anymore, of course :P
The sd2snes is the new Cadillac of the SNES scene. And I have one made by ikari himself.

So my only choice was to pay $250 total to get one imported from Japan through a deputy. A long time later, I also bought a GDSF7 from Tomy, but it broke right away (not his fault, it worked when I received it, they're just shitty products.)

my point this entire time has been that there is nothing concrete that from a pure technical standpoint removing the copier headers solves a problem

We disagree on what's a problem.

I personally find it to be a huge problem that people can't apply IPS patches directly. They have to know whether the patch was made with a copier header, and whether their game has a copier header, and they have to be able to add and remove them.

That, to me, is a technical problem. It's a problem we can solve. If we disallow headers, people will make patches on ROMs without headers going forward. Problem solved.

I've released multiple game translations. I have dealt with this issue personally. I have been in the translation scene for a long time, and seen many people deal with this. And it's all so silly. Nobody uses copiers anymore. These headers serve no purpose.

What I will agree with you on is that it is going to cause headaches for a bit. But only for a bit. Everyone who has used bsnes since 02-2011 has had a bit of trouble, gotten over it, and all is good now. Some guys on my forum even converted every game translation patch to BPS and posted a ZIP of them all.

You say it's fine for ROM hackers to keep adding and removing headers to work around flawed tools like Lunar Magic. Why is it then not okay for you to add and remove headers for your copier usage? Do you at least acknowledge you are in the extreme minority (copier users) here?

not something that says to me as an engineer "there's no other way to deal with this, we have to remove the headers from all the ROMs. It's the only choice for everyone"

I have admitted many times that you are right, small amounts of code can get copier headers to work. And I've also said many times that others are free to add this support to my emulator if they want it. One guy on aep-emu.de posts a build with ZIP support each time. I've only ever thanked him for his trouble.

My point has always been: I dislike headers, I think they are absurd in 2012. Clearly, as evidenced by this thread, many people agree with me. I have a right to speak my mind, and I do. I control my software, and it's a reflection of my values.

If I were trying to force you to remove headers, then I would have a closed source emulator, and I would have coerced the Snes9X devs to lose their copier support, eg "remove them or you can't use the APU I wrote for you." That's disgusting and I would never, ever do that.

But if you want to use -my- software, remove the headers or add support for them yourself. You're even free to share your modified build with everyone. That nobody has done this tends to show you how little most people value headers.

we don't have to ask everyone on the planet to change something that's been out there for 20 years

As a fellow technical person, I don't see how you can feel this way. What does age have to do with it? The whole point of computing is advancement. Getting better and better. We used to use BASIC 20 years ago, and now we have better languages. Things evolve, people change. That's how technology works.

An argument for keeping copier headers is an argument for keeping IE6 support around forever.

This is not change for the sake of change. I showed you 19 different things that would be improved if they were gone. As minor or trivial as you take them to be, they're big deals for myself and many others.

I hope we can at least come to terms in the sense that we see the standpoint one another takes, even if we don't agree.

I do understand your side. You are pragmatic. You think that if software can work around an issue, it's better than inconveniencing the user.

I am idealistic. I think that if something can be done better, we should do it the better way.

There's nothing inherently evil or wrong about either approach. Just as there's nothing wrong with my focus on accuracy versus ZSNES' focus on speed.

as one put it "you shit down his neck"

That sounds anatomically impossible, but impressive nonetheless.

As for the anonymous forum PM person, I'm sorry but that's going to remain that way

That's exactly my point.

I hope you realize how childish it is to bring up anonymous comments in a discussion. There's no proof that what you are saying is true. What if I told you that someone PM'ed me and told me you "shot (his) grandmother and raped (his) dog?" What if I told you that seventeen people I spoke with on my IRC channel told me you were a horrible monster who eats babies? Even if it were true, what does it have to do with our conversation other than trying (poorly) to piss you off?

I don't doubt your honesty, everyone I've banned from my forum has formed a personal vendetta against me. I can name a dozen people who hate my guts. And I couldn't care less. If they are too afraid to state their names, then don't bring them up.

Either they can speak for themselves, or they can stay cowards who insult people in private. In which case, it's no wonder we don't get along. I despise cowards.

By what factor does this exceed the number of people still using their floppy or parallel copiers?

Worst case, 100:1. Best case, 1000:1.

[3gengames]

The solution is for the IPS to have the byte offset of the ROM and not just the file with whatever else is in it. Then you can have the tool decide where to patch it and it would do it right. The problem is in the IPS system, not the file format. The format shouldn't matter and is easily manipulated if the IPS had better information.

[Near]

Some IPS patches actually try modifying the copier header to work on the author's copier type. A good example of this was Final Fantasy V.

But yes, if emulators made IPS treat offsets as only applying to the ROM content, that would solve the problem with IPS patches. After a bit of pain. Everyone would have to recode all the old IPS patches that relied on headers, and everyone who downloaded them would have to update their patch collections. But it would be a one-time expense to solve the problem. We could make tools to recode existing patches in two clicks, we could have emulators use heuristics to try and minimize disruptions. We could update RHDN's entire repository at once. If people were willing to attack the problem head-on.

BPS makes it even easier. You could apply the patch assuming no header. If the checksum fails, try again with a header. If it fails again, you say they have the wrong ROM. And even better, both bsnes and Snes9X enforce BPS application with no header, so no patches will be made with headers if people want them to be soft-patchable. BPS patches are guaranteed to either work, or tell you that you are using the wrong ROM.

If you took this one-time pain point, you would eliminate IPS patches from my list of objections to copier headers. You'd still have 18 more arguments to go, but yes. The most important point would be eliminated.

I guarantee you that if people are fighting me over copier headers, they will also fight me over a rule saying that IPS should only assume ROM offsets. Why? Because it breaks 20 years of history involving IPS patches errantly made against headered ROMs. People will have to change their existing files.

Apparently, change is a really big problem when it's not absolutely necessary.

[HJRodrigo]

BTW, beat has been added to RHDN. If you want anything corrected, just let me know.

[3gengames]

It'd be easier to fix the old IPS patches. Keep them the same just with a new format. Label them old. Then put a flag that says it just works on the raw file with no ROM separation. Then run them through a single converter. As long as you have every file in the world anyone would need, conversion is cake.

[Near]

I'm on your side here, let's fix the old IPS patches. Not as nice as using a format with a built-in checksum, but I'll take it as a positive step.

But I'm telling you, there will be just as many people upset about removing copier offsets from IPS patches as there are from people upset about removing copier headers from ROMs.

Both require modifications to existing files. That's the whole crux of koitsu's arguments, "why change files that can work now and have been around for a while? People already have headered ROM+headered IPS and unheadered ROM+unheadered IPS pairs for all their files and they currently work with existing emulators." In fact I'm curious: koitsu, what's your take on recoding IPS patches?

So by all means, please go forth and convince people to recode IPS patches based on headered ROMs. You have my full support and gratitude if you do.

[3gengames]

I mean I'm not saying I should do anything. Like I said, there may not be one stanard. But we, as a community who are dedicated to building these ROMS and maintaining their formats and such, we just need to get together and say this is how it will be done. Then I'd say that's when we need to go on the final crusade and piss everyone off, hehe. I myself believe that we need headered ROMS. Even if people don't like them, as long as it DOES work with everything and has a good quality format, you can always tell them just to split the ROM and CRC/MD5/whatever they desire from there. Just my $0.02 on this subject.

[Near]

Just to be clear, you're aware of the difference between iNES and SNES copier headers, right?

iNES tells emulators how to play NES games. Emulators can't play the games without them (or a database.)
SNES copier headers do not. Emulators completely ignore them.

The reason SNES copier headers exist is so that copiers can read the first sector of the file and determine where to place all subsequent sectors. This shaved off less than one second of seek/read time to read the internal SNES header at $7fc0/ffc0 first. These headers are nothing more than poor duplications of the internal header. So it'd be like if NES ROMs already had iNES footers, and we decided to add another one so that it was at the top.

[3gengames]

I wasn't 100% sure, but I understand that. I didn't know there was a huge difference but eh.