bsnes and headers

[qwertymodo]

You'll definitely want to tack down a basic metadata spec before people start using it, so patch applicators can agree on field names to parse. Obviously, being xml, it's meant to be extensible, but nail down the basics as soon as possible so it can be standardized.

[HJRodrigo]

Seems byuu's "friends" at RHDN, decided to step in an educate people that IPS is good enough and for the times it is not we should be using xdelta. But at least one ROM hacker stated he would use BPS for his N64 projects, so hopefully he will share his thoughts on the format.

I have only tried BPS on a small scale, I need to find a project that will really let me get to use it .

There have been 88 downloads, so maybe one of those people will share their thoughts on the format. byuu's "friends" have shared their thoughts on the format, but it somehow ends up being about byuu . byuu was not being sarcastic when he says he is a polarizing figure.

[blargg]

That's because you're pushing religion. Your own post focuses on people as well, with several scare quotes.

[HJRodrigo]

That's because you're pushing religion. Your own post focuses on people as well, with several scare quotes.

Religon? I don't follow your logic, but yes my post is mentioning people because at first it was a discussion of BPS, then people brought their feelings about byuu into the discussion. It is like comparing operating systems and having people bringing in feelings of Gates or Jobs into the discussion. It is not scientific or based on facts. If you feel, me wanting a discussion to be on the merits of a format without having its creator thrown in is religion, then you are confused. I need facts not feelings, if you prefer feelings then so be it. This has nothing to do with headers, if that is your motive for throwing in that jab of pushing religion.

What you call scare quotes, I call trying to be civil. I would rather refer to those people in a hash manner.

I legitimately find BPS interesting and think it could be a good thing in the long run. I don't care how many past failures byuu has had or if he has OCD, I want to hear technical flaws with the format. I heard three and I acknowledged them.

Flaw number 1
Patch stacking is not possible

Flaw number 2
Delta patching is slow on large files

Flaw number 3
No other implementations

I think judging a format based on your feelings for a man, sounds like the same logic used by religious fanatics. It is that same fanaticism that delayed man's technological progress, in many fields of science. In fact, explain to me how the following posts are technical assessments of the format instead emotionally driven comments and I will admit that I was in the wrong.

I don't really see a point to byuu's continued revisions of his patching format. They basically just suit his uses.

Because he doesn't see a point in a revision, that makes it a valid technical reason. Let us not forget the jab at the end towards byuu.

For old games there is just no advantage at all over UPS. The only new thing here is the delta patching, which is needed for big and new games. And that's where patch creation fails!

So he discredits delta patching when it is the very advantage that BPS has over UPS and states that delta patching is only needed for big games. It had already been mentioned that hacks with insertions or deletions benefit greatly from delta patching. He doesn't care because he wants to prove byuu's format is a failure, regardless of the facts.

But there is no reason to implement [Delta patching] again except for soft-patching.

He is stating here that there is no reason that anyone should implement their own delta patcher (except for soft-patching). Very good technical reason, no wait that is called an opinion with no scientific basis.

[New features of BPS are] Mostly the stuff UPS already had (handling of SNES headers, embedded readme) and no one cared about back then.

Very technical analysis of the format. BPS is bad because nobody cares about those feature. Very scientific approach.

Only to byuu and his OCD [are a programs license or complexity important]

I am at a loss for words. There are absolutely no other programmers in the planet that care about a programs complexity or license. Of course absolutes are part of a scientific mind.

The only thing that a new format could bring to the table would be one that works across the board from tiny to biggest files. And beat just fails here, end of story.

Wow, the ONLY thing a new format can bring to the table is that. How very open minded, this reminds of the same open mindedness seen in religious fanatics.

[Near]

I told you there was a reason I stopped wasting my time with them entirely.

You can't make people care about their own self-interests. All you can do is ignore them, and help out the people who do care. If we have to form a giant divide between { game folders, BPS, headerless images } vs { SMC, IPS, copiers }, then so be it. I'm not going to live in the past forever.

> but it somehow ends up being about byuu

Exactly. Whenever I'm involved, it becomes nothing but an Ad Hominem fest with no technical merit in the discussion.

Intelligent people will realize that, and stupid people I could care less about.

It doesn't matter who I am, or what I did or said in the past. That has no bearing on the technical description of the file format involved.

> Patch stacking is not possible

This is the same thing I've explained countless times. I am so god damn tired of repeating myself.

Patch stacking with IPS and BPS are equally possible. BPS has a checksum to verify you applied the patch correctly. It's impossible to support an arbitrary list of patching and have checksums for them all. So if you want to stack patches, ignore the checksum. Problem solved.

Right now, right this instant, 98% of patches on RHDN are meant to turn file A into file B, and not to be stacked. That's not hyperbole, that's not an exaggeration, that's fact. So the smart thing to do is default to validation.



Looky there, a check box to bypass validation, too.

But this is like long-form birth certificates, the truth doesn't matter, it's just a talking point.

> Delta patching is slow on large files

Yes, it is. That's a valid criticism, and one I'm working on.
It's the hardest technical challenge I've faced in my 15 years of programming, in fact.

I currently use a list of 64K nodes that keep 16-bit IDs of data at each address, and each node is a linked list of addresses that match, cutting the overhead to O((n/65536)^m), which is still quadratic. I don't know how Xdelta or bsdiff does it. Their technical descriptions use the entire Greek alphabet. bsdiff's author used it as his doctoral thesis: http://www.daemonology.net/papers/thesis.pdf

When you understand this:


You'll understand just how hard a problem this is.
Yes, it's my problem, and I have to solve it. But it's going to take some time.

Now to be fair, it only affects delta patches, which aren't made for ROM-based systems (NES, SNES, GBA, etc.)
And it's only an issue when the file size is >16MB.
And it's only patch creation, which happens one time.
And we're only talking ~15 minutes for a 16MB file.
Patch application is lightning fast.

> No other implementations

So far python-bps and Snes9X. Chicken and egg problem. How many other implementations are there of Xdelta? (hint: zero.)

beat isn't Xdelta nor bsdiff. It's goal is to be as simple as possible to implement a patcher for, at the cost of being slightly less efficient (typically ~1% difference in size)

Here is Xdelta3, over 500KB of math.

beat is 20KB of code to both compress and decompress. It's like comparing ZIP against PAQ/KGB.
And amusingly enough, in many instances, it beats Xdelta.

> explain to me how the following posts

In the future, I'd appreciate if you didn't quote that forum. It undermines my efforts to avoid them.

> That's because you're pushing religion. Your own post focuses on people as well, with several scare quotes.

I don't know what he said and I don't want to know. But that's unfortunate if it is true. Still, two wrongs don't make a right.

...

This is why I didn't bother posting there. I'm happy doing my own thing.

[HJRodrigo]

Patch stacking with IPS and BPS are equally possible. BPS has a checksum to verify you applied the patch correctly. It's impossible to support an arbitrary list of patching and have checksums for them all. So if you want to stack patches, ignore the checksum. Problem solved...

...Looky there, a check box to bypass validation, too.

That is good to know. That version you are using, I notice it is v03, is it available somewhere?

Right now, right this instant, 98% of patches on RHDN are meant to turn file A into file B, and not to be stacked. That's not hyperbole, that's not an exaggeration, that's fact. So the smart thing to do is default to validation.

Valid point, but I suppose the fear was the perceived loss of an ability already possible with IPS

Yes, it is. That's a valid criticism, and one I'm working on.
It's the hardest technical challenge I've faced in my 15 years of programming, in fact.

I wish you much success in that venture, especially considering all the other aspects of you hobby that you have on your plate.

In the future, I'd appreciate if you didn't quote that forum. It undermines my efforts to avoid them.

Understood. Alot of your comments make sense now, so I apologize for putting any salt in any old wounds.

I don't know what he said and I don't want to know. But that's unfortunate if it is true.

I have no clue why blargg used that terminology. Nobody mentioned any religions, churches, ceremonies, gods or goddesses. I think it was a reference to the discussions here that referred to your philosophies as a "crusade", which is a religious term. I think it was a jab at me for backing two of your philosophies here in this forum. He was basically using "pushing religion" as a euphemism for "crusade". Headers and the like was not even the issue of discussion over there. BPS, beat, byuu, xdelta, UPS, and soft patching were the items of discussion.

It is sad to hear that your views are now considered a religion. I am not even following your beliefs I am just agreeing with them, as I had those feeling toward headers before I ever even knew of you. I like BPS patches not because you made them, but because I was legitimately impressed with its features. There were others in that forum that were also impressed, so I don't see what is so strange with liking BPS as a format.

[Near]

> That is good to know. That version you are using, I notice it is v03, is it available somewhere?

It's the standalone, non-BPM patcher, designed more for when I was catering ROM hacking with direct-file patching.
It's available here: http://byuu.org/programming/bps

> I suppose the fear was the perceived loss an ability already possible with IPS

The problem is that I already addressed this when UPS first came out. About six times. To these same exact people.
And I've also addressed it more than once with BPS.

The person saying this knows for a fact it's not true. They are spreading FUD.

> I apologize for putting any salt in any old wounds.

It doesn't bother me. I've dealt with these people since 1997, so I have a rather thick skin.
They haven't discouraged me to stop working on things yet, and they never will.
In fact, for better or worse, I've come to use their negativity as motivation.
What I don't like is this forum having to deal with it too.

> Headers and the like was not even the issue of discussion over there.

Speaking of that, it's worth noting that beat itself doesn't push that issue.
With the checksum known, you can apply a beat patch to a headered ROM.
Now with the soft-patching in bsnes and Snes9X, we made it require headerless for simplicity and consistency. May as well, since there are no legacy patches to break, and emulators do not use copier headers at all.

[blargg]

HDRodrigo, I'd like to apologize on my religion comment. I reviewed your comments in this thread and elsewhere and saw that it was just that one message that set me off. There were two things going on:

You were referring to people as byuu's "friends", which I interpreted as an underhanded put-down of them (the meaning of the scare quotes). I tend to respond poorly to attacks on people.

I take it you just meant that you posted an announcement of BPS, and some other members of the forum replied with their thoughts that IPS was good enough and that BPS' advantages for their purposes didn't warrant the drawbacks of a format change.

My religion comment was how the focus is on how many people you get to use this format, as opposed to what problems it's solving for people (and what new ones it's creating), which is a focus not on what format(s) they are using, but on their entire workflow. It's also how an issue of what file format people uses becomes something almost moral, like you're purifying things. It rises above it simply being something that solves some technical issues, to something that must happen.

I would love it if technical discussions focused on technical issues and never brought people into it. I'm guessing that they brought byuu into the discussion because people arguing for new formats have unnecessarily brought their own character into the discussion in the past, or have pushed something technical that they had given signals that they weren't interested in.

I've emailed byuu about a possible speed improvement in the delta patcher, and am eager to hear whether it helped any.

[Near]

> It rises above it simply being something that solves some technical issues, to something that must happen.

Yeah, I never intend to force the entire world to do things my way. I only ask that they give me discretion when designing my own software to work how I want it to.

In this case, it's really not a big deal. If authors want to keep getting complaints about header vs no header patching problems, they're more than welcome.

But the general reasoning is that less tools are easier, so if people were to unify around one format that solves all problems, it would make things easier. Again, not really an issue in this case. Easy to make a tool that can apply all patch format types (even if you need a separate binary for Xdelta.)

> I'm guessing that they brought byuu into the discussion because people arguing for new formats have unnecessarily brought their own character into the discussion in the past

Basically there's probably around 20 people who have a grudge with me, at least half for more than 10+ years now. So whenever my name is brought up they go into instant Ad Hominem mode.

I'm not going to say I did nothing wrong. I can be extremely crass on occasion, and I'm all to happy to argue back with asshats (rather than ignore them as more refined individuals do.) But it is amazing how long people can hold a grudge for.

But yeah, it doesn't matter how much you like or hate someone. The only thing that matters is the technical merits. It's sad to see otherwise bright individuals reduced to howling obscenities.

> I've emailed byuu about a possible speed improvement in the delta patcher, and am eager to hear whether it helped any.

I'm having a bit of trouble with the implementation, but I didn't want to keep bothering you with yet another one of my problems.

But yeah, I can keep a hash of eg "offset A-{A+N}" where A={source 0-file size}, and N={arbitrary length to match} values. Now if I have a 700MB file, and a 32-bit hash, that's a tree of {32-bit addr, 32-bit hash} * 700MB = 2.8GB tree. I would have to keep it sorted to do logarithmic searches, so I'd have to use a fancy self-balancing binary tree. I could reduce the memory size by incrementing A by N each time, rather than by 1. So for 16-bytes, we'd be down to 175MB of RAM, which isn't horrendously bad at least.

But we'd miss some matches entirely now, like say if we wanted to match 8+ bytes, then a pattern from 9-22 wouldn't register. Although my goal isn't to beat Xdelta's efficiency (it has advanced compression techniques built-in; and my variable-length integers aren't ideal for post-compression efficiency); I'd still like to be somewhat competitive. I'm already somewhere between Xdelta3 -9 and Xdelta3 -9.7z, meaning I can't beat the best-case Xdelta already.

[tepples]

Video game storage media larger than 64 MB, the size of the biggest N64 Game Pak, tend to be block devices such as optical discs or DS cards. So perhaps for matches farther than 64 MB, you can optimize for 512-byte offsets, as that's the size of a sector on most block devices.

[HJRodrigo]

HDRodrigo, I'd like to apologize on my religion comment. I reviewed your comments in this thread and elsewhere and saw that it was just that one message that set me off.

No problem, but thanks for the apology. It is fully accepted.

You were referring to people as byuu's "friends", which I interpreted as an underhanded put-down of them (the meaning of the scare quotes). I tend to respond poorly to attacks on people.

It was the only civil way I could think of referring to them (without actually using their forum names). I guess I could have used acquaintances, but that seemed even more odd. I wanted to avoid saying enemies (or any similar term) or any expression that would connote my frustration with them.

I take it you just meant that you posted an announcement of BPS, and some other members of the forum replied with their thoughts that IPS was good enough and that BPS' advantages for their purposes didn't warrant the drawbacks of a format change.

I am fine with people stating that point. IPS does have its perks. It is widespread and is relatively simple to work with. I was disappointed with the dismissing of all of BPS' merits, based on feelings and emotions. It was pretty obvious that several individuals had not even looked at the specs of the format and were declaring that the format had no use.

My religion comment was how the focus is on how many people you get to use this format, as opposed to what problems it's solving for people (and what new ones it's creating)...

My focus was to get as many people to try it out, not switch to it, in the hopes that people would begin discussing what problems it's solved or created for them and much more. It seemed to me that certain people, that had no plans to ever try out BPS, were only willing to give opinions instead of trying it out and give an actual thoughtful analysis of BPS' pros and cons. Basically they only demonstrated an interest to tear it down and not offer any constructive criticism.

I will give my philosophical view on the matter. One reason, I feel supporting or evaluating new open source solutions is important is because technology is in a constant state of evolution/progress and it would be nice if for certain things, we controlled the direction it was headed, as opposed to businesses with commercial/financial interests. A random example of my view in practice is my backing of the Ouya project. I don't believe people should switch to that "open" platform, but I feel if more people try it out it may bolster more open source projects into the public's eyes. Theoretically, the more people trying out open source projects, the faster they can evolve to be competitive with closed projects. Closed projects will in turn try to surpass them as to not to fall behind, thus the open source projects can direct the growth. Here is a proverb I was told that illustrates how I feel competition can improve technology, "As iron sharpens iron, so one person sharpens another". There, that is the most religion, you will get out of me on this thread

I've emailed byuu about a possible speed improvement in the delta patcher, and am eager to hear whether it helped any.

That is good to know. Fresh ideas and points of views are good for progress. Maybe the speed barrier can get cracked sooner rather than later, if more individuals would just come together and help support the project (regardless of whether they will use it or not) rather than tear it down. Reading the discussions between authors of video encoders has always impressed me. They each have their own format that they are backing/specialize in, yet they still share ideas and discuss the different ways something could be implemented (This is based on the discussions I have seen take place on Doom9's forum).

[blargg]

Thanks for sharing more about your philosophy. I really like it and the benefits it has for open development. I take you as basically asking people to step out of their comfort zones a bit and give some things a try and really scrutinize them, so that the developers can fix problems and then put them out there again for further evaluation. This is the only way, as you say, that they can reasonably match the proprietary things out there, which have had a lot of this kind of evaluation behind the scenes. It's also made me think of my own approach, which is sadly more the "what, that's crap, nevermind I'll go back to what I was using before" kind. This makes me want to at the very least give a semi-detailed report of why I switched back so the developers can fix that for the next person.

This is all the more reason to seriously avoid any judgments of people who don't want to try a new format, to be as absolutely technical as possible when presenting it, and always allowing for it not being the end-all format, that current ones might really work better for some people. I personally have low patience for people who even have hint that they think their solutions are superior and could have no objections or shortcomings for particular uses. Humility and openness to hearing about problems win big points with me. I'd like to be less short with the former people, but eh this is how I am currently.

[Near]

> I personally have low patience for people who even have hint that they think their solutions are superior and could have no objections or shortcomings for particular uses.

I always try to list pros and cons, the pros are usually much longer, but the one recurring con that always seems to dominate is legacy support.

But for fun ... correct me if I've missed anything.

Against IPS:

BPS pros:
* validates patch, input and output files are correct (a huge deal thanks to copier headers)
* embeds authorship metadata if desired
* supports >16MB, and >4GB files
* supports data insertion, deletion, and move deltas
* smaller patch sizes
* no extensions exist, no extensions are possible that will break old patchers
* no ambiguities (EOF as an address versus an end file marker, etc)

IPS pros:
* can be implemented with half the code (both are under 4KB)
* ZSNES and older Snes9X releases can soft-patch these files
* some people already have IPS patches or tools

Against Xdelta3:

BPS pros:
* can be implemented in one twentieth (or more) of the code; allows embedded soft-patching support in emulators
* embeds authorship metadata

Xdelta pros:
* lightning fast delta patch creation (temporary advantage, but a significant one all the same)
* an Xdelta3 -9 (max compression) inside a 7z -mx=9 (max compression) archive is typically 5-10% smaller

[Shiru]

I guess an IPS to BPS converter would help to gain some popularity to the format, as it would allow to get back catalog available in the new format and allow to make comprasions between file sizes in different formats in an easy way.

[tepples]

If creation of Xdelta3 patches is lightning-fast, then couldn't one create an Xdelta3 patch, parse it, and represent the same data in BPS format? Or are the data models too different, like GIF and JPEG?