Emulation accuracy level

[Zepper]

- What does make you think a certain emulator is accurate? Could you define "accuracy" in terms of NES emulation?

- The "top" two names Nintendulator and Nestopia, mostly the second one, is largely mentioned as "accurate", as if all the others would have a lower level of accuracy... or the direct label "inaccurate". So, what's the rule after all?

- How can you analyze the CPU emulation? Is there a specific test suite that prints "OK"? If all the opcodes look "OK", can you say "it's accurate"?

- On other hand, what about the PPU emulation? Looks like the hardest part, quite complex if you take a test suite. Let's assume all those tests pass "OK", can you say "it's accurate"?

- It's boring to read over and over again about those two ones; it's much like (with respect) that Nike and Adidas produce the best tennis out there; so all the others are low quality?

[Near]

At least they actually are accurate.

Imagine how you'd feel if everywhere you went, everyone kept talking about how accurate NESticle was and how it runs everything perfectly fine, and the emulator you spent years trying to perfect was seen as a pointless waste of resources because everyone clung to their 486s for over a decade now.

That said, the vast super-majority of people have absolutely no idea what the word means. It's a cliche buzzword. You can't let yourself get upset over issues of popularity. Write your emulator for you and no one else. If that no longer makes you happy, then you should move on. Life is too short to be unhappy.

[MottZilla]

byuu is right that most people have no idea. Really though most NES emulators atleast are way more accurate than they were back in the days of Nesticle being top dog. Generally people that don't know any better will assume the emulator they like to use best, is the best in every way.

I think the main reason you hear Nestopia and Nintendulator alot are just because they are pretty accurate and they are very popular/have high profiles. Just like anything else you will find one type of "product" that has a high profile and is assumed the best or among the best but infact there are other "products" of that type that do the same job as well or even better.

To most people that use an emulator to just play games, accuracy to them is just about if the games they try to play work or not and if they work without glitches.

[tokumaru]

- What does make you think a certain emulator is accurate? Could you define "accuracy" in terms of NES emulation?

I define an emulator as accurate if it displays the same thing my NES does when running a particular program.

This may be subjective, but Nestopia scores some extra points from me because of the NTSC filter, which makes it's output even more similar to what my real NES displays.

- How can you analyze the CPU emulation?

I think CPU is not much of a problem. The 6502 is very well documented, and games themselves are a good test, IMO. If something is wrong with your CPU emulation games are likely to not function well (there are some instructions that are rarely used though).

- On other hand, what about the PPU emulation? Looks like the hardest part, quite complex if you take all those test ROMs created by blargg.

Yeah, this is the hard part. The PPU is pretty complex and there's still a lot we don't know about its internal behavior, so some things are likely to be off in emulators. Like that thing tepples recently ran into about sprites flickering when you turn rendering off mid-screen. That's because of some really obscure internal thing the PPU does no one really understands. I hope emulators will be able to implement that soon, like the controller reading bug when DPCM is used.

When I was doing some crazy tests with the PPU trying to blank the top 16 scanlines in order to hide scrolling glitches, I saw a lot of weird stuff. Emulators disagreed a lot on timing, and Nintendulator was the closest one to the real NES. Many emulators appeared to not emulate the timing of the sprite pattern fetching right, because I got varied results from bankswitching from a bank with blank tiles to one with actual graphics around the time the patterns were supposed to be fetched. I've also run into the flickering sprites that caused problems to tepples.

- It's boring to read over and over again about those two ones;

I have to admit I haven't used RockNES much, probably because I didn't like the UI when I first tried it. This means I can't say how accurate it is, but I know I'm not comfortable using it. Nestopia is a joy to use, and it's certainly my #1 choice for playing. The fact that it has proven to be very accurate placed it at the top of my development tools too.

Nintendulator has palette issues, the video tears up constantly (I couldn't find a VSync option for it), but it has saved my butt countless times. Using the modified version with the better debugger (because I simply can't stand the original debugger) I solved many bugs and adjusted several pieces of timed code (because it displays the current PPU cycle within the scanline).

Achieving the decent level of accuracy for playing is easy, because finished games are aware of how the hardware works and are usually programmed to avoid any issues. During the development of a game however, it's easy to end up in unexpected situations. Some of these situations trigger obscure behaviors in the console, and because these aren't normal situations it's expected that many emulators won't get the results right. And this is where accuracy shines, when an emulator is capable of reacting similarly to the real console under these uncommon situations. Sometimes the programmer even creates these situations on purpose, expecting to take advantage of the reaction of the console.

Emulators that mimic the console at a lower level are more likely to react correctly, because they don't abstract certain functions for better performance, they actually take the same little steps the console does. I guess this is why Nintendulator is accurate.

[tokumaru]

Fx3, is this really about you being upset because RockNES isn't as popular as those 2 emulators? Since I haven't used RockNES much, I couldn't really say if it deserved any attention in the accuracy department. So I just downloaded and gave it a shot.

The UI is like I remember, not very fun to use, but OK if all you do is open a ROM. I set up my usual keys, then opened Super Cars, which relies on $2004 reads and stuff. Not a big fan of the "scanlines" effect, it's old and doesn't look like a TV at all, so I turned it off. Game works initially, but the emulator crashes (!) when I select a track.

OK, that game is kinda tough, so I decided to pick something easier. Kirby. Initially OK, but once the game starts, the status bar is pushed down when certain sprites are used (smoke, fire, things like that). There's probably something wrong with the MMC3 scanline counter.

OK, another game. Batman - Return of the Joker. It goes directly to the "STAGE 1-1" screen, entirely skipping the title screen, and when the game is supposed to start I only see the HUD and some screwed up music. Another random game: The Fantastic Adventures of Dizzy. The status bar shakes. Bucky O'Hare: same thing.

I'm not kidding, I picked the games above at random. After those I ran a few games that worked fine: Smurfs, SMB, Pugsley's Scavenger Hunt. The emulator ran very slowly on my PC though, and I don't think there's a good reason for this, since I run Nestopia with the NTSC filter at full speed.

So, my impression wasn't very good. Several games were visibly wrong, meaning that the emulator isn't 100% for gaming, let alone for developing.

I'm not trying to be mean to you, but this is not the first time you complain about other emulators getting all the attention, and when I finally try yours I see it's obviously not as good as the popular ones. Nothing wrong with not being perfect, your emulator is a work in progress and might some day be really really good, but I have to be honest and say that it isn't yet. I hope you can see that and understand that those 2 emulators get the attention because they have their merits, and I haven't seen anything in yours that beats them, so I don't think you should be demanding attention until you have something to show that really stands out.

[Dwedit]

My standard for accuracy:
If it's shaking, it's inaccurate. That's about it.

[tepples]

To most people that use an emulator to just play games, accuracy to them is just about if the games they try to play work or not and if they work without glitches.

Producing the same output as the original hardware when running a particular known program is "compatibility". Accuracy is the same output with an arbitrary unknown program, and that's what Nintendulator and bsnes strive for. Other emulators might sacrifice accuracy for lower system requirements.

My standard for accuracy:
If it's shaking, it's inaccurate. That's about it.

Not always. I'll grant "If it's shaking, and it doesn't shake the same way on an NES, it's inaccurate."

[tokumaru]

Producing the same output as the original hardware when running a particular known program is "compatibility". Accuracy is the same output with an arbitrary unknown program, and that's what Nintendulator and bsnes strive for.

I write a freaking huge message trying to say something and tepples shows up and says exactly what I meant to say but using only 2 sentences. I guess I suck at english (I'm not that good at organizing thoughts to begin with)! =)

I'll grant "If it's shaking, and it doesn't shake the same way on an NES, it's inaccurate."

I was gonna point that out. There are several released games that shake or flicker on hardware, specially when switching from the gameplay window to the status bar or vice-versa, and screen splits in general.

[MottZilla]

tepples, I was talking about what the "average" emulator user tends to consider accuracy to be. I know that accuracy is as you stated and agree with that.

[Near]

Achieving the decent level of accuracy for playing is easy, because finished games are aware of how the hardware works and are usually programmed to avoid any issues. During the development of a game however, it's easy to end up in unexpected situations. Some of these situations trigger obscure behaviors in the console, and because these aren't normal situations it's expected that many emulators won't get the results right. And this is where accuracy shines, when an emulator is capable of reacting similarly to the real console under these uncommon situations. Sometimes the programmer even creates these situations on purpose, expecting to take advantage of the reaction of the console.

Oh man, this forum is heaven. Seriously, no place else I've ever gone had understood this. Yet you guys get it down to a tee! Why, why, why oh why are you guys limiting yourselves to just NESdev? :(

That's exactly it. Years of ROM hacking and seeing every last patch I've made blow up on hardware drove me to this mentality. Yet I still can't convince other ROM hackers until they hit the same walls I have. I guess we all have to learn this the hard way.

Accuracy is spending two weeks with the process of elimination, hundreds of tests and floppy disks, and lots of pizza and beer to figure out why your timing is off by 1/10,000,000th of a second, and taking a massive speed penalty to fix it, even though it affects no known software. Accuracy is seeking edge cases out before other people run into them, and before you find a commercial game that relies on it. Accuracy is not "does it play Zelda?" -- no, playing games is a side effect of accuracy.

And this is why you can't tell the accuracy of an emulator unless you know the system inside and out. A gamer has virtually no room to talk, a homebrew dev or ROM hacker has a bit of room to talk, but only an emulator developer for that system can really say for sure.

Frankly, I don't care if your emulator is the most accurate or not. If you're actively looking for new, quirky behaviors on hardware; I respect the hell out of you and would love to use your work and help you spot bugs. If you find a new behavior, that helps everyone. You don't have to be #1 to do that.

[tepples]

Oh man, this forum is heaven. Seriously, no place else I've ever gone had understood this. Yet you guys get it down to a tee! Why, why, why oh why are you guys limiting yourselves to just NESdev?

I wasn't aware of much of a Super NES scene until it exploded after release of the SNES PowerPak. At this point, I wonder if there'd be any objection to splitting up SNESdev into a category of forums that parallel the NESdev category.

Accuracy is not "does it play Zelda?" -- no, playing games is a side effect of accuracy.

To what extent do you believe the MAME people agree with this philosophy? I believe they do.

If you're actively looking for new, quirky behaviors on hardware; I respect the hell out of you and would love to use your work and help you spot bugs.

In that case, what would I need to do to help narrow down the exact behavior of the NES OAM corruption bug that I discovered during the development of LJ65?

[Near]

To what extent do you believe the MAME people agree with this philosophy? I believe they do.

I think some of their emulation code follows the philosophy (like MooglyGuy's N64 coverage emulation), and some is the complete opposite. I was told that MESS for instance doesn't have the ability to cycle step opcodes, they are waiting on core changes.

I also see them as a "do whatever it takes to get the game playing, and then worry about accuracy when someone comes along and wants to improve it" group. Which is not at all a bad thing when there is no other emulator available for said games.

They're awesome people, though. Every last one I've ever talked to has been extremely talented and very nice to me. When I mention something, they fix it. Compare to another group that I've been reporting the same bug for the last eleven years now to. I think they're just stretched too thin. They want accuracy but it's like 20 guys emulating 800+ hardware devices and it just doesn't always work out very well.

In that case, what would I need to do to help narrow down the exact behavior of the NES OAM corruption bug that I discovered during the development of LJ65?

Make a list of every possible underlying behavioral cause that can do what you observed. Then start testing them one by one and rule things out. You'll probably discover other quirks as you go about trying to fully understand that one. When that happens, it's your call whether to chase the new one or stay on target. I'd say do whichever is easier. Once you finally have it down, make a test ROM+source listing+documentation that all emu authors can use to fix it in their programs.

Sometimes I find it a huge shame that we can't all work together, but we're all on vastly different skill levels with very different programming styles and ideas. It just sucks that Nestopia gets some things right while Nintendulator gets other things right. It'd be awesome to have one program that gets it all right.

[Zepper]

- My emulator or personal opinion about emulator A or B isn't overlayed in this discussion. As far as I can improve/fix/whatever my emulator and it passes through all test ROMs and picky games, that's enough. About the GUI, don't forget zSNES: it's for Windows (too) and has no Win32 menu style and it's still popular, but "popular" isn't the key. I mean "how could you define accuracy for a NES emulator".

[Banshaku]

Accuracy is hard to define. It's all based on the user group you will talk to.

Like mentioned above, for a game player, they mix compatibility with accuracy so you cannot really rely on that group for the answer. For them, even nesticle is accurate as long games are playing well inside.

From a game developer point of view, accuracy of an emulator is when it can support the same little quirk of the hardware but the problem is: we don't know all of them so how do you define accuracy in that case?

If I would be harsh, I could say for example that all emulators are not accurate since they don't support the same behavior of real mappers when the game fail to initialize it properly. In a commercial game, they already went through the harsh time of failing to do it properly but when you're an homebrew programmer, you can only rely on emulators for testing your results so when it works on all emulators and doesn't on hardware, you can be searching for a while about that mistake. I had experience with the CHR bank that are all set to 0 on real hardware but not emulators and now I have another bug with PRG banks that works fine on all emulator but gives me a grey screen on the real think. But is it the only factor to say about how accurate an emulator is? Not really.

So this is hard to define clearly what accuracy means but if it react like the real hardware, all quirk included, then it's getting accurate.

As for my bug, I think I know the source. I recently moved some code around and I putted by accident the mm3 reset code in another bank ($C000 range) . Emulators didn't cringe about it since they put the bank in reset state in always a specific position but the nes didn't like it, must have jumped to some non existent code. So basically, for mapper behavior, they are very compatible for games but not accurate yet in the reset state.

Edit:

Hmm... Moving the code didn't fix it. Maybe my devcart is failing then. I must exclude that possible cause first but my rom burner is so slow that 1 test every 5 min (!) to figure out the cause is not very tempting..

[tokumaru]

- My emulator or personal opinion about emulator A or B isn't overlayed in this discussion.

Sorry if I got the wrong idea, but you made this thread instants after the 2 emulators were mentioned as accurate in another thread. Since you have complained in the past that those emulators are always mentioned, I might have jumped to conclusions. I'm truly sorry if all you meant to do was get an accurate ( ) definition of what emulator accuracy is, without bringing any actual emulators to the discussion.

As far as I can improve/fix/whatever my emulator and it passes through all test ROMs and picky games, that's enough.

Enough to qualify it as a nice toy, but not as a good development tool.

So you are only interested in behavior other people have discovered and made test ROMs for? Don't get me wrong, blargg is really good at reverse engineering, but I doubt he'll ever cover everything about the NES. If you are really aiming for accuracy you should consider doing some RE of your own.

If you are only interested in running existing games perfectly and passing all existing tests, it seems like you don't care about making the emulator actually accurate, you just want a certificate saying it is so. It's like taking classes in college just to get the best score in all tests and a diploma, rather than actually absorbing the knowledge and using it in the real world (and unfortunately this does happen a lot!).

About the GUI, don't forget zSNES

Personally, I don't like its UI that much either. It's slightly more responsive and better designed than the one in RockNES, but still a bit annoying. I've actually switched to bsnes as my primary SNES emulator, I like it as much as I like Nestopia for the NES.

The UI is the least important thing though, IMO. If an emulator has other features to compensate for it (accuracy, debugging tools, etc), I'll take whatever UI it has to offer.