audio buffering and timing discrepancies

[miker00lz]

Hi all, I haven't been on here in a while but my emulator is pretty decent these days... however, there is one thing that has been bugging the crap out of me since I started writing it. There are always going to be small timing discrepancies regarding the sample rate, the hardware requesting more audo data, and the actual rate at which emulator code is generating samples.

It it very, very close but it's still off by enough that it can cause buffer underruns now and then, or the emulator has filled the buffer before the hardware wants it all. What is the best way to handle this issue without having audible hiccups occasionally? I've played around with real-time stretching but it isn't a real good solution, that itself introduces some issues.

What do you guys all do to handle all of this type of thing? The best solution I've come up with so far is to create my emulator's buffer to be double the size of what I initalize for the buffer sample size when kicking off SDL's audio stuff. Then when SDL requests more data, I pull from the bottom of my buffer and memmove everything past that point down to the start of my buffer. Then I subtract the amount SDL requested from my current buffer position. This provides a nice smooth output, but still now and then there will be some hiccups that are audible or samples may have to be dropped at some point if I overflow my buffer.

I also do not do the initial SDL_PauseAudio(0) until the first time the buffer fills completely. If I don't do that, it's a lot more likely to not buffer in time.

[ooPo]

If you really really want perfect audio, you can use the audio callback (does SDL use a callback?) as your timer:

Code: Select all

run_a_frame();
start_audio();

....

void callback( ... )
{
  write_audio_buffer();
  run_a_frame();
}

The main problem you'd have then would be dropped video frames, or tearing if you disabled vsync.

You're on the right track with the buffering you're doing. I do the same. You may want to look into a ring buffer to avoid the memmove/memcpy. When I detect a buffer underrun is happening soon I render two frames at once to stuff a bit extra in the buffer.

[James]

I've played around with real-time stretching but it isn't a real good solution, that itself introduces some issues.

This is what I do, but I do it in a Windows-specific way. Every couple of frames, I query the position of the primary output buffer's pointers. If audio is lagging behind, I increase the playback frequency, if it's playing too fast, I decrease the playback frequency.

It works well; I don't notice any adjustments and video frame rate stays locked to vsync.

One thing to watch out for: processor throttling (i.e., power saving stuff) needs to be disabled for this to work reliably.

[MottZilla]

I've heard of both using the audio buffer for frame throttling as a solution as well as adjusting the actual number of samples generated for the buffer to better match VSYNC. I think the actual NES runs slightly faster than 60hz which causes a problem if you generate the correct number of samples and sync the emulator with 60hz vsync. It depends on how you generate your audio but you may be able to do the adjustment when generating your audio data to make sure the number of total samples is equal to how many you need for each video frame.

[koitsu]

Tying to vsync also has a very cute side-effect on Windows XP (and due to the design, I imagine Windows Vista and 7 as well; do not care about 8): if the render surface is used in windowed mode, and that surface/area is covered completely by another window, vsync is no longer honoured (at the driver level -- at least by nVidia drivers). For example, run Nestopia in windowed mode, make sure Timing -> Vsync is checked as well as Preferences -> Run in background. Load a game with music, then cover the window entirely with another (make sure every pixel is covered -- if even one pixel on the surface is visible vsync applies). Music running at hyperspeed, whee!

I cannot recommend that people adjust playback frequency rate, for the exact reasons James mentioned -- CPU "throttling" (referring to CPU power-saving states (Cx) as well as things like EIST/clock frequency) is here to stay, it's not going away. It's especially important on laptops. Plus to those of us with good ears, playback frequency adjustment sounds like utter shit.

Isn't there some way in Windows to tie in to a hardware timer interrupt (such as ACPI timers and HPET) reliably? I see mentions in Microsoft's docs about something called SetTimer() and that looks relevant. Just set up one of those set for a constant rate (60Hz) and key everything off of that. It looks like SetTimer() takes an interval in milliseconds.

[James]

Plus to those of us with good ears, playback frequency adjustment sounds like utter shit.

Have you tried nemulator? I think I have pretty good ears and I'm really happy with the results. Not disputing your opinion; I'm just genuinely interested in feedback.

Isn't there some way in Windows to tie in to a hardware timer interrupt

Presumably from a driver, but I don't know how you'd guarantee scheduling of a user-mode process.

[koitsu]

Plus to those of us with good ears, playback frequency adjustment sounds like utter shit.

Have you tried nemulator? I think I have pretty good ears and I'm really happy with the results. Not disputing your opinion; I'm just genuinely interested in feedback.

I haven't, and I can't/won't, quote: "nemulator uses DirectX 10, therefore Windows Vista or Windows 7 is required". Sorry, I don't do either of those OSes. I use XP exclusively for reasons that aren't your fault/concern for the most part, so that's my own business going on there.

Only have familiarity with Nestopia, VirtuaNES, FCEUX DSP, and Nintendulator. Have tried a few miscellaneous others but they aren't "up to par" as those 4.

Isn't there some way in Windows to tie in to a hardware timer interrupt

Presumably from a driver, but I don't know how you'd guarantee scheduling of a user-mode process.

SetTimer() is user-mode (according to official MSDN documentation).

I just fired off a message to a colleague of mine who writes both userland processes on Win32 as well as kernel drivers, asking him exactly how on Windows you tie something to, say, 60Hz. His response was: you use QueryPerformanceCounter(). SetTimer() has some kind of variance, something like 30-200ms, which is often too much.

[miker00lz]

Tying to vsync also has a very cute side-effect on Windows XP (and due to the design, I imagine Windows Vista and 7 as well; do not care about 8): if the render surface is used in windowed mode, and that surface/area is covered completely by another window, vsync is no longer honoured (at the driver level -- at least by nVidia drivers). For example, run Nestopia in windowed mode, make sure Timing -> Vsync is checked as well as Preferences -> Run in background. Load a game with music, then cover the window entirely with another (make sure every pixel is covered -- if even one pixel on the surface is visible vsync applies). Music running at hyperspeed, whee!

I cannot recommend that people adjust playback frequency rate, for the exact reasons James mentioned -- CPU "throttling" (referring to CPU power-saving states (Cx) as well as things like EIST/clock frequency) is here to stay, it's not going away. It's especially important on laptops. Plus to those of us with good ears, playback frequency adjustment sounds like utter shit.

Isn't there some way in Windows to tie in to a hardware timer interrupt (such as ACPI timers and HPET) reliably? I see mentions in Microsoft's docs about something called SetTimer() and that looks relevant. Just set up one of those set for a constant rate (60Hz) and key everything off of that. It looks like SetTimer() takes an interval in milliseconds.

I don't believe there is a way to get callbacks directly from the hardware timers in Windows in a user mode application. I tinkered with SetTimer a long time ago and it is very inaccurate unforunately. I believe it's the exact interface used in the old VB6 for the timer objects.

[miker00lz]

I just fired off a message to a colleague of mine who writes both userland processes on Win32 as well as kernel drivers, asking him exactly how on Windows you tie something to, say, 60Hz. His response was: you use QueryPerformanceCounter(). SetTimer() has some kind of variance, something like 30-200ms, which is often too much.

Yeah, I use QueryPerformanceCounter (or gettimeofday if __WIN32__ isn't defined, which is a similar function in *NIX/Linux which is accurate to 1000000 ticks/sec) a whole lot in another emulator of mine (a PC emu, http://sourceforge.net/projects/fake86 ) for timing but there are still going to be issues with audio generation because the audio hardware callback is based on the sound card's internal timer while it's playing the sound stream and they're rarely if ever perfectly accurate.

This whole issue can be very tricky, I guess I should just be satisfied with "good enough" but I will play with it a bit more still because all, the good emulators out there have no audible timing issues so it can certainly be done. Maybe I'll look at the code of some of them for pointers.

[miker00lz]

Thanks for the advice everybody. I'll see what I can do, maybe I'll post some code later on if anybody wants a closer look. Maybe I'm doing something stupid that should be obvious to me, but I'm missing.

[natt]

Bizhawk has a throttle to audio mode, where you block in the audio thread if there's not enough room for all of the samples, and a wizard mode, where various black magics are used to attempt to stretch\squeeze audio without sounding horrid. (To be used with other throttles like clock-based or vsync-based).

Honestly, the whole thing is horrible and you'll never please everyone, even if your emulator is bristling with options.

[miker00lz]

Okay, I think I've some something that is working pretty well. If anybody wants to try it on their system and let me know if the audio remains smooth, that would be awesome.

Here is a download, source code included. Pre-compiled executable in MoarNES\Release\

http://rubbermallet.org/moarnes-0.12.11 ... source.zip

I still have glitches in a number of games, but most should be reasonably playable. Some are perfect. (Mega Man 2 is perfect for example, as far as I can tell) I also still have some issues with my audio sweep code, so that can sound odd sometimes like when you go down a pipe in SMB1.

Edit: btw, there are four quicksave slots. F1 to F4 saves, F5 to F8 loads.

[koitsu]

The audio remains "smooth" for me (Windows XP Pro SP3), the only time I can get it to hiccup is when using GUI features or when it loses/regains focus.

No offense intended in the least by this statement, but I can hear and detect all *sorts* of issues in this emulator, some relating to sound emulation, others relating to simple usage -- including some audio issues in Megaman 2 (I can hear 3 consistent problems there). I also got the emulator to flat out crash (full application crash) when loading a common/popular ROM (i.e. not some crazy homebrew or Chinese game).

Also, one of the sound issues I've heard/detected pertains to overall playback rate being at the wrong base frequency. It's quite obvious that you're correlating the "base frequency" rate of the NES audio system, somehow, with the actual frequency/sample rate of a soundcard. I can tell because if I drop from 96000Hz to 48000Hz, the actual "base frequency" of the audio drops, and then the emulator craps itself with sound buffer overflows. :-) However, even at 96000Hz, the base frequency for the NES audio in general is still wrong (doesn't require a good ear either) -- it's way too high. Go load up a Megaman 2 NSF in an NSF player (Nestopia, NSFplug, etc.) and compare. :-)

I should warn you about correlating those two things, especially if the design is "use the highest frequency the soundcard advertises": many soundcards -- including built-in audio ICs as well as physical HBAs and even USB DACs -- will claim high rates but then behave oddly when those rates are used. I remember emulator authors 10 years ago talking about how on some cheap sound cards which advertised 48000Hz, for example, the audio would "freak out" (sound wrong, buffer oddities, some intermittent latency, etc.) yet 44100Hz worked fine. So I would be wary about this. You'd be better off picking something like 44100Hz, which is something everything should work with.

Where would you like feedback to be provided about these issues, or do you want folks to hold off until you're ready for bug reports? I'm cool either way.

Consider me impressed so far, and I'm very happy to see someone using SDL to boot. :-) Yay!

[kyuusaku]

"Underclock" the rest of the system to suit your sample rate.

Say 112 pixels / (1/48000) = 5376000 Hz = 0.12% error and everything will be in sync.

[Mednafen]

Using gettimeofday() for emulator timing purposes is probably a bad idea unless your code can handle time jumping backwards or forwards by a non-trivial amount without too much disruption.

clock_gettime() with CLOCK_MONOTONIC is a better choice on modern UN*Xes, though Linux has done somewhat weird and questionable things with it(and IIRC there used to be some Linux kernel bug(s) that would cause the time to jump backwards a small amount sometimes):

Code: Select all

       CLOCK_MONOTONIC
              Clock  that  cannot  be  set  and  represents  monotonic  time since some unspecified starting point.  This clock is not
              affected by discontinuous jumps in the system time (e.g., if the system administrator manually changes the  clock),  but
              is affected by the incremental adjustments performed by adjtime(3) and NTP.

       CLOCK_MONOTONIC_RAW (since Linux 2.6.28; Linux-specific)
              Similar  to  CLOCK_MONOTONIC, but provides access to a raw hardware-based time that is not subject to NTP adjustments or
              the incremental adjustments performed by adjtime(3).