NSF Note Detection?

[Mr. Podunkian]

First, some full disclosure and background -- a few years ago, I wrote a Rockband/Guitar Hero/NES Emulator hybrid called 8-BITar Hero as part of my university digital art coursework. It ran alongside a modified version of FCEUX and generated note tracks in real time by monitoring 4003/4007/400B (having had no real experience with NES development, and using NESSOUND.txt and the FCEUX source as my only real references, it was the best I could come up with). Needless to say, given the timeframe I had to work in, in addition to having to deal with technology I really hadn't worked with, I came up with something that looked nice on the outside, but sot so much on the inside

So, for the past week or so I've been revisiting the idea -- this time implementing the audio side of things as an NSF player, so that the game could be played stand-alone, without having to have another computer/player "driving" the emulator.

Now...

I've got myself to roughly the same point I was before -- namely, the code checks for writes to 4003/4007. This seems to work fine for some games, but on others I'm noticing some notes aren't showing up. Some examples I can think of off the top of my head -- the first track to "Metal Gear", the first stage music for "Castlevania 1", first stage music for "Yo Noid!"

Having only a cursory knowledge of how the NES works, and going purely off of what I've read in the nesdev wiki about the APU, my two guesses would be that there's something going on with looped envelopes, or that there's some writing to 4002/4006 (which I'm not really doing anything with at the moment). Or maybe there's some huge chunk of the puzzle I'm missing?

Another weird quirk I've noticed happens in the "Bionic Commando" area 1 music -- there's a few sustained notes that constantly write to 4003/4007. At first I thought this was an error on my part, but listening to the individual channels isolated, I can audibly hear some clicking, which makes me think it's just how the music's written?

Any information on this, or any insight into how I could better detect the start of a note would be much appreciated -- again, I've got a very superficial knowledge of how the NES works, so any and all information, no matter how obvious it is will be useful for me.

[Shiru]

As the APU doesn't operate with notes, you have to check ranges of possible period values. It will give detection errors with strong vibrato, though.

[thefox]

Having only a cursory knowledge of how the NES works, and going purely off of what I've read in the nesdev wiki about the APU, my two guesses would be that there's something going on with looped envelopes, or that there's some writing to 4002/4006 (which I'm not really doing anything with at the moment). Or maybe there's some huge chunk of the puzzle I'm missing?

I take it you're more trying to detect when the notes start, instead of detecting the frequency of the notes? Unfortunately there's no one reliable way to do this. Some games will write to $4003 when a new note starts, yes (because writing that register also resets the phase of the pulse channel), but some may just use the constant volume control in $4000. Or the volume envelope. After all, the change in the volume is the thing that mostly defines what we perceive as separate notes. So you're going to need some heuristics.

If you want to detect the actual note value in a better/more linear way, take the 11-bit timer value ($4002 and $4003), convert that to a frequency f = CPU / (16 * (t + 1)) (where CPU is the CPU frequency), and convert the frequency to a note, knowing that f = 2^(note/12) (this places note #0 at 1Hz, might want to adjust that).

Another weird quirk I've noticed happens in the "Bionic Commando" area 1 music -- there's a few sustained notes that constantly write to 4003/4007. At first I thought this was an error on my part, but listening to the individual channels isolated, I can audibly hear some clicking, which makes me think it's just how the music's written?

Yeah could very well be that it's just written that way.

[Mr. Podunkian]

I take it you're more trying to detect when the notes start, instead of detecting the frequency of the notes? Unfortunately there's no one reliable way to do this. Some games will write to $4003 when a new note starts, yes (because writing that register also resets the phase of the pulse channel), but some may just use the constant volume control in $4000. Or the volume envelope. After all, the change in the volume is the thing that mostly defines what we perceive as separate notes. So you're going to need some heuristics.

That's right -- the big hurdle now is trying to figure out where the notes start. I hadn't thought of constant volume control -- thanks for the heads up!

If you want to detect the actual note value in a better/more linear way, take the 11-bit timer value ($4002 and $4003), convert that to a frequency f = CPU / (16 * (t + 1)) (where CPU is the CPU frequency), and convert the frequency to a note, knowing that f = 2^(note/12) (this places note #0 at 1Hz, might want to adjust that).

Yeah, did that exact thing today. The old version literally just compared the period value to the one before it -- the new version takes the frequency based on that equation, and compares in relative halfsteps -- I'll play around with it more the more I work this out.

Also, whereabouts on IRC do you hang out? I remember you joined my channel on EsperNet when we were talking about the STREEMERZ NES port -- I tried #NESDEV on EsperNet but there was nothing there?

[Shiru]

I think that detection of relatively big change (halftone) in the period could be used as an additional factor to detect a new note.

[thefox]

Also, whereabouts on IRC do you hang out? I remember you joined my channel on EsperNet when we were talking about the STREEMERZ NES port -- I tried #NESDEV on EsperNet but there was nothing there?

I'm thefox@EFNet, fo@EsperNet, fo@IRCNet and thefox@QuakeNet

#nesdev is on EFNet.

[rainwarrior]

As said above, detecting a note onset is a sticky problem. I will try to summarize.

1. Your attempt to use $4003/4007 will only work for a small subset of games that primarily use envelopes to trigger their notes. The majority of games don't do this.

2. Most games will disable envelopes and write $4003/4007 only when these high bits change. This is done of course to prevent unnecessary phase reset clicks. As a result, though, there is no one register that you can expect to be written at the start of a note for any particular game. The "note" concept is higher up in the particular music engine, what makes it through to the APU registers will no longer be a 1:1 representation of that concept.

3. Bionic Commando plays repeated notes on the same pitch, and does reset them with $4003/4007. It's extremely atypical for a game to do this, but yes that was correct.

So... unfortunately if you want to do this well you will have to tweak your detection technique on a game-by-game basis. You can try to detect frequency changes that cross a pitch boundary (as stated, vibrato is a problem), you can try to detect sudden jumps in volume (repeated notes that don't decay too much may be a problem), etc. It's kind of an arbitrary call that needs to be made to decide what's really a new note and what isn't.

An alternative you may consider is my NSF Import tool for Famitracker. You can use this to recreate accurate Famitracker versions of the tunes you want by hand (somewhat time consuming, but it's easy methodical work), and in doing this you will manually be deciding what is or isn't a note. At that point you can use the Famitracker NSF driver and tap into its high level concept of a note.

[Mr. Podunkian]

Got it -- I'll be seeing how much mileage I can get from setting up parameters for frequency and volume changes. Following a chat with thefox, I cleared up some misunderstandings I had about external sound chips as well, which gave me some relief.

I don't know that going the NSF Import/Famitracker route is viable, as I want to eliminate as many intermediate steps between NSF to gameplay level, and I do like the idea of just plugging in any old NSF file you have lying around and having it just work. Of course, the extent to which I can get it to "just work" is yet to be seen, haha.

What I'm hoping to achieve here is a solution that'll work in the majority of cases, even if it doesn't work in all of them.