New NMI while still in NMI

[Vectrex2809]

I'm currently programming a game that executes a set of complex routines while playing, and it seems like NMIs are starting to occur while I'm executing my game logic (while I'm still in the NMI because I do everything in NMI), even though this didn't really happen to me before (I never coded such a big project before...)
Is that possible? When it happens, my stack starts rising like crazy (Probably because the NMI trips during a subroutine or something). Is there a way of fixing this without having to reprogram the whole game? (If it really is that problem).

[Joe]

Is that possible?

Yes.

Is there a way of fixing this without having to reprogram the whole game?

Yes.

One option is to create a new variable to track if the NMI handler is currently running. At the beginning of your NMI handler, check if another NMI handler is currently running, and if so, immediately return.

Another way to handle this is to disable NMI generation at the start of your NMI handler, and enable it afterward. Be sure to read PPUSTATUS ($2002) immediately before enabling NMI, to prevent triggering a NMI partway through vblank. There is a slight risk of causing a graphical glitch.

[Vectrex2809]

Is that possible?

Yes.

Is there a way of fixing this without having to reprogram the whole game?

Yes.

One option is to create a new variable to track if the NMI handler is currently running. At the beginning of your NMI handler, check if another NMI handler is currently running, and if so, immediately return.

Another way to handle this is to disable NMI generation at the start of your NMI handler, and enable it afterward. Be sure to read PPUSTATUS ($2002) immediately before enabling NMI, to prevent triggering a NMI partway through vblank. There is a slight risk of causing a graphical glitch.

Fixed it with the second one!!! Thanks a lot man. I had the same idea of disabling NMI but I didn't know about PPUSTATUS, which made graphical glitches happen.

[tokumaru]

Fixed it with the second one!!!

IMO, the first one is a better fix.

I had the same idea of disabling NMI but I didn't know about PPUSTATUS, which made graphical glitches happen.

Reading PPUSTATUS doesn't avoid glitches directly, it just prevents NMIs firing in the middle of VBlank, which results in your PPU update code not having as much time to run as it should, and when PPU updates spill over the visible frame, there are glitches.

The other glitch Joe mentioned is one that affects SMB (most notably, but also other games), which is caused by modifying $2000 mid frame. If the write happens at a certain time (when the PPU is setting up the scroll for the next scanline, I believe), it may cause the scroll to look wrong for 1 scanline. You're taking this risk if you're using the 2nd fix.

Someone even hacked SMB to use the first fix instead of the second to completely get rid of that glichy scanline that appeared from time to time.

[psycopathicteen]

I don't know if this applies to the NES, but sometimes on the SNES, it goes into NMI late causing tearing at the top of the screen. Any way to prevent this?

[Joe]

Yes.

Be sure to read PPUSTATUS ($2002) immediately before enabling NMI, to prevent triggering a NMI partway through vblank.

(Basically copied from the wiki.)

[tokumaru]

I don't know if this applies to the NES, but sometimes on the SNES, it goes into NMI late causing tearing at the top of the screen.

On the NES this only happens if you enable NMIs in the middle of VBlank (an NMI will fire immediately). The side effect is not tearing though, but corrupted VRAM data (from trying to write to VRAM with while the PPU is rendering) and wrong scroll settings (from not having set the scroll before rendering started).

Any way to prevent this?

On the NES you just need to read PPUSTATUS (to clear the VBlank flag) before enabling NMIs. No clue about the SNES.

[psycopathicteen]

On the SNES, it looks like reading $4210 during vblank prevents this glitch.

[Bisqwit]

On the NES, there is no such thing as an NMI that is "over". NMI is a trigger-type event, and it is triggered at predictable intervals. Whatever occurs after the trigger is up to your program.
Furthermore, RTI is not a special instruction that signals any device that any interrupt or the such is over. Rather, it is a simple CPU instruction that pops the PC and P from stack. You may use the RTI instruction whenever you need that outcome, as many times as you want, regardless of when interrupts happen.