FMV on the NES ... kind of

[SriK]

Hi everyone! Recently I've been working on a PC game tentatively titled Steel Assault, which aims to emulate a late NES graphic style (inspired by games such as Shatterhand and Batman where heavy dithering and details in negative space are used to create a dark atmosphere within system limitations). The game isn't adhering strictly to limitations (two of my assumptions are infinite cartridge space and expanded sprite limits lol), but I'd like for most of it to be plausible on the NES in some limited form or another. Anyway, one of my ideas for the game is to have a short (60 seconds or so) FMV intro at the start, detailing the game's scenario and background. Searching these forums, it seems like not many people have tackled the problem of FMV on the NES (aside from the Bad Apple demo, which is super cool), so I decided to try my hand at it this past week, and also learn more about NES coding (I have a bit of homebrew experience on 16-bit consoles, but this was my first major foray into 8-bit). My approach was inspired by this post by tokumaru, though I haven't implemented all of the ideas he suggested yet.

Well, it works! It's kind of stuttery at times, though (especially when there are large changes from frame to frame), and the pixelation/lack of double buffering isn't helping (more on this later). Here are some screenshots:

The videos are clips from 500 Days of Summer, Goodfellas, and Blade Runner respectively. The .nes files are attached; they look best in Nintendulator, but work on FCEUX and Nestopia as well. (My favorite is the Goodfellas clip, which plays part of the movie's famous Copacabana tracking shot.) Here are some statistics on the clips used:

• goodfellas - 51 seconds; 441,086 bytes of video
• 500ds - 74 seconds; 482,548 bytes of video
• bladeRunner - 48 seconds; 452,686 bytes of video

Note that the ROMs crash after the clip ends lol, because I haven't bothered putting in video length values yet. Also, there isn't any music, and I doubt there's currently enough rendering time left to play any (but I haven't tried yet).

The main bottleneck seems to be that the compressed video format is bit-aligned rather than byte-aligned.* I've implemented various techniques to mitigate the effect of this, but I still can't get it fast enough to consistently render all (15FPS video) frames in under 4 NES frames; in fact, some take as many as 6 to 8. In addition, given the approach I'm taking, double buffering wastes another frame or more when turned on (as it has to copy 2048 bytes from one RAM bank to another), so video is only single buffered. I'm going to try and come up with a byte-aligned format for interframes later and see if I can get better speed while still maintaining a decent compression ratio (right now it's about 25-30%, compared to an uncompressed video which uses 2KB for each frame).

But yeah, I hope you enjoy the demos! And if anyone's interested, I can post more about the compression format and how rendering works later. (Or, you can try and find out yourself... :D)

Sri




* This decision was partially because, until I actually implemented decompression on the NES itself, I was using Huffman encoding on the delta values and naively thought I would be able to decode them in realtime.

[Kasumi]

Heh. I knew "Steel Assault" sounded familiar.

Anyway, this stuff is really, really cool. And I'd certainly love to read about your video compression/implementation. I was happy enough just to get 53% on text, heh.

[thefox]

Cool stuff. Steel Assault seems nice too, despite not being an actual NES title.

Some day I want to try to do hardware accelerated FMV on PowerPak.

[ccovell]

Pretty cool to see some FMV going on for that long.

I did some NES experiments with animation/FMV back in 2000, but these had no compression and used no mapper:



http://www.chrismcovell.com/data/anims.zip (2 animation ROMs in archive)

[tepples]

Have you had time to look over my reverse engineering of Bad Apple's video format? Only the bitmap of what areas have changed is bit-based; the actual nametable values are byte-based.

Anyway, assuming infinite cartridge space isn't really a cheat. Pretend you have a 128 kilobyte RAM and an eMMC (NAND flash) that you can read through a separate mapper port. Someone on this board has proposed exactly this configuration.

[BMF54123]

What would be really cool is something like the MSU-1 for the PowerPak and/or Everdrive N8. Perhaps something with the same basic capabilities as an FME-7 or VRC6, but with the ability to stream a PCM sound file to the sound expansion pin, and a PRG/CHR data file to either a mapper register or directly to PRG/CHR RAM. Surely both cartridges have enough hardware to accomplish this.

[SriK]

Thanks, guys!

Anyway, this stuff is really, really cool. And I'd certainly love to read about your video compression/implementation. I was happy enough just to get 53% on text, heh.

The compression is actually extremely simple. The first frame is an intraframe (actual image data), encoded as follows:

Code: Select all

1  = repeat current value for 3 bytes (12 pixels)
10 = repeat current value for 1 byte (4 pixels)
11 = change current value to following value (next 8 bits)

And all the other frames are interframes (delta values), encoded as follows:

Code: Select all

1  = no change for 3 bytes (12 pixels)
10 = no change for 1 byte (4 pixels)
11 = add following value to current value (next 8 bits)

There's some other stuff about lossy encoding of delta values on the encoder side, but that's all the decompressor needs to do.

The implementation is a little bit trickier. Rendering is done using a program I wrote to autogenerate a giant unrolled loop, which executes MMC5 ExRAM nametable changes until HBlank then sets the scroll appropriately (writing to the nametable as it's rendering so that new rows of the image are transferred just as the old ones finish). Part of the image is also transferred beforehand, at the start of every new video frame, to help allow a bit of leeway. Decompression is relatively straightforward in comparison, though still fairly optimized (I think about as much as it can be without changing the format, or relying on self-modifying code stored in RAM).

If you or anyone else is interested, I can clean up the encoder/ASM generation programs as well as the asm6 source and post them in a bit. I don't know how much more I'm going to work on this myself, it was just a quick proof of concept for fun/out of curiosity, but maybe others can expand on it and make it more viable.

I did some NES experiments with animation/FMV back in 2000, but these had no compression and used no mapper.

Cool stuff! Especially awesome given that it was almost 15 years ago, I'm sure the state of NES emulation was far worse back then and there were far more unknowns. The mapper is kind of key to how this works as I mentioned above, so I'm not sure this same technique could be used for FMV off the MMC5 sadly. Regular nametable writes just seem too slow...

Have you had time to look over my reverse engineering of Bad Apple's video format? Only the bitmap of what areas have changed is bit-based; the actual nametable values are byte-based.

LOL, can't believe I didn't think of that! That's awesome, I'm sure something similar could be used for 4-color video too (could possibly even save space?)

Anyway, assuming infinite cartridge space isn't really a cheat. Pretend you have a 128 kilobyte RAM and an eMMC (NAND flash) that you can read through a separate mapper port. Someone on this board has proposed exactly this configuration.

So, theoretically, I could do things like e.g. render large bosses on arbitrary backgrounds and just use the excuse of "lol CHR ROM"? (The interior of the boss would be tiles and the exterior would be sprites, in this hypothetical scenario.) Here's a link to the game's current style guide BTW.

What would be really cool is something like the MSU-1 for the PowerPak and/or Everdrive N8. Perhaps something with the same basic capabilities as an FME-7 or VRC6, but with the ability to stream a PCM sound file to the sound expansion pin, and a PRG/CHR data file to either a mapper register or directly to PRG/CHR RAM. Surely both cartridges have enough hardware to accomplish this.

That would be awesome. Compressing audio could also save more space here, I did some experiments with ADPCM on an 8-bit processor (the Z80 in the Sega Genesis) a few years back and it actually got to a pretty advanced stage. I'm sure similar techniques could be used on the NES (especially if video "decompression" just consists of switching CHR banks).

Also, completely unrelated, are you the same BMF who made the Sonic 2 Genocide City mockup many years ago? lol

[lidnariq]

So, theoretically, I could do things like e.g. render large bosses on arbitrary backgrounds and just use the excuse of "lol CHR ROM"?

CHR RAM and slow memory; it's more like a Sega CD (or PC Engine, or maybe even the PS1). Still has attribute clash. Still bandwidth limited. You can only really upload about 220 bytes per vsync without reducing the vertical height, or extended blanking gets you enough bandwidth to completely redraw the nametables at a visible height of ≈192 scanlines. Updating tiles is even worse; you could only upload the same amount of data as you consumed with a visible height of ≈40 scanlines.

[tokumaru]

Here's a link to the game's current style guide BTW.

I have to say this: the technical justification for the unlimited number of sprite palettes is pretty bogus. All graphical expansions provided by the MMC5 have to do with carefully timed switching of memory external to the PPU.

Greater attribute resolution is possible only because the PPU fetches attribute bits every 8 pixels. Normally, the same bits are fetched twice in a row, but the MMC5 is able sync with this fetch pattern and provide unique data for each fetch. You could even make a mapper with more attribute resolution than the MMC5, and have each row of each tile use a different palette.

The simultaneous access to 16384 tiles is also only possible because both the name table memory and the pattern memory are external to the CPU, and the MMC5 can tell WHAT is being accessed WHEN and manipulate the data it feeds the PPU accordingly. The same goes for the 512 patterns for sprites.

Palettes, however, are internal to the PPU, and even if the mapper was able to tell when they're being accessed, there's no way to interfere with that data. Sprite OAM is also internal to the PPU, so you can't give a purpose to unused attribute bits. Unfortunately you cannot pin this on "expansion chips", unless the chip you're talking about is an entirely new PPU, which has to be installed inside the NES itself instead of the cartridge.

There are a lot of things that can't possibly be changed by mappers: maximum of 8 sprites per scanline, maximum of 64 sprites per frame, the amount of colors per palette, the number of palettes, the 50-something colors master palette, the sprite clipping that happens at the top and left edges of the screen, and probably others I can't think of right now. All of these have to do with things that happen only inside the PPU and are not exposed or accessible to the outside in any way.

[tepples]

Updating tiles is even worse; you could only upload the same amount of data as you consumed with a visible height of ≈40 scanlines.

A few mappers did combine CHR RAM with bank switching. Pinbot, RacerMate, and Videomation come to mind. So if your fictional cartridge has 128K PRG RAM, 32K CHR RAM bankable in 1K units, and a gigabyte of NAND flash, that could be close enough to unlimited CHR ROM.