CHR and Indexed PNG Tools

[segaloco]

Just wanted to share an improvement on my CHR conversion utilities. I decided to drop any further investment in my BMP tools since a PNG can natively be 2bpp indexed color, opting to write a new version of my utility with libpng.

Along the way, I realized several of my cases, single tile, tile bank, BG mapping, are all variations on take a BLOB of CHR tiles and arrange them in a grid, sometimes the grid is 1x1, sometimes its 256x256 for a bank, sometimes its some other set of configurable criteria.

As such, I put together "chrtopng" the source of which is here: https://gitlab.com/segaloco/misc/-/blob ... chrtopng.c

Included is a manpage but some of the applications may not be readily apparent just from the documentation. Essentially there are three configurable values as well as the input of the tile data itself. By default, the utility will process up to 256 tiles into a PNG image 16-tiles wide, resulting in a 16x16 tile (128x128 pixel) PNG on the standard output. The PNG uses a palette and bit depth of 2bpp for simplicity. The palette is by default assembled from an included set of PPU color IDs, although the "-p" option can be used to provide a four-byte file containing the palette IDs to use instead. This points to one limitation, a single pass only applies a single palette, this tool can't currently assemble an image with variations in palette IDs for different tiles. I don't think I'll add that to this tool, instead if that is desired, one would produce all the PNGs of the desired tiles at the desired colors and then some separate utility could sample those into an image from a mapping.

Anywho, the per-row and maximum tile limits are also options, "-r" and "-m" respectively, which accept an integer describing these dimensions. While a file can be passed as an argument, this utility will read the standard input if no such file is supplied. This allows for using this utility as a filter in pipelines. Here are a few application examples, note I've suffixed each pipeline with:

Code: Select all

... | convert png:- -scale 200% png:- | png

As a means of scaling up and displaying the result. This is just for demonstration so I'll omit repeating this in each example. It is however in the screenshots. I'll be using tiles and palettes from Super Mario Bros. for these examples:

So first is the most basic case, reading a single tile supplied on stdin:

Code: Select all

chrtopng <data/chr.smb/chr_bg_0C.bin

This is the background tile for the letter "C", which displays as expected (remember, there is a default palette, in this case with black and red as the first and second entries):

This demonstrates an important point. Despite the default row maximum of 16 tiles, if less than a full row of tiles is provided, the width is adjusted accordingly. This is to avoid a bunch of dead space when looking at just a handful of tiles.

However, providing a BLOB with enough tiles to meet the 256 tile default maximum:

Code: Select all

chrtopng smchar.fc.bin

Yields:

Two new facts are observed here. First, the input file can be passed as an undecorated argument, internally it is simply freopen(3)'d as stdin. Second, the dimension limitations are now in effect. After reaching the 16th tile, the image generation wrapped around. In addition, despite this being the full CHR image, only the OBJ page has printed, demonstrating the 256 tile default maximum in action. While the maximum does not force a smaller number of tiles to this max, it cannot be exceeded. Internally, this value is used to size the buffer used for the operation and as a result, the maximum number of tiles the tool will read. The sizing of the image is then based on how much was *read*, not how much was requested. Really this is just to keep things pipe-friendly. The buffer size can't be as easily determined automatically by a pipe since you can't, for instance, seek to the end of a pipe reliably to tell its length. Additionally, as you can see here, the default built-in palette is shamelessly Mario's colors. This is the bank I worked it up with so that's how it fell, my last iteration of tools was the Dragon Quest player character palette for similar reasons.

Anywho, these just demonstrate the basic lower and upper bounds of operation provided by default, things get more interesting by manipulating these limits.

A quite useful feature of reading stdin is piping in from other data filtering tools like dd(1):

Code: Select all

dd if=smchar.fc.bin bs=16 skip=256 count=10 | chrtopng

In this case, I get:

So basically I was able to use dd(1) to manipulate the BLOB in units of tiles and snip out a subset of tiles to send down the pipe to my tool. As a result, I can snip out the digit tiles from the CHR BLOB simply by knowing they begin the BG segment. Operate in 16 byte (CHR) blocks, discard the first 256 (OBJ), and then take the next 10 (digits). This can prove quite useful in needle-in-haystack situations and could easily be made into a script of its own wrapping this for simple CHR lookups in a bank. What is tile 32 of the BG bank? Skip the first 256 to get to the bank, another 32 to get to the tile in question, and count 1 to pull just that tile back. Pipe it to something like the png CLI and you've got a "show me these tiles" tool with no fuss.

Anywho, the default palette is just so there is something to refer to, but likely one would want to see graphics in context:

Code: Select all

printf "\x0F\x30\x0F\x0F" >pal.bin
dd if=smchar.fc.bin bs=16 skip=256 count=10 | chrtopng -p pal.bin

With this, I supply the palette $0F $30 $0F $0F rather than the built-in, giving a bit more natural of a font appearance:

While the 16 wide and 256 total limits come from one of the most common cases, viewing an entire bank, both limitations are adjustable.

The row width can either be raised:

Code: Select all

chrtopng -r 24 smchar.fc.bin

or lowered:

Code: Select all

dd if=smchar.fc.bin bs=16 skip=256 count=10 | chrtopng -p pal.bin -r 2

The maximum tiles can also be raised or lowered. Raising the value in particular is useful for CHR BLOBs larger than a single bank:

Code: Select all

chrtopng -m 512 smchar.fc.bin

Now with the max set to 512, a 512 tile buffer is reserved and just as many tiles are read off the input until EOF. The other defaults still apply in this case, with the row wrapping applying at 16 tiles and the default palette being applied.

This hopefully demonstrates some of the capabilities. One follow on tool I intend to write is a utility that will take a complete (or partial, but contiguous) nametable map and sample from a CHR collection into a BLOB arranged appropriately such that it can be passed through this tool to get an image of that screen. In reality it's just creating a bank that is arranged such that the desired tiles appear in the output in the right places. With the row and maximum settings, this could be further enhanced to display individually mapped things like sprites and metatiles by ordering them in memory properly and then applying the correct dimensions.

Anywho, licensing is included in the file, typical BSD stuff, do what you want with it, just don't come crying to me if it eats your morning breakfast. I'm lazy so didn't provide a Makefile, this isn't going to replace your favorite graphic editor of the day, it's just a tool primarily targeting bozos like me that type words at black and white screens most of the time and are scared of mice. My compile line is:

Code: Select all

cc -O2 -I/usr/include/libpng16 -o chrtopng chrtopng.c -lpng

Ymmv with local conditions, when in doubt you probably have pkg-config around to dole out this information. I'm planning on this and its reverse pngtochr (still working on it) to serve as the low level pieces of some higher level stuff so when I get there whatever that looks like might be a bit more friendly to usual build niceties.

Finally there is some error checking but naturally don't trust this any more than any other random code you get off the net. Pipes are your friend, then my grubby code doesn't touch your pristine files at all. Anywho as I build up new parts of this stuff I'll share it here.

[segaloco]

And now another quick tool this made desirable, "nttochr" is a simple wrapper over dd(1) that steps over a stream of bytes (nametable) and emits the tiles from a bank of CHR corresponding with those bytes.

https://gitlab.com/segaloco/misc/-/blob ... ls/nttochr

This one is also meant for pipelines. One argument is required, a CHR bank from which to draw tiles. By passing either a second file argument or a file on stdin, the bytes in that file will be taken as IDs of tiles to emit from the CHR bank in sequence. Essentially this prepares a CHR bank expressing the nametable. Of course, this is only accurate when the nametable stream represents an entire 32x20 tile region. However, there are a few useful applications even with partial data.

For instance, the title screen of Doki Doki Panic is made up of several different elements. Among these is a cloud with the Yume Kojo text, the name of the eponymous promotion. This cloud is 11 tiles wide by 5 tiles tall and represented by the display list:

Code: Select all

        .dbyt   $2073
        .byte   (:++)-(:+)
        : .byte $7B, $75, $75, $75, $75, $75, $75, $75, $75, $75, $78
        :

        .dbyt   $2093
        .byte   (:++)-(:+)
        : .byte $7C, $00, $01, $02, $03, $04, $05, $06, $07, $08, $79
        :

        .dbyt   $20B3
        .byte   (:++)-(:+)
        : .byte $7C, $09, $0A, $0B, $FC, $0C, $0D, $0E, $0F, $10, $79
        :

        .dbyt   $20D3
        .byte   (:++)-(:+)
        : .byte $7C, $11, $12, $13, $14, $15, $16, $17, $18, $19, $79
        :

        .dbyt   $20F3
        .byte   (:++)-(:+)
        : .byte $7D, $76, $76, $76, $76, $76, $76, $76, $76, $76, $7A
        :

When decoded, these are rows of tiles that would sit right on top of each other. While this tool itself doesn't interpret the display list format (one is coming ) it does handle the general case of putting a series of tiles in order in memory for further processing. Since this mapping represents a rectangular object, to generate a PNG image of what this mapping represents, the rows must simply be entered in a file and then the combination of nttochr and chrtopng can generate the image appropriately:

Code: Select all

nttochr title_bg.bin <nt.bin | chrtopng -r 11 -p pal.bin

Since I know the width of the resulting block of the nametable is 11 and the tiles are stacked, telling chrtopng to generate an image 11 tiles wide results in the mapped data. This is all directly consuming native system primitives. The next tool I'm focused on is something that can clobber a nametable together from a display list. Since this tool can act as a filter, that thing would then sit at the front, interpreting display lists into nametables, this then turning nametables into mappings from a bank, and chrtopng finally spitting it all out as a mapped image. With all of that it should be able to pipeline inspection of arbitrary nametable data with just a CHR bank and palette in hand. Another possible case is prototyping mappings at the command-line by printing arbitrary tile IDs with printf(1). This one is already proving quite useful...just gotta get that display list one done and I can really kick graphics analysis into higher gear without leaving the comfort of xterm.

[segaloco]

And now I've got a Nintendo display list script too:

https://gitlab.com/segaloco/misc/-/blob ... ls/dlntont

This tool takes a series of display lists on the standard input and spits out a full visible VRAM nametable (i.e. all four pages, just the visible part) containing the display lists slotted in at the appropriate spots, otherwise filled with $24 (a very common blank in Nintendo fonts). The input can be passed as an argument instead, and there is a "-b" option used to select the ID to place in all of the blank spaces (if it differs from $24).

The end result is yet another tool that can be piped together with these other ones for inspecting graphical data. For instance, knowing where the title screen display list is in the MAIN-PRO file of Doki Doki Panic, I can use this along with the other utilites, a BG CHR bank, and a palette, to inspect what the display list will result in:

Code: Select all

dd if=main_pro.bin bs=1 skip=24369 count=539 | dlntont -b `echo "ibase=16;FB" | bc` | nttochr title_bg.bin | chrtopng -r 32 -m 1024 -p pal.bin

So stepping this pipeline, first I extract the display list from main_pro.bin using dd, skipping to the offset it starts at and then only taking the 539 bytes that make it up. Aside, I'm going to make another script that reads stdin for display lists until a terminator is observed, that I can stick in front of this, then I only need to say where the display list is, the script will cut it off at the end and close the pipe to dlntont.

So the nametable is isolated, passed down pipe to dlntont. In this case, Doki Doki Panic uses a blank byte of $FB rather than $24, so I pass this to prime the nametable. Otherwise the nametable being inspected is coming down the pipe. After dlntont generates the full VRAM mapping, that result is passed to nttochr, which will create a memory bank containing the nametable tiles in order from tiles in title_bg.bin. Finally, this is passed into chrtopng, the row width is set to 32 tiles since it is a nametable rather than bank being inspected, and the maximum is increased to 960 so it'll accept all the tiles making up a single quadrant of the nametable VRAM.

The result:

So now with just:

• A bank of tiles
• An example palette
• A file containing a nametable
• The coordinates/dimensions of that nametable

I can plug this information into various parts of this pipeline and inspect the nametable in context. This demonstrates the ability to then wrap this pipeline in a script that further allows providing coordinates to a CHR bank and a palette, foregoing any need to isolate these components manually in the filesystem.

The main detriment to doing it in a bunch of broken up pieces and mostly in shell is it isn't the fastest. Generating this on my Raspberry Pi took about 9-10 seconds. The fact that emulation works shows that there are obviously much faster ways to interpret this data into a complete image, apparently mechanisms that can do it 60 times in a second on most hardware. However, since this is just meant to be an analysis tool, I'm not too concerned that it takes a little bit of time. The 9-10 seconds it takes is still significantly faster than finding the data and then interpreting it manually by looking up tiles in some tool. To me, that's a win, I can interpret a nametable in a matter of seconds rather than minutes or hours depending on how far I've gotten in identifying tiles, palettes, what have you. Plus, the scripting is basic and hopefully easy enough to read, lowering the maintenance burden vs. something in super optimized C.

Probably the last script in this pipeline set will be the one that terminates a display list automatically and then the overarching command that hooks into this with just addresses and a binary file containing them.

Once that is done I'll be looking into the reverse of these various bits, ways to take a 2bpp indexed PNG and reverse the process, potentially create maps, tiles, and display lists from it. That'll probably take a bit longer and is a secondary goal, but hopefully one that'll make this stuff just that much more useful.

[segaloco]

And now the last of this stuff for a little while, the bits here strung together in a nice way.

https://gitlab.com/segaloco/misc/-/blob ... s/dlntopng

This is dlntopng, you provide a CHR map and a Nintendo-format display list and it creates a PNG of the four resulting nametables stacked vertically. I have plans to eventually get them in the more conventional orientation but for now the tool meets my needs.

A typical use is as such

Code: Select all

dd if=main_pro.bin bs=1 skip=24369 | dlntopng -c title_bg.bin -p pal.bin -t 0 -b 251

From left to right, first dd(1) is used to establish a stream pointer on a pipe starting at a display list. This is one I've been testing with, the first list of the title sequence of Doki Doki Panic. This just as easily could be an already extracted display list in an isolated file, doesn't matter, the flexibility is left up to the consumer to find and provide the head of a display list.

A few options come into play:

First, -c must be provided, this denotes the bank of CHR to create the image from. I suppose I could make this optional...but then this would just be an expensive large blank PNG generator.

The -p option then points to a palette consisting of four color generator IDs. This one can be omitted as a fallback palette was easily embedded in the PNG generator. However, for the correct palette it must be supplied as a file.

The -t option is used to set the terminating byte of the display list. Each list is an address followed by a length and a payload, then another address, wash rinse repeat, until some terminating byte. Different versions of the format have different terminators so this leaves that flexible. If not used, the default of 255 ($FF) from the underlying tool is used. This is based on $FF being the default in the FDS boot ROM shared versions of the format. Getting the right terminator is pretty important, although the utility will also stop on EOF and process anything it was able to read out. This means if you snip out a display list with or without a terminator, this tool should also work with it. However, if you wind up feeding it something where the terminator never appears, a lot more of the file flows down the pipeline than the resulting PNG may imply.

The -b option is then used to pass the blank tile ID. This is the tile each nametable is primed with before the display lists are overlaid into it. By default this is 36 ($24) which is one of the more common blank tile assignments observed in Nintendo titles.

So in this case the BG tile bank is in title_bg.bin, palette in pal.bin, I want to handle display lists terminated with 0, and I want to prefill my namespaces with tile $FB, the typical blank in Doki Doki Panic. The result:

As mentioned, this will stack all four namespaces on top of each other. This exploits the fact that the PNG tool reads linearly. By default it cuts at 256 tiles, but expanding the tile limit allows it to just keep stacking the rows of the next nametable right after another.

Similarly, the width can be expanded to 64 tiles wide, although this would result in the incorrect placement of rows since nametable rows are 32-wide, so I'm thinking to address this sometime down the line I'll write a filter that sits between the nametable generator and the tile mapper and will shuffle the nametable contents such that the thing that comes out the very end looks like a 2x2 nametable grid. That'll allow the tools to remain precise in isolation, I just add a fork in the usual path they don't need to know about. That's why I like pipelines so much, even if they result in slower operation or a little more work to get them to behave like C code. It's just too easy to swap things in and out, that and monitor with tee(1), which has been quite useful for sampling the behavior of the various stages along the way.

Anywho, a program does more than one thing, since this is meant to handle the format pretty generally, it should work with a game using a different blank byte and terminator scheme:

Code: Select all

dd if=zldata3.bin bs=1 skip=`echo "ibase=16;B59B-B400" | bc` | dlntopng -c zlbg.bin -p pal.bin -t 255 -b 36

This demonstrates another nicety of what my tool *doesn't* do. Since it just expects a nametable, and I use something else to provide that, I have the freedom to apply my usual dd(1) pattern with bc(1) calculating an offset for me from the hex origin of a file and the address of a list in that file.

The result:

Recall that this does not acknowledge attributes and is just creating a single file with all possible map contents. As such, only one palette applies to the whole of the image. Creating a PNG with each tile referring to an attribute would be a whole different kettle of fish, especially if the attributes aren't in the display list itself. Or they may be but they may refer to tiles not drawn by the display list.

Anywho, this is a tool I've been wanting for a long time, I decided it was finally high time to just make it myself. I'm sure there's other stuff out there you can also achieve this sort of thing with, but for me, this is quick and painless, I can further wrap the calls for particular files in a given disassembly such that I just do things like

Code: Select all

zldata3disp 0xB59B | png

And it maps that down onto the right command. Not a script I'd bother uploading since it is so specific, but something I could write once at the start of a disassembly and then as I'm stepping down a bank, just run list after list through to help identify bits quicker. I don't have to leave my current shell session, mess with any cursors or desktop toolkits, it "just works".

Anywho there'll be more but this achieves the initial goal I set out with, to simplify display list analysis to the point of a single command I can just plug things into and view the results of. I see this speeding up my analytical process quite a bit.

Potential future directions include:

• Reverse tools, various comparable utilities turning 2bpp PNG into NES formats.
• Expansion to other platforms, I tried to isolate variables like bit depth, nametable size, tile size, etc. to constants that are easy to adjust.
• Optimization, who doesn't like a little more speed.
• 2x2 format instead by way of another filter in the pipeline and adjustmets to PNG sizing
• A transparency option for palette entry 0
• A similar pipeline for OAM

In particular, working out the last two along with some priority stuff, there may be a way to use some off-the-shelf PNG utilities to overlay OAM and BG data with transparency and construct scenes from both, useful in the event a screen has both an initial display list and initial OAM setup, one could recreate the initial scene with such tooling.

If you do wind up using any of this and encounter bugs, I'd love to hear about them. I've identified several places where things can get just a little wacky, but given it's a tool for a pretty technical purpose already, I trust anyone using it would know what to do if it started speaking in tongues.