Tiled palette quantization tool

Discussion of hardware and software development for Super NES and Super Famicom. See the SNESdev wiki for more information.

Moderator: Moderators

Forum rules
  • For making cartridges of your Super NES games, see Reproduction.
none
Posts: 117
Joined: Thu Sep 03, 2020 1:09 am

Re: Tiled palette quantization tool

Post by none »

turboxray wrote: Tue Sep 06, 2022 7:13 pm I take it you have no experience in this kind of stuff. It's not trivial in the least. I've written my own, refined it over the years, and I gave up on trying to make it "perfect". The best it does is get really close, and then lets the artist finish it off with manual touch ups. This is bar far THEE most impressive attempt I've seen yet - maybe only matched by Nitro Character Studio for GBA development that does this (and good luck finding that official app from Nintendo).
As I said, I know it's hard, and I also said that I probably couldn't do better myself. Excuse me if this came across as nitpicking but it's really meant as constructive criticism.
turboxray wrote: Tue Sep 06, 2022 7:13 pm If you want to use two layers, simply write your own app that splits ALL colors in tiles by half (regardless of the amount), and running the "upper" and "lower" through this app.
It's not that easy though. The tricky question is how to split colors in half.
Rilden wrote: Tue Sep 06, 2022 7:06 pm Can you post my tool's output image you used for comparison? How many palettes and colors per palette did you use? I want to look into this.
Screenshot_2022-09-07_12-45-13.png
I set fraction of pixels to "1.0" (higher is better I guess?), the square artifacts are less noticable with higher values but still there, especially in the dark / blue parts, or around the edges of the star in the bottom right corner you can notice there's an area that's more purple than the sorrounding part.

They're also visible in the demo images you posted for example in the buzzlook one.

I've glanced over the code (in worker.js) though not in detail, and it seems you're shuffling tiles into different palettes and then requantize with a kind of hill climbing method?
none
Posts: 117
Joined: Thu Sep 03, 2020 1:09 am

Re: Tiled palette quantization tool

Post by none »

Here's an idea how to fix it - again i only quickly glanced over your code.

It seems you're using some kind of accumulated error (mean squared error?) to figure out what palettes are "better" or "worse" than another set of palettes.

Maybe you can improve on the idea of "error" by also taking the distance to neighbor pixels into account and, if that doesn't fix it on its own, amplify the error for boundary pixels by some (user definable?) degree.
turboxray
Posts: 348
Joined: Thu Oct 31, 2019 12:56 am

Re: Tiled palette quantization tool

Post by turboxray »

none wrote: Wed Sep 07, 2022 4:13 am As I said, I know it's hard, and I also said that I probably couldn't do better myself. Excuse me if this came across as nitpicking but it's really meant as constructive criticism.
Nah I'm a fan of constructive criticism (and hard critiques - they get straight to the point);
I think the palettes it chooses aren't really optimal
Just some of the statements like this.. as odd (and false). I mean, optimal compared to what? I have a range of test images that I use for palette sorting software like this, and some of them are down right brutal and result in lots of tile-edge (jpeg-like) artifacts in other palette sorting apps. I've thrown almost everything I've had as test images and this thing performs way beyond anything related to it.

All my tests demonstrated to me that the palette selection is very optimal. I dunno. I'm using all 8 palettes for SNES, and all 16 palettes for PCE. Maybe it doesn't work so great with less palettes (which I haven't really tested), but what I'm seeing is most definitely optimal results for everything I've tested.

turboxray wrote: Tue Sep 06, 2022 7:13 pm It's not that easy though. The tricky question is how to split colors in half.
Yeah but the point is instead of worrying about what's easy or not, first see what your starting point looks like.

I mean, I have no idea what content you're working with and what you're trying to do with said content - that'll also drive direction, methods, and results. As in, if you're using two overlaid images combined to get higher color fidelity image 'ala 31 colors per tile' then at that point you're already using the same amount of space for 8bit tiles - so why not use 8bit tiles (assuming SNES)? Or are you simply trying to display two independent layers of high fidelity color but with a shared 8 palette set? If it's the latter, then simply give this app a combined image of each, side by side or such, and let it create a single set of palettes that both reference.
Even though it only uses one palette I think it still looks better...
Wait what? You're saying you think that 16 color image looks better than the 8 palette sorted image????
none
Posts: 117
Joined: Thu Sep 03, 2020 1:09 am

Re: Tiled palette quantization tool

Post by none »

turboxray wrote: Wed Sep 07, 2022 8:44 am Just some of the statements like this.. as odd (and false). I mean, optimal compared to what? I have a range of test images that I use for palette sorting software like this, and some of them are down right brutal and result in lots of tile-edge (jpeg-like) artifacts in other palette sorting apps. I've thrown almost everything I've had as test images and this thing performs way beyond anything related to it.
I don't want to get into an argument about this with you, lots of this is a matter of taste I guess.

But I think you're cherry picking quotes here, because in the same sentence I clarified what I mean with "optimal"... you're just implying that comparison stuff

Anyways to keep this constructive, I wasn't aware there are lots of other tools around that do tile based quantization? Which ones do you mean?
Even though it only uses one palette I think it still looks better...
Wait what? You're saying you think that 16 color image looks better than the 8 palette sorted image????
Yes. I think less colors still looks better than square artifacts. Again it's a perception thing but too me it makes it look somehow "artificial" in a sense.
Rilden
Posts: 21
Joined: Wed Jun 22, 2022 5:34 am

Re: Tiled palette quantization tool

Post by Rilden »

The tool tries to optimize the palette so that the image looks as close as possible to the original image.
Comparison between the original image and neuquant:
full-neuquant.gif
full-neuquant.gif (41.68 KiB) Viewed 4053 times
Comparison between the original image and the 8 palette output:
full-tiled.gif
full-tiled.gif (50.09 KiB) Viewed 4053 times
The 8 palette output is much closer to the original image than neuquant.
lidnariq
Posts: 11432
Joined: Sun Apr 13, 2008 11:12 am

Re: Tiled palette quantization tool

Post by lidnariq »

What's the error-measuring function you're using?
Rilden
Posts: 21
Joined: Wed Jun 22, 2022 5:34 am

Re: Tiled palette quantization tool

Post by Rilden »

I use the square distance between the rgb values, weighted by 2*red, 4*green, 1*blue:

Code: Select all

2 * deltaR^2 + 4 * deltaG^2 + 1 * deltaB^2
I'm working on adding dithering to make block boundaries less visible. Here is an example image with snes's high res mode:
buzzlook-highres-dither.png
Rilden
Posts: 21
Joined: Wed Jun 22, 2022 5:34 am

Re: Tiled palette quantization tool

Post by Rilden »

Dithering options are now available on the website. Dithering: fast does dithering after palette reduction, slow does dithering while generating the palette. Dither weight controls the amount of dither (between 0 and 1).
Here is a comparison between a dithered and undithered picture, using genesis restrictions (4 palettes, 3 bits per channel). I used dither: slow, dither weight of 0.5:
buzz_genesis.png
The same image with snes restrictions:
buzz_dither_05.png
buzz_dither_05.png (32.11 KiB) Viewed 3924 times
93143
Posts: 1717
Joined: Fri Jul 04, 2014 9:31 pm

Re: Tiled palette quantization tool

Post by 93143 »

It seems to work better than whatever Randy Linden and Sculptured Software had access to back in '95.

Starting with an 8bpp image at SNES resolution, containing 237 colours total:

Doom-TITLEPIC_SNES8bpp_NTSC.png
Doom-TITLEPIC_SNES8bpp_NTSC.png (72.44 KiB) Viewed 3836 times

I used eight 16-colour palettes with common black and no dither, with "Fraction of pixels" = 1.0, to arrive at this image, which has 107 colours:

Doom-TITLEPIC_SNES4bppRilden_NTSC.png
Doom-TITLEPIC_SNES4bppRilden_NTSC.png (76.55 KiB) Viewed 3836 times

For comparison, the actual title screen of SNES Doom, which was reduced to 4bpp to jam it into the 2 MB ROM, turned out like this (82 colours total):

Doom-TITLEPIC_SNES.png
Doom-TITLEPIC_SNES.png (35.6 KiB) Viewed 3836 times

...

Is there something wrong with my toolchain, or does this thing scatter RGB channel values a bit around the reduced-bit-depth positions? Before I posterized it, the saved image from the web page (the second one above) had well over 200 unique colours in it.
Rilden
Posts: 21
Joined: Wed Jun 22, 2022 5:34 am

Re: Tiled palette quantization tool

Post by Rilden »

If you quantize the image to 8 palettes of 16 colors, with 1 shared color, the output shouldn't have more than 121 colors. This is what the image looks like after reducing it to 8x16 colors, with black as the shared color:
Doom-8x8-8p16c-s.png
Doom-8x8-8p16c-s.png (53.88 KiB) Viewed 3791 times
This image has 106 colors. If you select 5 bits per channel the output is already posterized to 32 values per color channel.
One reason for "scattering" RGB channel values could be the difference in how those 32 values are chosen.
The 32 values chosen by my tool are:
0,8,16,25,33,41,49,58, 66,74,82,90,99,107,115,123, 132,140,148,156,165,173,181,189, 197,206,214,222,230,239,247,255
In your last image (Doom-TITLEPIC_SNES.png) the 32 values are:
0,8,16,24,32,40,48,56, 66,74,82,90,98,106,114,122, 132,140,148,156,164,172,180,188, 198,206,214,222,230,238,246,254
93143
Posts: 1717
Joined: Fri Jul 04, 2014 9:31 pm

Re: Tiled palette quantization tool

Post by 93143 »

No, I mean the colours in the image I saved off your web page were approximately 15-bit RGB, but there was a bit of visible scatter in the colorcube plot, and there were over 200 unique 24-bit colours. When I posterized in GIMP, it dropped to 107 unique colours, and that result is what I've showed as the second image in my previous post.

Note that the input image (Doom-TITLEPIC_SNES8bpp_NTSC.png) was already posterized to 32 levels in GIMP, since it was intended as a title screen mockup for a hypothetical re-port with more ROM. I don't know if that makes any difference...

Here's the result of another run (because I didn't save the un-posterized version of the first run):

test.png
test.png (54.3 KiB) Viewed 3705 times

This image has 243 colours in it. Posterizing it reduces that to 108.

That's another thing - it seems the output is not deterministic. I couldn't duplicate my first result exactly on a second run, and it looks like you couldn't either. I suppose if I were to look at the source I'd be able to figure out why not...

...

Just to be clear, the last image, Doom-TITLEPIC_SNES.png, was not run through your program, nor did I posterize it myself. It's just a screenshot from the actual port. This may explain the discrepancies in RGB channel value notch positions you've noted - I imagine the emulator did it a bit differently.
Rilden
Posts: 21
Joined: Wed Jun 22, 2022 5:34 am

Re: Tiled palette quantization tool

Post by Rilden »

What browser do you use? I used Firefox, Chrome and Edge and the output has less than 121 colors on all of them.
I made some changes to how rounding is done, try it now and see if it fixes the scattering of colors.
it seems the output is not deterministic
The algorithm selects pixels from the input image in a random order, that's why the output is not deterministic.
This is basically how the algorithm works:

Code: Select all

repeat n times:
	select a random tile from the image
	closestPalette = find the palette that's closest to the tile's colors
	inputColor = select a random color from the tile
	paletteColor = find the color in closestPalette that's closest to inputColor
	move paletteColor closer to inputColor
tepples
Posts: 22708
Joined: Sun Sep 19, 2004 11:12 pm
Location: NE Indiana, USA (NTSC)
Contact:

Re: Tiled palette quantization tool

Post by tepples »

Might something in the chain be doing color space conversion followed by dithering?
turboxray
Posts: 348
Joined: Thu Oct 31, 2019 12:56 am

Re: Tiled palette quantization tool

Post by turboxray »

MrJ — 09/05/2022
Using the tiledpalettequant site, pressing the quantize button twice after one another will get you two different “results”.
I know that Mr J reported this, but I'm not sure on the exact details. So maybe there's something that's persisting (if you click the button twice instead of loading a new image).

EDIT: Never mind, just read rilden's second part in their post.
93143
Posts: 1717
Joined: Fri Jul 04, 2014 9:31 pm

Re: Tiled palette quantization tool

Post by 93143 »

Rilden wrote: Sat Sep 10, 2022 4:37 am What browser do you use?
Brave.

When I download an image from this forum, it works fine. It's only your demo page that gives me the weirdly scattered colours.

I'm right-clicking and selecting "Save image as...". Is there another way?
Post Reply