SPC7110 Reverse Engineering Project

[kammedo]

Don't worry. Real life has to get his priorities.

I Agree too.

Andreas, do you feel comfortable programming in assembly language?

Last time i programmed in assembler was about 11 years ago, and for the 80x86. But yes, at that epoch i programmed a lot in assembler and, if needed i could probably learn the neccesary about 65x816 in some days place if you provide "skeleton" functions to manage the hardware. If you prefer we go that way, i should start inmediately. Could you suggest some reasonable cross-assemblers/emulators for me to make tests in a PC?

If we are talking about SNES Assembler, I would be pleased to help - I even have an old "ASM studio" project that i use for my projects, unfinished, so maybe its time!

[neviksti]

Bleh, I'll be tired tomorrow, but I was too curious to sleep anyway. I ran some tests with the FIFO setup.

I took a look at the QM probability estimation tables. In a IBM's document, with registers of 16 bits and a table of 113 values, the first values looks like this:

0x5a1d
0x2568
0x1114
0x080b
0x03d8
0x01da
....

So if we were to make a small 8-bits version... {0x5a, 0x25, 0x11, 0x08, 0x03, 0x01} would be a reasonable option?

Yep, that is now verified. The probabilities are $5A, $25, $11, $08, $03, $01.

I had a program keep lowering the input data to find new cutoffs for one more MPS. I saw one bitplane evolve all the way to a prob of $01.

I'll clean up the code a bit (hopefully tomorrow) and share it as example "skeleton code" for those considering writing their own tests. I'll also post the raw data from my test if people want, but it is not very interesting now that the probability table is known.

Could you suggest some reasonable cross-assemblers/emulators for me to make tests in a PC?

Hmm... I didn't think ahead very well on that. It will be very hard to debug everything this way. Maybe this isn't such a good idea.

But there really only seems to be two programs we need:
One to decompress custom data supplied in a file (which shouldn't be difficult to do now and if we decide on formats I could have done fairly quickly).
And a second one to find the prob and mps values for a particular sequence.

As long as that's all we go for, it shouldn't be too bad. If we don't focus our efforts and everyone tries to make their own personal version, unless everyone is a perfect coder without need to debug on the real thing, it will get out of hand.


Andreas, maybe it would be best if you wrote a C program to do what you want (and don't use any local stack variables or heavy "pass by value" function calls) and I can quickly translate it to asm. This way I'll understand exactly what you want and better understand the details of the program in case we need to debug. Also you won't have to waste time on a side project of learning the SNES memory layout, instructions, etc. I think this may be the most time efficient method.

Keep in mind the following things:
- Each run I can only save 32kBytes of data.
- In my code library I have something similar to a printf command, so have the code give some status updates of how it is doing every once in awhile so I can see that the code doesn't get "stuck" on the real hardware.
- I still haven't figured out for sure what memory locations trigger a U2 ROM access (what is complicating things is that it is somehow not entirely consistent... maybe it depends on initialization of the chip somehow?). What this means for your program is that it would be really nice to keep the compressed -> uncompressed data buffers to be no more than 2kB total so I can fit everything in a region I know is safe.

[Andreas Naive]

Andreas, maybe it would be best if you wrote a C program to do what you want (and don't use any local stack variables or heavy "pass by value" function calls) and I can quickly translate it to asm. This way I'll understand exactly what you want and better understand the details of the program in case we need to debug. Also you won't have to waste time on a side project of learning the SNES memory layout, instructions, etc. I think this may be the most time efficient method.

OK. Not sure if i'm going to be able to work on this this evening, but i will pass you a C version of what i want when possible.

- Each run I can only save 32kBytes of data.

Yes, i know. Without compression, i only need 2 bytes of storage per bit, so we could reach 2^14 bits if used for a single example.

- In my code library I have something similar to a printf command, so have the code give some status updates of how it is doing every once in awhile so I can see that the code doesn't get "stuck" on the real hardware.

Understood.

- I still haven't figured out for sure what memory locations trigger a U2 ROM access (what is complicating things is that it is somehow not entirely consistent... maybe it depends on initialization of the chip somehow?). What this means for your program is that it would be really nice to keep the compressed -> uncompressed data buffers to be no more than 2kB total so I can fit everything in a region I know is safe.

2^14 bits=2^11 bytes=2KiB, so no problem on that side. But... wait. 2KiB for ALL the buffers or 2KiB for every one? I mean, i'm supposing you refer to the buffers used when triggering the decompression, so we will only need 2 of them (one for the compressed/input stream, and one for the uncompressed/output stream). I suppose i can freely move the results of the decompression to other buffer outside that 2KiB zone, right?

[MatthewCallis]

Has anyone here contacted d4s about helping with this project? He has the official development hardware and could probably provide a great source for running some tests really easily.

[tepples]

Andreas, maybe it would be best if you wrote a C program to do what you want (and don't use any local stack variables or heavy "pass by value" function calls) and I can quickly translate it to asm.

With the help of CC65?

[kyuusaku]

Has anyone here contacted d4s about helping with this project? He has the official development hardware and could probably provide a great source for running some tests really easily.

The Emulator SE doesn't emulate the SPC7110, it also requires a DOS/SCSI workstation which copiers don't. I think there's a chance reading/writing cartridges isn't implemented in hardware or at least software too since it's intended function is to debug code from emulation memory and to test candidate cartridges, not to reverse engineer. The only hardware that would help this effort is just a few 74 series chips allowing the SNES bus to change the SPC memory, but nobody is confident about building it because of the wire count and board reworking necessary.

[neviksti]

2^14 bits=2^11 bytes=2KiB, so no problem on that side. But... wait. 2KiB for ALL the buffers or 2KiB for every one? I mean, i'm supposing you refer to the buffers used when triggering the decompression, so we will only need 2 of them (one for the compressed/input stream, and one for the uncompressed/output stream). I suppose i can freely move the results of the decompression to other buffer outside that 2KiB zone, right?

I meant input + output bufer = 2kB total.
Yes we can freely move the results around afterwards.
The sensitive time is after stuffing the FIFO with our values and doing the decompression / reading out the FIFO results. Any "unplanned" U2 ROM access here will cause a loss of a FIFO byte and will cause problems.

You are just looking for changes in the output right? Could we just read out and ignore the first xx output bytes and then just save the few you are interested in checking? That's kind of what I did with my "all mps" prob evolution test.

[MatthewCallis]

The Emulator SE doesn't emulate the SPC7110, it also requires a DOS/SCSI workstation which copiers don't. I think there's a chance reading/writing cartridges isn't implemented in hardware or at least software too since it's intended function is to debug code from emulation memory and to test candidate cartridges, not to reverse engineer. The only hardware that would help this effort is just a few 74 series chips allowing the SNES bus to change the SPC memory, but nobody is confident about building it because of the wire count and board reworking necessary.

I know it doesn't emulate the SPC7110 and I doubt he has the board for it but he's emulated the BS-X bios in his Ultra 16 and built me a cart to dump BS-X memory carts. I was just saying he's got the hardware knowledge to build something complicated.

[neviksti]

Andreas, maybe it would be best if you wrote a C program to do what you want (and don't use any local stack variables or heavy "pass by value" function calls) and I can quickly translate it to asm.

With the help of CC65?

No by hand. Without local variables C translates almost directly (well, local variables aren't too bad, but I don't like dealing with stack frames on the SNES).
Plus if I'm going to debug it on real hardware, I want to know every single memory access.

Has anyone here contacted d4s about helping with this project? He has the official development hardware and could probably provide a great source for running some tests really easily.

Thanks for the suggestion, but we currently have the ability to run custom data through the chip. And Kammedo sounds like he is going to be making a system as well (of a different design, with its own advantages). So we should be all set.

The only hardware that would help this effort is just a few 74 series chips allowing the SNES bus to change the SPC memory, but nobody is confident about building it because of the wire count and board reworking necessary.

Yeah, it is annoying to have to carefully avoid triggerring U2 ROM accesses with the FIFO mod, but in the end it is just a minor software consideration and it works... that's good enough for me at the moment.

---------------
EDIT: Here's an interesting tidbit. Writing a zero value to $420B doesn't trigger a U2 rom access, writing a non-zero value does. This chip is so weird. I still don't understand DarkForce's comment about the "dma reg" setting... why does this chip need to follow any of that. Unless there is a useful feature we are missing, there is absolutely no reason to.

[kammedo]

Probably its not weird, but just not perfect? That DMA channel matching remains a mistery to me too..I've been fiddling around with it yesterday, not getting anything intresting. Is it really needed for something? I mean, the SPC can't program the SNES DMA, as the SNES is the master of it's own bus....

Another option : could it be that the DMA is used to access the U2 chip in some strange way? But no, yet this doesnt make much sense as that would mean share the SNES's bus also, which would conflict with the SNES itself.

Or another option : The SPC has a DMA interface that it uses to get data from the U2. This DMA needs to be synchronized somehow with the SNES's dma and thus needs to be mapped on a DMA channel.

I am going to check that closer tomorrow.

[Andreas Naive]

You are just looking for changes in the output right? Could we just read out and ignore the first xx output bytes and then just save the few you are interested in checking? That's kind of what I did with my "all mps" prob evolution test.

Yes, at every decompression i'm interested in exactly one bit, so we would only need one byte. I will try to make this clear in the code somehow. I haven't had time to start it yet, so i guess i won't have it done till thursday, probably.

[neviksti]

Okay, here's some example test code. (it searches for the cutoff values to output another mps, look at the output at Cutoffs.bin using a hex editor to see the results -- every entry is $10 bytes).
http://neviksti.com/SPC7110/FIFO_cutoff_test.zip
People can use this as skeleton code for their own tests if you want. You will probably only need to change the file "testcode.asm".

I included the assembler (Win32 binary) so you have everything you need right here (and also because this is an older version, and I'm too lazy at the moment to make sure all my code still works with the newer versions). Open a command prompt and run make.bat to produce the SNES program (the game doctor 3 formatted rom: "SF2t7110").


My meetings went great today, and as a side benefit I don't have as much to do for tomorrows meetings as I thought. Since I have a bit of free time tonight, I'll see if I can whip together that "prob" and "mps" calculator for a specific sequence.

[Andreas Naive]

Since I have a bit of free time tonight, I'll see if I can whip together that "prob" and "mps" calculator for a specific sequence.

OK. I will just sit and wait then.

[neviksti]

Alright, here's the code:
http://neviksti.com/SPC7110/testcode.asm

The program takes a 256 byte input.
In then basically reconstructs compressed data which yields that data as output from the chip... in doing so it figures out the prob and mps for each bit.

The way the code is setup, I can easily change it so the 256 byte input is considered a sequence of mps and lps instead of actual bit values if that would be more useful to you.

Notes about the code:
- it assumes we never need more than $400 bytes compressed data to produce $100 bytes decompressed data.
- the code does NOT assume the probabilities are the entries in that table ( $5A, $25, $11, $08, $03, $01 ). This will make it easier to later catch problems that may still exist in the code.


The output format:
For each bit, the program outputs one byte: bit7=mps, bit6-0 are the prob value. Here's example output for an input of all $00's.

http://neviksti.com/SPC7110/output.bin

EDIT: Well, the program can actually handle input datasets of other sizes without changing any of the code. But the $400 byte size of "compressed data buffer" is chosen by hand (it can be increased, but that causes everything to run slower).

I'm going to get some sleep now, but if you have any datasets I can run them tomorrow.

[Andreas Naive]

So... how are we supposed to make new tests? Should i pass you a binary file with the desired 256 bytes?

At this point, i'm only interested in making some statistical analysis, so a random input (50%-50% distribution for 0's and 1's) would be fine for me.


EDITED: Ummm, i changed my mind: a 70%-30% distribution would be better.