I now have Super Power League 4, as well as another FEoEZ cart (which I am going to try the FIFO mod on later). The electronic parts still have not arrived though. Hmm. I was hoping to work on this over holiday.
I ran some more tests while monitorring on the oscilloscope and believe I understand the 'spurious' accesses to the U2 rom now. Accessing banks $7E or $7F (even writing to them strangely enough) cause /OE to be pulsed low. Accessing lower WRAM in bank $00 doesn't cause this, and neither does access to the $48xx regs (well, I only checked a couple).
The datalines appear to be pulled low through a resistor, for after access they slowly drop to zero. I have yet to see any access passthrough the SNES data lines to the U2 bus. So, while already believed to be the case, it seems confirmed that it is highly unlikely that one could write directly to a chip through the U2 bus.
With the 'spurious reads' now understood and easily under control, I think the FIFO mod will work fine.
Kammedo, it would be prudent to continue working towards getting your setup working. But unless you want a SPC7110 dumper just for the fun of having one to play with (that EPROM/FLASH programmer could be useful for many projects of course), I'd suggest holding off buying anything until we find out if the FIFO mod will work... in which case we will probably not need you to build anything.
SPC7110 Reverse Engineering Project
Moderator: Moderators
Forum rules
- For making cartridges of your Super NES games, see Reproduction.
Uhm - wasnt it SPL3?neviksti wrote:I now have Super Power League 4, as well as another FEoEZ cart (which I am going to try the FIFO mod on later). The electronic parts still have not arrived though. Hmm. I was hoping to work on this over holiday.
I ran some more tests while monitorring on the oscilloscope and believe I understand the 'spurious' accesses to the U2 rom now. Accessing banks $7E or $7F (even writing to them strangely enough) cause /OE to be pulsed low. Accessing lower WRAM in bank $00 doesn't cause this, and neither does access to the $48xx regs (well, I only checked a couple).
The datalines appear to be pulled low through a resistor, for after access they slowly drop to zero. I have yet to see any access passthrough the SNES data lines to the U2 bus. So, while already believed to be the case, it seems confirmed that it is highly unlikely that one could write directly to a chip through the U2 bus.
With the 'spurious reads' now understood and easily under control, I think the FIFO mod will work fine.
Kammedo, it would be prudent to continue working towards getting your setup working. But unless you want a SPC7110 dumper just for the fun of having one to play with (that EPROM/FLASH programmer could be useful for many projects of course), I'd suggest holding off buying anything until we find out if the FIFO mod will work... in which case we will probably not need you to build anything.
Well, i have ordered the EPRoms already, so why not
.Or perhaps I should try the fifo as well?
Oh well. Maybe I ordered the wrong one.kammedo wrote:Uhm - wasnt it SPL3?
I guess it doesn't really matter anyway, as Caitsith already has all the games. It was just a cheap bundle with the FEoEZ so I went for it.
(EDIT: I think it is SPL4, or at least according to Caitsith's site http://www.caitsith2.com/ )
The "replace the ROM" method is the most straightforward. And we can use our current dumping code with it. So it may well be the quickest route to success.kammedo wrote:Well, i have ordered the EPRoms already, so why not
Or perhaps I should try the fifo as well?
As for the FIFO, I would not suggest it unless you understand what is going on enough to write your own code and debug both the hardware and code. For I don't feel up for a "over the internet debug session" (sorry) with the FIFO design + code.
I probably won't even bother making the FIFO setup (even though I ordered the parts) if you get your setup working first. I don't really care if I have a custom dumping setup, I just want to play with the data
-
Andreas Naive
- Posts: 104
- Joined: Mon Nov 26, 2007 2:06 am
- Location: Madrid, Spain
- Contact:
I have finished doing a small app that take the data and generate a binary tree with the PROB and MPS evolution.
Basicly, it takes a list of compressed-uncompressed pairs and parse it to calculate the PROB and MPS. I have only applied it to the 16 first bits' set, but it could be used with more data simply taking care of ordering the pairs in increasing umcompressed streams order.
In case somebody consider this useful, here you have the source code and the file generated with it. It can be modified to output other data of interest.
http://andreasnaive.en.eresmas.com/SPC7110/test9.zip
The format i have used in the output file is easy to follow after giving it a thought, but to put an example:
0E L
5a 1
means:
mps=0
prob=0x5a ('E'xact, as opposed to 'G'uessed)
And, at that point a 'L'PS is outputted, shifting the state variables '1' bit.
As you can see, many times you can know what is 'mps' even if you can not calculate 'prob'.
I solved a number of stupid bugs in the code, and i'm now expecting it's almost bug-free, but there are no guarantees, of course.
Basicly, it takes a list of compressed-uncompressed pairs and parse it to calculate the PROB and MPS. I have only applied it to the 16 first bits' set, but it could be used with more data simply taking care of ordering the pairs in increasing umcompressed streams order.
In case somebody consider this useful, here you have the source code and the file generated with it. It can be modified to output other data of interest.
http://andreasnaive.en.eresmas.com/SPC7110/test9.zip
The format i have used in the output file is easy to follow after giving it a thought, but to put an example:
0E L
5a 1
means:
mps=0
prob=0x5a ('E'xact, as opposed to 'G'uessed)
And, at that point a 'L'PS is outputted, shifting the state variables '1' bit.
As you can see, many times you can know what is 'mps' even if you can not calculate 'prob'.
I solved a number of stupid bugs in the code, and i'm now expecting it's almost bug-free, but there are no guarantees, of course.
Ok, nice news in here.
I managed to get MDH 3 to work on my GDSF (GDSF7). The procedure i used is the same suggested by kyuusakku some posts ago (as in loading a game without spc with all banks in pass-through, plug the card in and resettting).
I dont know how much this is good for my gdsf. I tried to test the clock signal, and noises arise in the picture when i do.
Next step is getting all the stuff, and Ill mod the cart right away.
FEoEZ fails the test, but i can imagine this is due to the SRAM being mapped in a different bank probably. I have to trace the code back again and reverse it to come ahold of the differences between FEoEZ (ver 2.2 of test program) and MDH (ver 3.0).
I managed to get MDH 3 to work on my GDSF (GDSF7). The procedure i used is the same suggested by kyuusakku some posts ago (as in loading a game without spc with all banks in pass-through, plug the card in and resettting).
I dont know how much this is good for my gdsf. I tried to test the clock signal, and noises arise in the picture when i do.
Next step is getting all the stuff, and Ill mod the cart right away.
FEoEZ fails the test, but i can imagine this is due to the SRAM being mapped in a different bank probably. I have to trace the code back again and reverse it to come ahold of the differences between FEoEZ (ver 2.2 of test program) and MDH (ver 3.0).
Wow, amazing stuff here. I'm really impressed that you guys were able to figure out the underlying algorithm so quickly. I could never make any sense of caitsith's data sets.
I wanted to apologize for not being of any help, but arithmetic is most certainly the most difficult compression I've ever worked with.
From the one time I worked on an ari coder though, this sounds a bit different. What's with the MPS/LPS stuff? Does this also have a predictive context or something on top of standard arithmetic?
My understanding was that arithmetic just has a fixed table with a probability percentage of each symbol, similar to huffman. Or the table can be adaptive and calculated on the fly.
Anyway, I'm still willing to help out financially. If you guys need any parts that aren't crazy expensive, let me know and I'll be glad to help out. Myself, I have very little hardware -- FEoEZ and a UFO 8.3j (the copier has absolutely no concept of passthru, sadly.) And no EE skills whatsoever. I couldn't solder paperclips together if my life depended on it.
I was personally looking to get ahold of the test setup so that I could refine the non-compression registers, as these aren't going to be simple to guess like the S-DD1. And then from there attempt a setup to run custom data for anyone who wanted.
Honestly kind of surprised it's so difficult to find someone to pay to make custom testing hardware with all the crazy stuff I always see on the web like "portable" consoles, copier devices, 600+ mods to the EeePC, etc.
I wanted to apologize for not being of any help, but arithmetic is most certainly the most difficult compression I've ever worked with.
From the one time I worked on an ari coder though, this sounds a bit different. What's with the MPS/LPS stuff? Does this also have a predictive context or something on top of standard arithmetic?
My understanding was that arithmetic just has a fixed table with a probability percentage of each symbol, similar to huffman. Or the table can be adaptive and calculated on the fly.
Anyway, I'm still willing to help out financially. If you guys need any parts that aren't crazy expensive, let me know and I'll be glad to help out. Myself, I have very little hardware -- FEoEZ and a UFO 8.3j (the copier has absolutely no concept of passthru, sadly.) And no EE skills whatsoever. I couldn't solder paperclips together if my life depended on it.
I was personally looking to get ahold of the test setup so that I could refine the non-compression registers, as these aren't going to be simple to guess like the S-DD1. And then from there attempt a setup to run custom data for anyone who wanted.
Honestly kind of surprised it's so difficult to find someone to pay to make custom testing hardware with all the crazy stuff I always see on the web like "portable" consoles, copier devices, 600+ mods to the EeePC, etc.
-
Andreas Naive
- Posts: 104
- Joined: Mon Nov 26, 2007 2:06 am
- Location: Madrid, Spain
- Contact:
This morning i thought it could be worthwhile to do a patent search again now that we have more knowledge about how this stuff works. Well, i could be extremely lucky, but i assure you that it only took 2 minutes to hit the target.
Searching more, i found that Seiko Epson filled a number of patents describing compression schemes over 4-bit data using arithmetic coding and such... (By example, USP#5764804 and USP#5859926 give a big number of "preferred embodiments" of such things). But i'm not goint to talk about those; i want to talk about UDP#5859926 (yes, the described ideas are all related, but this is THE patent i'm most interested in ).
(Ummm, i'm having problems both with my usual FTP account and with MassMirror, so i'm not being able to upload the pdf file. If someone is willing to upload it, PM me and give me a mail account to send it to you (you can always take it from free sites, of course)).
This is the document i found this morning in less than 2 minutes, though i have only examined it in the evening.
Well, for you to understand why i think this is related with the SPC7110, i will summarize what i believe is important in the patent:
It works on 4 bits pixels, so the contexts/states are nibble-based (the SPC7110 would be using 2 such devices in parallel; one processing the low nibbles; the other one processing the high nibbles).
For every one of the 4 bitplanes, it takes the previous bits in the pixel/nibble to determine (part) of the contexts/states (i must warn that the word "state" is used with different meanings along the entire document; sometimes it refer to bitplanes, sometimes to which we have been calling contexts, some times to internal calculation states, ...), so, it have 1 "state" for bitplane #1 (the paten start the numbering by 1), 1 fot the bitplane #2, and so on for #3 and #4, totallizing 15 "states"/contexts. (figure 5)
Furthermore, the surrounding pixels/nibbles are used to determine more context/state bits: basicly, for three surrounding pixels/nibbles (in the case described they are the pixels at the left, top and top-left --figure 6--, though i guess in ours it would be simply the previous ¿3? nibbles, it takes). As using 2^12 states would be excesive, it "degenerate" the states by doing a 2 steps degeneration: first, each pixel/nibble is "degenerated" to a single bit (i it could be simply the bit of the previous nibbles having the same bitplane number or it could be something more convolved), and then, with those 3 "degenerated" bits from the previous 3 pixels/nibbles, it take one of 5 "states" depending of which one of those 3 are equals/distinct (figure 7).
By combining both ideas, we would have 5*15=75 contexts/states, but, to save space, it use one more degeneration table (figure 10) so that only 31 states remain (basicly, for 4 of the 15 first states, all the other 5 states are considered, while the other 11 of the first 15 states are grouped together whatever the other 5-possibilities state bits be), so we have 5*4+11=31 "final" contexts/states.
The core idea behind all this is that, as you can not efficiently store a big number of contexts (let's say 2^12*5), you use "degeneration" tables so as to group together patterns with similar probability distributions and obtain a reasonably small number of states/contexts.
Of course, all this should be taken with a bit of salt, as those patents are always pretty generic and we shouldn't suppose this is directly the description of how the SPC7110 algo works, but i definitively think this is very closely related. If you look USP#5963672 you will see a number of generalizations of these ideas, by example.
Even if i'm right and this is related to the SPC7110, we still would need a lot of work before filling in the details, and the custom hardware interface is still probably needed to allow us to access to long streams of MPS/PROB pairs.
By the way, the patents are centered on describing the evolution/context thing, while for the arithmetic compression, they refer to the arithmetic compression framework described in the international standard ISO-IEC11544:1993 (that is, the JBIG standard). I haven't found a free copy of this, but if somebody has access to it, i would like to give it a look (please, don't wast money on this; after all , the arithmetic coding is the part we have better understood in the SPC7110, so i don't think we would *need* it, but it would be nice to check that document).
Searching more, i found that Seiko Epson filled a number of patents describing compression schemes over 4-bit data using arithmetic coding and such... (By example, USP#5764804 and USP#5859926 give a big number of "preferred embodiments" of such things). But i'm not goint to talk about those; i want to talk about UDP#5859926 (yes, the described ideas are all related, but this is THE patent i'm most interested in
).(Ummm, i'm having problems both with my usual FTP account and with MassMirror, so i'm not being able to upload the pdf file. If someone is willing to upload it, PM me and give me a mail account to send it to you (you can always take it from free sites, of course)).
This is the document i found this morning in less than 2 minutes, though i have only examined it in the evening.
Well, for you to understand why i think this is related with the SPC7110, i will summarize what i believe is important in the patent:
It works on 4 bits pixels, so the contexts/states are nibble-based (the SPC7110 would be using 2 such devices in parallel; one processing the low nibbles; the other one processing the high nibbles).
For every one of the 4 bitplanes, it takes the previous bits in the pixel/nibble to determine (part) of the contexts/states (i must warn that the word "state" is used with different meanings along the entire document; sometimes it refer to bitplanes, sometimes to which we have been calling contexts, some times to internal calculation states, ...), so, it have 1 "state" for bitplane #1 (the paten start the numbering by 1), 1 fot the bitplane #2, and so on for #3 and #4, totallizing 15 "states"/contexts. (figure 5)
Furthermore, the surrounding pixels/nibbles are used to determine more context/state bits: basicly, for three surrounding pixels/nibbles (in the case described they are the pixels at the left, top and top-left --figure 6--, though i guess in ours it would be simply the previous ¿3? nibbles, it takes). As using 2^12 states would be excesive, it "degenerate" the states by doing a 2 steps degeneration: first, each pixel/nibble is "degenerated" to a single bit (i it could be simply the bit of the previous nibbles having the same bitplane number or it could be something more convolved), and then, with those 3 "degenerated" bits from the previous 3 pixels/nibbles, it take one of 5 "states" depending of which one of those 3 are equals/distinct (figure 7).
By combining both ideas, we would have 5*15=75 contexts/states, but, to save space, it use one more degeneration table (figure 10) so that only 31 states remain (basicly, for 4 of the 15 first states, all the other 5 states are considered, while the other 11 of the first 15 states are grouped together whatever the other 5-possibilities state bits be), so we have 5*4+11=31 "final" contexts/states.
The core idea behind all this is that, as you can not efficiently store a big number of contexts (let's say 2^12*5), you use "degeneration" tables so as to group together patterns with similar probability distributions and obtain a reasonably small number of states/contexts.
Of course, all this should be taken with a bit of salt, as those patents are always pretty generic and we shouldn't suppose this is directly the description of how the SPC7110 algo works, but i definitively think this is very closely related. If you look USP#5963672 you will see a number of generalizations of these ideas, by example.
Even if i'm right and this is related to the SPC7110, we still would need a lot of work before filling in the details, and the custom hardware interface is still probably needed to allow us to access to long streams of MPS/PROB pairs.
By the way, the patents are centered on describing the evolution/context thing, while for the arithmetic compression, they refer to the arithmetic compression framework described in the international standard ISO-IEC11544:1993 (that is, the JBIG standard). I haven't found a free copy of this, but if somebody has access to it, i would like to give it a look (please, don't wast money on this; after all , the arithmetic coding is the part we have better understood in the SPC7110, so i don't think we would *need* it, but it would be nice to check that document).
The arithmetic coder variant used in JBIG is called the QM-coder. Google can find you more info.
For comparison, this Wikipedia article covers range coding, a variant of arithmetic coding used by free software.
For comparison, this Wikipedia article covers range coding, a variant of arithmetic coding used by free software.
-
Andreas Naive
- Posts: 104
- Joined: Mon Nov 26, 2007 2:06 am
- Location: Madrid, Spain
- Contact:
I have seen QM-coder descriptions in the past, but that document seems to describe a general framework for arithmetic compression, what is what the patents refer to. That's what i was interested in.
Concretely, this sections (i found the index somewhere in the net):
Concretely, this sections (i found the index somewhere in the net):
6.8
Arithmetic coding ..............................................................................26
7
Test methods and datastream examples ....................................................43
7.1
Arithmetic coding ..............................................................................44
-
Andreas Naive
- Posts: 104
- Joined: Mon Nov 26, 2007 2:06 am
- Location: Madrid, Spain
- Contact:
I took a look at the QM probability estimation tables. In a IBM's document, with registers of 16 bits and a table of 113 values, the first values looks like this:
0x5a1d
0x2568
0x1114
0x080b
0x03d8
0x01da
....
So if we were to make a small 8-bits version... {0x5a, 0x25, 0x11, 0x08, 0x03, 0x01} would be a reasonable option?
0x5a1d
0x2568
0x1114
0x080b
0x03d8
0x01da
....
So if we were to make a small 8-bits version... {0x5a, 0x25, 0x11, 0x08, 0x03, 0x01} would be a reasonable option?
-
jolly_codger
- Posts: 14
- Joined: Wed Jun 04, 2008 10:38 am
Those are some quality finds!
Chose to not store a MPS variable (default 0 always).
8 PROB vars (one per bitplane). Updated per bit discarder.
Which gets:
(LASTMPS ~ last bits output)
Bits 18-20 devolve into pure chaotic mess.
Bit 21 does not follow bit 17.
The above shows the last bits written out. Output wants MPS to be 1 (5F < *).
(..........)
(..........)
(..........)
Never seen that patent before.
http://www.mediafire.com/?xsrub3f1jgi
EDIT:
Here's JBIG ITU-T recommendation T.82
http://www.mediafire.com/?1291bvw1rkr
Chose to not store a MPS variable (default 0 always).
8 PROB vars (one per bitplane). Updated per bit discarder.
Which gets:
(LASTMPS ~ last bits output)
Code: Select all
// bit 9,13
if( bits_correct==8 || bits_correct==12 )
{
if( BIT(LASTMPS,7) == 1 )
mps_flip = true;
}
// bit 10,14
if( bits_correct==9 || bits_correct==13 )
{
if( BIT(LASTMPS,7) == 1 &&
BIT(LASTMPS,8) == BIT(LASTMPS,0) )
mps_flip = true;
if( BIT(LASTMPS,8) != BIT(LASTMPS,0) )
PROB = 0;
}
// bit 11,15
if( bits_correct==10 || bits_correct==14 )
{
if( BIT(LASTMPS,7) == 1 &&
BIT(LASTMPS,8) == BIT(LASTMPS,0) &&
BIT(LASTMPS,9) == BIT(LASTMPS,1) )
mps_flip = true;
if( BIT(LASTMPS,8) != BIT(LASTMPS,0) ||
BIT(LASTMPS,9) != BIT(LASTMPS,1) )
PROB = 0;
}
// bit 12,16
if( bits_correct==11 || bits_correct==15 )
{
if( BIT(LASTMPS,7) == 1 &&
BIT(LASTMPS,8) == BIT(LASTMPS,0) &&
BIT(LASTMPS,9) == BIT(LASTMPS,1) &&
BIT(LASTMPS,10) == BIT(LASTMPS,2) )
mps_flip = true;
if( BIT(LASTMPS,8) != BIT(LASTMPS,0) ||
BIT(LASTMPS,9) != BIT(LASTMPS,1) ||
BIT(LASTMPS,10) != BIT(LASTMPS,2) )
PROB = 0;
}
// bit 17
if( bits_correct==16 )
{
if( BIT(LASTMPS,7) == 1 ||
BIT(LASTMPS,15) == 1 )
mps_flip = true;
}
Bit 21 does not follow bit 17.
Code: Select all
00100000
00100000
1010
CODE = 5F | CUTOFF = AD (78,AD,C1) | PROB = 01 | MPS = 0 (1,0) | HIGH = D2
bits_correct = 20 | out=14 | decode=AC
1F177E77 / 2020AC9F
(..........)
(..........)
(..........)
Never seen that patent before.
http://www.mediafire.com/?xsrub3f1jgi
EDIT:
Here's JBIG ITU-T recommendation T.82
http://www.mediafire.com/?1291bvw1rkr
Yup that's the idea. We're dealing with the 'adaptive' case.byuu wrote:My understanding was that arithmetic just has a fixed table with a probability percentage of each symbol, similar to huffman. Or the table can be adaptive and calculated on the fly.
I would strongly suggest giving some EE stuff a try. It really isn't that bad, and knowing that you can turn to that if necessary is a useful mindset / skill.byuu wrote:And no EE skills whatsoever. I couldn't solder paperclips together if my life depended on it.
Does your computer have a working parallel port? If so, Caitsith just laid out an updated PCB for our cart dumper. Since your copier doesn't have much passthrough capabilities (I don't know anything about the UFO copiers, so I'll take your word on that), you'll need something like this to communicate with the cartriges anyway. It would be a really nice and fairly simple start, for none of the parts are surface mount.
Hmmm... this may be difficult, I didn't realize you wanted the equipment sent to you.byuu wrote:I was personally looking to get ahold of the test setup so that I could refine the non-compression registers, as these aren't going to be simple to guess like the S-DD1. And then from there attempt a setup to run custom data for anyone who wanted.
I put together the FIFO mod last night and it seems to be working. But since I knew it wouldn't fit in the cartridge when I started, I have the chips hovering spider-like from the wires I soldered to the cartridge. Basically, it don't look pretty, and while I trust it to hold together for the project, I'd feel really hesitant to trust it after packaging and mailing (at the least you'd probably need to touch up some connections).
What you describe in that quote though doesn't require custom decompression. Since you are willing to buy stuff, I'd suggest getting a SF3 from tototek for yourself. They are quite cheap. I can give you a memory viewer program so you can put a cartridge in the external slot and play with the registers to your hearts content.
--------------
EDIT: I debated whether mentioning about the FIFO mod or not. I have several meetings with the people that basically fund my advisors lab this week, and will be a bit too busy to work on other stuff. So yes, I know it is a bit of a tease, but I haven't finished writing up the software to let me "easily" do runs of custom data yet (I'm not using the SF7 so it will never be computer automated). So everyone will have to wait... sorry.
Kammedo will probably be up and running soon anyway.
Andreas, do you feel comfortable programming in assembly language?
The fact that the setup allows custom data on the fly lends itself better to your suggestions than the EPROM method. But we'd need to write a program to take advantage of it. I can of course write the programs for everyone if need be (next week when I have more free time), but having a "middle-man" between you and your data can be frustrating sometimes. So I can share the skeleton of my code so people can make their own custom tests.
--------------
EDIT(2): By the way, /XIN of the FIFO needs to be grounded (not tied to Vcc) if anyone decides to do the mod themselves. If you want it to be compatible with my setup, I also chose FIFO write to occur on /WR low and BA7 high (I did end up using a 74'138 chip).
-
Andreas Naive
- Posts: 104
- Joined: Mon Nov 26, 2007 2:06 am
- Location: Madrid, Spain
- Contact:
Don't worry. Real life has to get his priorities.So everyone will have to wait... sorry.
Last time i programmed in assembler was about 11 years ago, and for the 80x86. But yes, at that epoch i programmed a lot in assembler and, if needed i could probably learn the neccesary about 65x816 in some days place if you provide "skeleton" functions to manage the hardware. If you prefer we go that way, i should start inmediately. Could you suggest some reasonable cross-assemblers/emulators for me to make tests in a PC?Andreas, do you feel comfortable programming in assembly language?