za909 wrote:I have illegal instructions in lines 7 and 9.
That's because you tried to use an operand with the RTS operator, but RTS doesn't take any operators. You probably meant to use JSR here, to call the same subroutine twice. OK, now let's see what else is wrong:
The actual joyinit looks correct, but unlike your first call to it suggests, it doesn't return any results in the accumulator, it stores the controller bits directly into the this_frame variable. This means that there's no point in having "sta this_frame" after the call. If the bits are already in this_frame, you're actually trashing them with the STA. Which leads to another problem:
If the joyinit routine is overwriting the contents of this_frame, you have to back that up to prev_frame BEFORE calling it, otherwise you'll lose the previous frame's bits. Just do the "lda this_frame; sta prev_frame" before the calls to joyinit. Similarly, you will trash the first call's bits if you don't back them up before the second call to joyinit. You don't seem to be using the Y register, so that seems like a good place to temporarily put the bits in (put "ldy this_frame" between the two calls to joyinit).
Now, after the second call, you already have the previous frame's bits in prev_frame, this frame's first read in Y and this frame's second read in this_frame. There's no need to store and load a bunch of stuff like you did, you can simply compare Y to this_frame with "cpy this_frame". Now, the original suggestion was that you used the previous frame's bits in case the new reads didn't agree, but it seem you are trying to do the two reads again. Both are possible solution, and if you want to try again you have to jump back to the part after you have backed up last frame's bits, otherwise you'll trash them (if you think about it, you won't even the need the previous frame's bits y=if you're reading the joypad over and over, so you can get rid of everything related to prev_frame). If you just want to use the old bits, you can "lda prev_frame; sta this frame". I see no need for the usable_frame variable, because the contents of this_frame will be the usable data.
I guess this is all I can see wrong with your code. I have to agree that it looks very chaotic, and that you haven't planned the execution order very well in your head. Keep in mind that instructions in a program are executed in order, so it's easy to do a preliminary run of it in your head to see whether variables are being overwritten and where all the data is going when you move it around.