Resources over usb2snes

Discussion of hardware and software development for Super NES and Super Famicom. See the SNESdev wiki for more information.

Moderator: Moderators

Forum rules
  • For making cartridges of your Super NES games, see Reproduction.
Post Reply
UnDisbeliever
Posts: 124
Joined: Mon Mar 02, 2015 1:11 am
Location: Australia (PAL)
Contact:

Resources over usb2snes

Post by UnDisbeliever »

Over the last month I added a communication link between my unnamed-snes-game and my PC using usb2snes. This link allows me to transfer resources and code from my PC to my SNES console on demand and quickly test any changes by simply loading a new room.

Here is a video of it in action.

I figured I'd write a brief overview about how I achieved this feat.


usb2snes is an alternative firmware for the sd2snes/FXPAK by RedGuyyyy (now official as of FXPAK Firmware v1.11.0 beta 1). It uses the USB socket on a FXPAK and allows computers to remotely:
  • Access the files on the SD-card
  • Boot a game from the SD-card
  • Reset the console
  • Read ROM, Cart-RAM and Work-RAM while a game is running
  • Write to ROM and Cart-RAM while a game is running
  • Write to Work-RAM via code injection (some libraries/toolkits do not support this)
QUsb2Snes is a websocket server that provides a websocket API for the usb2snes firmware and a few emulators. (Link to protocol)



The first step was to refactor my code so that all resources (rooms, tiles, MetaTiles, MetaSprites, etc) are accessed through a load_resource or load_room function and only one resource is accessed at a time. When I wanted to load multiple resources, important data had to copied to RAM before the next resource is requested.

Since usb2snes does not allow me to directly send data to my PC, I took a page out of SPC communications. When the game wants to load a resource, it writes the following struct to Work-RAM. To signify this is a new request, the request_id is incremented (skipping 0) and written last.

Code: Select all

struct Request {
    request_id   : u8,
    request_type : u8,
    resource_id  : u8,
};
A python script uses a usb2snes GetAddress command to read the request bytes from Work-RAM 10 times a second, waiting for the resource_id byte to change. When it sees a new request, the corresponding compiled resource data is retrieved from a data store. A usb2snes PutAddress command copies the resource data to a fixed ROM address (the beginning of the first free data bank). Finally, another usb2snes PutAddress command writes a 4 byte Response struct to ROM.

Code: Select all

enum ResponseStatus : u8 {
    NOT_CONNECTED        = 0,
    OK                   = 0x20,
    OK_RESOURCES_CHANGED = 0x21, // Only `room` requests return this response.
    INIT_OK              = 0xbb, // Only `init` requests return this response.
    NOT_FOUND            = 0x40,
    ERROR                = 0xff,
};

struct Response {
    data_size    : u16,
    status       : ResponseStatus,
    response_id  : u8,
};
While this happening, the game will repeatedly read response_id, waiting for it to match request_id. If status is OK, the load_resource function will return a (far pointer, size) struct and the game runs as normal.


After I got the game to successfully retrieve resources over usb2snes, I added a filesystem monitor to detect any changes to the resource files and recompile them in a background thread. If there an error compiling a resource and the game requests that resource, the system will wait until the error has been fixed before transferring it (forcing me to fix the errors immediately).

There were two data structures I could not easily convert to a request-response model, the entity data and MetaSprite table data (as they both depended on the compiled output of all MetaSprite resources). These data structures are accessed via ROM addresses (like normal) and are updated by the python script whenever a room or MetaSprite resource was requested. I dedicated an entire 64KiB bank to the MetaSprite Table data as its size can vary wildly every time a MetaSprite is changed.


I also added a method for the python script to update the game code. When resources_over_usb2snes.py starts, it reads the ROM code running on the device and compares it to the .sfc file on my PC. If they do not match, it sets ROM address 0x00ffb9 (inside the reserved bytes of the ROM Header) to 0x42 and resets the console. When the game resets, it checks 0x00ffb9 and if is non-zero; copies and executes the following code in Work-RAM (with interrupts and HDMA disabled).

Code: Select all

// 8 bit A, 8 bit Index, DP = 0, DB = 0
Loop:
    lda a:$ffb9
    sta z:0,x
    inx
    bra Loop
This is a spin-loop that will repeatedly write the value in 0x00ffb9 to every byte in the zero-page. The python script will use a usb2snes GetAddress command to read the zero-page and wait until it matches 0xffb9. To ensure this is not a fluke, the python script will repeat this test with a few different byte values.

Once the system has confirmed the game is executing the spin-loop (and it is safe to write to ROM), the new build will be transferred to cart ROM (not the SD-card) and a usb2snes Reset command will reset the console.
brizzo
Site Admin
Posts: 34
Joined: Mon Feb 03, 2014 1:08 am

Re: Resources over usb2snes

Post by brizzo »

That is next level awesome. Thank you for taking the time to share details about your workflow :)
Pokun
Posts: 2681
Joined: Tue May 28, 2013 5:49 am
Location: Hokkaido, Japan

Re: Resources over usb2snes

Post by Pokun »

Well I can only agree! Good job!
UnDisbeliever
Posts: 124
Joined: Mon Mar 02, 2015 1:11 am
Location: Australia (PAL)
Contact:

Re: Resources over usb2snes

Post by UnDisbeliever »

Last week, I completed a major refactor to the resources compiler of my unnamed-snes-game. Now resource-over-usb2snes and the release-build use the same code paths for compiling resources.

While I was working on my tooling, I added something I've been meaning to do but hadn't found the motivation for; a GUI to resources-over-usb2snes.
errors-tab.png
This GUI shows any resource compile errors that occurred when compiling a resource. Thanks to the magic of background threads and python-watchdog, the errors tab will update whenever a source file changes.

Most importantly, I see an immediate visual representation about which tiles (in the source image) cannot be converted to SNES tiles. This is a dramatic improvement over the old method, which involved scrolling through the console to find the error message and manually converting tile indexes to image positions.

Code: Select all

File Changed: images/game-over.aseprite
    make images/game-over.png
aseprite --batch "images/game-over.aseprite" --save-as "images/game-over.png"
File Changed: images/game-over.png
Compiling bg_images[1] game_over
ERROR: bg_images[1] game_over: Cannot find palette for 44 8px tiles in images/game-over.png:
     517,  518,  519,  530,  531,  532,  549,  551,  552,  553,  554,  555,  556,  557,  558,  562
     564,  566,  567,  568,  569,  581,  582,  583,  584,  585,  586,  587,  588,  589,  590,  594
     595,  596,  597,  598,  599,  600,  601,  629,  630,  631,  632,  633

I have also added commands to resources-over-usb2snes. The GUI can now send a command (a u8 command enum with an optional blob of data) to the console and the game will execute it. The communication protocol is similar to the protocol for requesting resource data, just in reverse.

Currently I have three commands:
  • null - used for syncing the command_id between the GUI and console.
  • common_audio_data_changed - notifies the console that the common audio data (samples and sound effects) have changed and will need to be reloaded at a later time.
  • upload_song - transfer the data in the command to Audio-RAM and play it.

This allowed me to create a GUI that can be used to rapidly test sound-effect bytecode with my audio engine (twitter video of the GUI in action).
sound-effect-tab.png
Post Reply