Is there an assembler that supports unit testing?

Discuss technical or other issues relating to programming the Nintendo Entertainment System, Famicom, or compatible systems. See the NESdev wiki for more information.

Moderator: Moderators

User avatar
gauauu
Posts: 779
Joined: Sat Jan 09, 2016 9:21 pm
Location: Central Illinois, USA
Contact:

Re: Is there an assembler that supports unit testing?

Post by gauauu »

Mesen has a mode that lets you launch a rom and a lua script test harness, and automatically exit and return results. It can be useful for automated testing as well. You'd have to dig through the docs though, I don't remember the flags for it.
cppchriscpp
Posts: 102
Joined: Fri Dec 27, 2013 4:28 pm

Re: Is there an assembler that supports unit testing?

Post by cppchriscpp »

I've had playing with this on my to-do list forever - in my case all I really want is a sanity check that the rom starts, and gets past the title screen without crashing. (Okay, that's a bit more of an integration test, but hey, we have almost nothing right now!)

Here's the documentation for starting in testrunner mode, since it's a little hard to find: Mesen Docs

One easy thing to do is to write a known value to a known memory location from your rom, then validate that this value was written using the emulator. This proves that we got to the location that writes this value successfully. Let's do that! (Disclaimer: there's always a slight chance something else wrote the correct value if things go wrong)



In my case, I want to prove that we got past the start menu in my game. The game has an intro screen that shows for a set number of frames before you can hit start, then a main start menu. I'll insert a write to a location in ram I'm not otherwise using (0x02fe) right after I detect that second start press. Here's what that looks like from my C program. (Though I'm sure you can guess the assembly version!)

Code: Select all

            case GAME_STATE_TITLE_INPUT:
                wait_for_start();
                __asm__("lda #23");
                __asm__("sta $02fe");
                gameState = GAME_STATE_POST_TITLE;

Alright, that's actually the only thing we need to do to our game! (Note: if we wanted to avoid any changes to the game, we could have used gameState, after putting it in a known spot)

Now, we need to test it. Here's a lua script to do so that I tried to comment pretty heavily. (Call it "sanity.lua")

Code: Select all

-- Sanity test our rom to make sure it works.

-- Number of frames to wait after system start to press start to skip the intro screen
firstStartPressFrames = 100
firstStartPressDone = false
-- Number of frames to wait after system start before pressing start to skip the title screen
secondStartPressFrames = 140
secondStartPressDone = false
-- Number of frames to wait to see if the value was written (also from system start)
valueTestFrames = 150

-- expected value we're writing, as well as the address.
expectedMemoryLocation = 0x2fe
expectedMemoryLocationValue = 23



-- Called whenever input is polled (Probably 1x/frame, but games sometimes poll more often)
function doInput()
    -- Grab frame count from the emulator
    currentFrameCount = emu.getState().ppu.frameCount

    -- Check if our first start press should happen yet
    if (currentFrameCount > firstStartPressFrames and firstStartPressDone == false) then
        firstStartPressDone = true
        emu.setInput(0, {start = true})
    elseif (currentFrameCount > secondStartPressFrames and secondStartPressDone == false) then
        secondStartPressDone = true
        emu.setInput(0, {start = true})
    elseif (currentFrameCount > valueTestFrames) then
        -- The emulator will try to run our test immediately unless we put it in a callback, so we put it here for brevity.
       
        -- Check what value is written at our address.
        memoryValue = emu.read(expectedMemoryLocation, emu.memType.cpuDebug)

        -- Set the exit code based on whether or not the memory value matches our expectation
        if (memoryValue == expectedMemoryLocationValue) then
            emu.stop(0)
        else
            emu.stop(1)
        end

    end
    -- No need to set input otherwise, it is automatically cleared out by the emulator
end

-- Set the input function to run every time input is polled
emu.addEventCallback(doInput, emu.eventType.inputPolled)
What this does, in brief, is run a function every time the nes requests input that checks how many frames have been drawn, then decides to either press start if enough frames have passed, or check the memory location we set and exit the emulator with either a success or failure return code.

Now if we run the script, the exit code will tell us whether the rom worked:

(Note: this is shell scripting for sh/bash. For windows cmd you'll have to do things in a .bat file and be a little more clever)

Code: Select all

$ ./mesen --testrunner rom/working.nes tests/sanity.lua
$ echo "Test result: $? (0 is success, 1 is failure)"
Test result: 0 (0 is success, 1 is failure)
If I change the rom to be broken by adding an infinite loop after the intro screen shows, the result changes!

Code: Select all

$ ./mesen --testrunner rom/broken.nes tests/sanity.lua
$ echo "Test result: $? (0 is success, 1 is failure)"
Test result: 1 (0 is success, 1 is failure)
It's not super pretty, but you could stick this in an automated release process to make sure your rom isn't completely busted. I'd bet with a little wrapper and a few more features, this could be turned into a small test library.

I've attached a zip of my script and two test roms, as well as shell and bash scripts to run it and show a result, in case it helps anyone! (Fair warning: the title on the test rom is just a grass tile repeated a thousand times - this is expected)
nes-testing.zip
(29.92 KiB) Downloaded 37 times
Edit: I just realized that the address I picked is part of the sprite bank for most games - I was aiming for the software stack at 0x300 but I wasn't thinking. Another ram address might be better.

----------

nes-test

I did the thing. There's a repo for this now. https://gh.nes.science/nes-test

It's a standalone binary that can run tests written in jasmine. Those look like this:

Image

It's very barebones, but it works, and supports both windows and linux. If you try it, let me know what you think!

----------

Update 2 As of version 0.2.0, the tool supports reading symbols from .dbg files, so you can test like this:

Image

(If you use c, getRamByteFromC exists too, and works the same way!)
Last edited by cppchriscpp on Fri Jan 26, 2024 9:54 pm, edited 1 time in total.
User avatar
RJM
Posts: 55
Joined: Mon Jul 27, 2020 11:56 am
Location: Rzeszów, Poland

Re: Is there an assembler that supports unit testing?

Post by RJM »

I unit test collision detection in my game, using Mesen.
As others pointed out, it can run in fast headless mode and supports Lua scripting.
At some point when you'll be writing these tests you'll find out the test code is pretty repeatable.
Instead of typing the same stuff over and over, I've simply written a single lua script, which parses test files written in human-like language and changes them to emulator instructions.

So my tests look smth like:

Code: Select all

test Player1 should walk closer and punch Player2 and reduce his hp to 76

enable screenshots

at 0 move p1 right
at 68 stop p1
at 75 move p1 down
at 76 stop p1
at 77 move p1 down
at 78 stop p1
at 79 move p1 down
at 80 stop p1
at 110 end

expect p2 hp equals 76
On every test, I pass the name of the test text file as an SCENARIO_FILE environment variable that the Lua script reads.
Rest is pretty much the same as folks have already presented. I hook move events to emu.eventType.inputPolled and have to export dbg from ld65 to use humanly readable memory locations.
Sometimes headless is not good enough, as you want to see what is going on the screen, therefore I have added the option to take a screenshot every frame.
For a passed test I'm exiting Mesen with 0 exit code, 1 means failure.

If this is anything interesting to you, script I use looks like this (labelLookup table has to contain memory labels from your rom, if you want to use these in the tests):

Code: Select all

function LoadAndParseScenario()
    local readPath = os.getenv("SCENARIO_FILE")
    local file = assert(io.open(readPath, "r"))
    local expectLine
    local lines = {}
    local outputLines = {}

    for line in io.lines(readPath) do
        local words = {}
        line = string.gsub(line, "[\n\r]", "")

        if line:find('expect') == 1 then
            expectLine = line
        elseif line:find('enable screenshots') == 1 then
            screenshots = true
        elseif line:find('disable test') == 1 then
            disable = true
        elseif line:find('output') == 1 then
            table.insert(outputLines, line)
        else
            for word in line:gmatch("%w+") do
                table.insert(words, word)
            end
            table.insert(lines, words)
        end
    end

    file:close()

    ParseTextScenario(lines)
    ParseExpect(expectLine)
    ParseOutputs(outputLines)
end

function ParseTextScenario(lines)
    for _, words in pairs(lines) do
        if words[1] == "at" then
            if words[3] == "move" then
                local key = words[5]
                local input = keyNamesLookup[key]
                table.insert(moves, { at = tonumber(words[2]), joypad = (words[4] == "p1" and 0 or 1), input = input })
            elseif words[3] == "stop" then
                table.insert(moves, { at = tonumber(words[2]), joypad = (words[4] == "p1" and 0 or 1), input = NO_MOVE })
            elseif words[3] == "end" then
                table.insert(moves, { at = tonumber(words[2]), joypad = nil, input = nil })
            end
        end
    end
end

function ParseExpect(expectLine)
    local expectedLineReminder = string.sub(expectLine, string.len("expect ") + 1)

    expectedLineReminder, leftReadInstruction = GetCpuReadInstruction(expectedLineReminder, labelLookup)
    expectedLineReminder, operator = GetValueFromLookupTable(expectedLineReminder, operatorLookup)
    expectedLineReminder, rightReadInstruction = GetCpuReadInstruction(expectedLineReminder, labelLookup)

end

function GetCpuReadInstruction(expectedLineReminder, lookupTable)
    local label = nil
    local readInstruction = nil

    expectedLineReminder, label = GetValueFromLookupTable(expectedLineReminder, lookupTable)

    if label ~= nil then
        readInstruction = 'emu.read(' .. label .. ', emu.memType.cpu)'
    else
        readInstruction = string.match(expectedLineReminder, "^[a-z0-9]+")
        expectedLineReminder = string.sub(expectedLineReminder, string.len(readInstruction) + 2)
    end

    return expectedLineReminder, readInstruction
end

function GetValueFromLookupTable(expectedLineReminder, lookupTable)
    local value = nil

    for key, v in pairs(lookupTable) do
        if expectedLineReminder:find(key) == 1 then
            expectedLineReminder = string.sub(expectedLineReminder, string.len(key) + 2)
            value = v
            break
        end
    end

    return expectedLineReminder, value
end

function CheckScenarioResult()
    local test = load('return ' .. rightReadInstruction .. ' ' .. operator .. ' ' .. leftReadInstruction)
    emu.stop(test() and 0 or 1)
end

function ParseOutputs(outputLines)
    for _, outputLine in pairs(outputLines) do
        local tokens = {}
        local size = 0

        for t in string.gmatch(outputLine, "_*%w+") do
           table.insert(tokens, t)
           size = size + 1
        end

        if size == 2 then
            table.insert(outputs, {label = tokens[2], offset_start=0, offset_end=0})
        elseif size == 3 then
            table.insert(outputs, {label = tokens[2], offset_start=tonumber(tokens[3]), offset_end=tonumber(tokens[3])})
        else
            table.insert(outputs, {label = tokens[2], offset_start=tonumber(tokens[3]), offset_end=tonumber(tokens[4])})
        end

    end
end

function PrintOutputs()
    local file = io.open(os.getenv("SCENARIO_OUTPUT") .. "/outputs.txt", "w")
    for _, output in pairs(outputs) do
        file:write(output.label .. ':')
        for offset=output.offset_start, output.offset_end do
          emu.log(output.label)
             file:write(' ' .. emu.read(emu.getLabelAddress(output.label) + offset, emu.memType.cpu))
        end
        file:write('\n')
    end
    file:close()
end

function NextMove()
    local nextInput = nil
    local joypad = 0

    for _, move in pairs(moves) do
        if counter >= move.at then
            nextInput = move.input
            joypad = move.joypad
        else
            break
        end
    end

    if nextInput ~= nil then
        emu.setInput(joypad, nextInput)
    else
        PrintOutputs()
        CheckScenarioResult()
    end

    counter = counter + 1
end

function SaveScreenshot()
    local file = io.open(os.getenv("SCENARIO_OUTPUT") .. "/" .. frameCounter .. ".png", "wb")
    file:write(emu.takeScreenshot())
    file:close()
    frameCounter = frameCounter + 1
end
  

NO_MOVE = {
    a = false, b = false, start = false, stop = false,
    up = false, down = false, left = false, right = false
}

keyNamesLookup = {
    ["up"] = { up = true },
    ["down"] = { down = true },
    ["right"] = { right = true },
    ["left"] = { left = true },
    ["a"] = { a = true },
    ["b"] = { b = true },
    ["start"] = { start = true },
    ["stop"] = { stop = true },
}

operatorLookup = {
    ["equals"] = "==",
    ["not equals"] = "~=",
}

labelLookup = {
    ["p1 hp"] = 'emu.getLabelAddress("_player1Status")',
    ["p2 hp"] = 'emu.getLabelAddress("_player2Status")',
}

moves = {}

outputs = {}

counter = 0

screenshots = false

disable = false

frameCounter = 0

LoadAndParseScenario()

if disable then
    emu.stop(0)
end

if screenshots then
    emu.addEventCallback(SaveScreenshot, emu.eventType.endFrame)
end

emu.addEventCallback(NextMove, emu.eventType.inputPolled)

asai
Posts: 1
Joined: Sun Mar 27, 2022 12:03 am

Re: Is there an assembler that supports unit testing?

Post by asai »

I have found out another solution here: https://github.com/AsaiYusuke/6502_test_executor
It still doesn't fit all cases, but its easy to use.
Post Reply