Is there an assembler that supports unit testing?
Moderator: Moderators
Re: Is there an assembler that supports unit testing?
Mesen has a mode that lets you launch a rom and a lua script test harness, and automatically exit and return results. It can be useful for automated testing as well. You'd have to dig through the docs though, I don't remember the flags for it.
My games: http://www.bitethechili.com
-
- Posts: 102
- Joined: Fri Dec 27, 2013 4:28 pm
Re: Is there an assembler that supports unit testing?
I've had playing with this on my to-do list forever - in my case all I really want is a sanity check that the rom starts, and gets past the title screen without crashing. (Okay, that's a bit more of an integration test, but hey, we have almost nothing right now!)
Here's the documentation for starting in testrunner mode, since it's a little hard to find: Mesen Docs
One easy thing to do is to write a known value to a known memory location from your rom, then validate that this value was written using the emulator. This proves that we got to the location that writes this value successfully. Let's do that! (Disclaimer: there's always a slight chance something else wrote the correct value if things go wrong)
In my case, I want to prove that we got past the start menu in my game. The game has an intro screen that shows for a set number of frames before you can hit start, then a main start menu. I'll insert a write to a location in ram I'm not otherwise using (0x02fe) right after I detect that second start press. Here's what that looks like from my C program. (Though I'm sure you can guess the assembly version!)
Alright, that's actually the only thing we need to do to our game! (Note: if we wanted to avoid any changes to the game, we could have used gameState, after putting it in a known spot)
Now, we need to test it. Here's a lua script to do so that I tried to comment pretty heavily. (Call it "sanity.lua")
What this does, in brief, is run a function every time the nes requests input that checks how many frames have been drawn, then decides to either press start if enough frames have passed, or check the memory location we set and exit the emulator with either a success or failure return code.
Now if we run the script, the exit code will tell us whether the rom worked:
(Note: this is shell scripting for sh/bash. For windows cmd you'll have to do things in a .bat file and be a little more clever)
If I change the rom to be broken by adding an infinite loop after the intro screen shows, the result changes!
It's not super pretty, but you could stick this in an automated release process to make sure your rom isn't completely busted. I'd bet with a little wrapper and a few more features, this could be turned into a small test library.
I've attached a zip of my script and two test roms, as well as shell and bash scripts to run it and show a result, in case it helps anyone! (Fair warning: the title on the test rom is just a grass tile repeated a thousand times - this is expected)
Edit: I just realized that the address I picked is part of the sprite bank for most games - I was aiming for the software stack at 0x300 but I wasn't thinking. Another ram address might be better.
----------
nes-test
I did the thing. There's a repo for this now. https://gh.nes.science/nes-test
It's a standalone binary that can run tests written in jasmine. Those look like this:
It's very barebones, but it works, and supports both windows and linux. If you try it, let me know what you think!
----------
Update 2 As of version 0.2.0, the tool supports reading symbols from .dbg files, so you can test like this:
(If you use c, getRamByteFromC exists too, and works the same way!)
Here's the documentation for starting in testrunner mode, since it's a little hard to find: Mesen Docs
One easy thing to do is to write a known value to a known memory location from your rom, then validate that this value was written using the emulator. This proves that we got to the location that writes this value successfully. Let's do that! (Disclaimer: there's always a slight chance something else wrote the correct value if things go wrong)
In my case, I want to prove that we got past the start menu in my game. The game has an intro screen that shows for a set number of frames before you can hit start, then a main start menu. I'll insert a write to a location in ram I'm not otherwise using (0x02fe) right after I detect that second start press. Here's what that looks like from my C program. (Though I'm sure you can guess the assembly version!)
Code: Select all
case GAME_STATE_TITLE_INPUT:
wait_for_start();
__asm__("lda #23");
__asm__("sta $02fe");
gameState = GAME_STATE_POST_TITLE;
Now, we need to test it. Here's a lua script to do so that I tried to comment pretty heavily. (Call it "sanity.lua")
Code: Select all
-- Sanity test our rom to make sure it works.
-- Number of frames to wait after system start to press start to skip the intro screen
firstStartPressFrames = 100
firstStartPressDone = false
-- Number of frames to wait after system start before pressing start to skip the title screen
secondStartPressFrames = 140
secondStartPressDone = false
-- Number of frames to wait to see if the value was written (also from system start)
valueTestFrames = 150
-- expected value we're writing, as well as the address.
expectedMemoryLocation = 0x2fe
expectedMemoryLocationValue = 23
-- Called whenever input is polled (Probably 1x/frame, but games sometimes poll more often)
function doInput()
-- Grab frame count from the emulator
currentFrameCount = emu.getState().ppu.frameCount
-- Check if our first start press should happen yet
if (currentFrameCount > firstStartPressFrames and firstStartPressDone == false) then
firstStartPressDone = true
emu.setInput(0, {start = true})
elseif (currentFrameCount > secondStartPressFrames and secondStartPressDone == false) then
secondStartPressDone = true
emu.setInput(0, {start = true})
elseif (currentFrameCount > valueTestFrames) then
-- The emulator will try to run our test immediately unless we put it in a callback, so we put it here for brevity.
-- Check what value is written at our address.
memoryValue = emu.read(expectedMemoryLocation, emu.memType.cpuDebug)
-- Set the exit code based on whether or not the memory value matches our expectation
if (memoryValue == expectedMemoryLocationValue) then
emu.stop(0)
else
emu.stop(1)
end
end
-- No need to set input otherwise, it is automatically cleared out by the emulator
end
-- Set the input function to run every time input is polled
emu.addEventCallback(doInput, emu.eventType.inputPolled)
Now if we run the script, the exit code will tell us whether the rom worked:
(Note: this is shell scripting for sh/bash. For windows cmd you'll have to do things in a .bat file and be a little more clever)
Code: Select all
$ ./mesen --testrunner rom/working.nes tests/sanity.lua
$ echo "Test result: $? (0 is success, 1 is failure)"
Test result: 0 (0 is success, 1 is failure)
Code: Select all
$ ./mesen --testrunner rom/broken.nes tests/sanity.lua
$ echo "Test result: $? (0 is success, 1 is failure)"
Test result: 1 (0 is success, 1 is failure)
I've attached a zip of my script and two test roms, as well as shell and bash scripts to run it and show a result, in case it helps anyone! (Fair warning: the title on the test rom is just a grass tile repeated a thousand times - this is expected)
Edit: I just realized that the address I picked is part of the sprite bank for most games - I was aiming for the software stack at 0x300 but I wasn't thinking. Another ram address might be better.
----------
nes-test
I did the thing. There's a repo for this now. https://gh.nes.science/nes-test
It's a standalone binary that can run tests written in jasmine. Those look like this:
It's very barebones, but it works, and supports both windows and linux. If you try it, let me know what you think!
----------
Update 2 As of version 0.2.0, the tool supports reading symbols from .dbg files, so you can test like this:
(If you use c, getRamByteFromC exists too, and works the same way!)
Last edited by cppchriscpp on Fri Jan 26, 2024 9:54 pm, edited 1 time in total.
Re: Is there an assembler that supports unit testing?
I unit test collision detection in my game, using Mesen.
As others pointed out, it can run in fast headless mode and supports Lua scripting.
At some point when you'll be writing these tests you'll find out the test code is pretty repeatable.
Instead of typing the same stuff over and over, I've simply written a single lua script, which parses test files written in human-like language and changes them to emulator instructions.
So my tests look smth like:
On every test, I pass the name of the test text file as an SCENARIO_FILE environment variable that the Lua script reads.
Rest is pretty much the same as folks have already presented. I hook move events to emu.eventType.inputPolled and have to export dbg from ld65 to use humanly readable memory locations.
Sometimes headless is not good enough, as you want to see what is going on the screen, therefore I have added the option to take a screenshot every frame.
For a passed test I'm exiting Mesen with 0 exit code, 1 means failure.
If this is anything interesting to you, script I use looks like this (labelLookup table has to contain memory labels from your rom, if you want to use these in the tests):
As others pointed out, it can run in fast headless mode and supports Lua scripting.
At some point when you'll be writing these tests you'll find out the test code is pretty repeatable.
Instead of typing the same stuff over and over, I've simply written a single lua script, which parses test files written in human-like language and changes them to emulator instructions.
So my tests look smth like:
Code: Select all
test Player1 should walk closer and punch Player2 and reduce his hp to 76
enable screenshots
at 0 move p1 right
at 68 stop p1
at 75 move p1 down
at 76 stop p1
at 77 move p1 down
at 78 stop p1
at 79 move p1 down
at 80 stop p1
at 110 end
expect p2 hp equals 76
Rest is pretty much the same as folks have already presented. I hook move events to emu.eventType.inputPolled and have to export dbg from ld65 to use humanly readable memory locations.
Sometimes headless is not good enough, as you want to see what is going on the screen, therefore I have added the option to take a screenshot every frame.
For a passed test I'm exiting Mesen with 0 exit code, 1 means failure.
If this is anything interesting to you, script I use looks like this (labelLookup table has to contain memory labels from your rom, if you want to use these in the tests):
Code: Select all
function LoadAndParseScenario()
local readPath = os.getenv("SCENARIO_FILE")
local file = assert(io.open(readPath, "r"))
local expectLine
local lines = {}
local outputLines = {}
for line in io.lines(readPath) do
local words = {}
line = string.gsub(line, "[\n\r]", "")
if line:find('expect') == 1 then
expectLine = line
elseif line:find('enable screenshots') == 1 then
screenshots = true
elseif line:find('disable test') == 1 then
disable = true
elseif line:find('output') == 1 then
table.insert(outputLines, line)
else
for word in line:gmatch("%w+") do
table.insert(words, word)
end
table.insert(lines, words)
end
end
file:close()
ParseTextScenario(lines)
ParseExpect(expectLine)
ParseOutputs(outputLines)
end
function ParseTextScenario(lines)
for _, words in pairs(lines) do
if words[1] == "at" then
if words[3] == "move" then
local key = words[5]
local input = keyNamesLookup[key]
table.insert(moves, { at = tonumber(words[2]), joypad = (words[4] == "p1" and 0 or 1), input = input })
elseif words[3] == "stop" then
table.insert(moves, { at = tonumber(words[2]), joypad = (words[4] == "p1" and 0 or 1), input = NO_MOVE })
elseif words[3] == "end" then
table.insert(moves, { at = tonumber(words[2]), joypad = nil, input = nil })
end
end
end
end
function ParseExpect(expectLine)
local expectedLineReminder = string.sub(expectLine, string.len("expect ") + 1)
expectedLineReminder, leftReadInstruction = GetCpuReadInstruction(expectedLineReminder, labelLookup)
expectedLineReminder, operator = GetValueFromLookupTable(expectedLineReminder, operatorLookup)
expectedLineReminder, rightReadInstruction = GetCpuReadInstruction(expectedLineReminder, labelLookup)
end
function GetCpuReadInstruction(expectedLineReminder, lookupTable)
local label = nil
local readInstruction = nil
expectedLineReminder, label = GetValueFromLookupTable(expectedLineReminder, lookupTable)
if label ~= nil then
readInstruction = 'emu.read(' .. label .. ', emu.memType.cpu)'
else
readInstruction = string.match(expectedLineReminder, "^[a-z0-9]+")
expectedLineReminder = string.sub(expectedLineReminder, string.len(readInstruction) + 2)
end
return expectedLineReminder, readInstruction
end
function GetValueFromLookupTable(expectedLineReminder, lookupTable)
local value = nil
for key, v in pairs(lookupTable) do
if expectedLineReminder:find(key) == 1 then
expectedLineReminder = string.sub(expectedLineReminder, string.len(key) + 2)
value = v
break
end
end
return expectedLineReminder, value
end
function CheckScenarioResult()
local test = load('return ' .. rightReadInstruction .. ' ' .. operator .. ' ' .. leftReadInstruction)
emu.stop(test() and 0 or 1)
end
function ParseOutputs(outputLines)
for _, outputLine in pairs(outputLines) do
local tokens = {}
local size = 0
for t in string.gmatch(outputLine, "_*%w+") do
table.insert(tokens, t)
size = size + 1
end
if size == 2 then
table.insert(outputs, {label = tokens[2], offset_start=0, offset_end=0})
elseif size == 3 then
table.insert(outputs, {label = tokens[2], offset_start=tonumber(tokens[3]), offset_end=tonumber(tokens[3])})
else
table.insert(outputs, {label = tokens[2], offset_start=tonumber(tokens[3]), offset_end=tonumber(tokens[4])})
end
end
end
function PrintOutputs()
local file = io.open(os.getenv("SCENARIO_OUTPUT") .. "/outputs.txt", "w")
for _, output in pairs(outputs) do
file:write(output.label .. ':')
for offset=output.offset_start, output.offset_end do
emu.log(output.label)
file:write(' ' .. emu.read(emu.getLabelAddress(output.label) + offset, emu.memType.cpu))
end
file:write('\n')
end
file:close()
end
function NextMove()
local nextInput = nil
local joypad = 0
for _, move in pairs(moves) do
if counter >= move.at then
nextInput = move.input
joypad = move.joypad
else
break
end
end
if nextInput ~= nil then
emu.setInput(joypad, nextInput)
else
PrintOutputs()
CheckScenarioResult()
end
counter = counter + 1
end
function SaveScreenshot()
local file = io.open(os.getenv("SCENARIO_OUTPUT") .. "/" .. frameCounter .. ".png", "wb")
file:write(emu.takeScreenshot())
file:close()
frameCounter = frameCounter + 1
end
NO_MOVE = {
a = false, b = false, start = false, stop = false,
up = false, down = false, left = false, right = false
}
keyNamesLookup = {
["up"] = { up = true },
["down"] = { down = true },
["right"] = { right = true },
["left"] = { left = true },
["a"] = { a = true },
["b"] = { b = true },
["start"] = { start = true },
["stop"] = { stop = true },
}
operatorLookup = {
["equals"] = "==",
["not equals"] = "~=",
}
labelLookup = {
["p1 hp"] = 'emu.getLabelAddress("_player1Status")',
["p2 hp"] = 'emu.getLabelAddress("_player2Status")',
}
moves = {}
outputs = {}
counter = 0
screenshots = false
disable = false
frameCounter = 0
LoadAndParseScenario()
if disable then
emu.stop(0)
end
if screenshots then
emu.addEventCallback(SaveScreenshot, emu.eventType.endFrame)
end
emu.addEventCallback(NextMove, emu.eventType.inputPolled)
Re: Is there an assembler that supports unit testing?
I have found out another solution here: https://github.com/AsaiYusuke/6502_test_executor
It still doesn't fit all cases, but its easy to use.
It still doesn't fit all cases, but its easy to use.