NES to USB Keyboard interface?

[bitcores]

If the hardware you build can do both, choosing a default is probably situational.

My intent with the inclusion of an absolute mode was to make it easy to integrate mouse support that works seamlessly to move between the emulator and surrounding tools. So, in an emulated window, absolute is likely the best default, but in other situations relative is probably better.

This interface can do both modes, though I haven't actually coded relative handling into my test program (you can see the cursor move relatively to 0,0 before it snaps back) and it is just an option in the host python script.
This makes me wonder if there is a suitable input device for emulator for testing in the first place. I'm not entirely sure how the SNES Mouse works in Mesen (I assume on D0), but a compatible input device might need to be made for emulators as well.

Otherwise, it seems fine if the mouse/keyboard present bits can change at runtime to indicate connect/disconnects. Software should generally be able to respond to this, The bit is just indicates whether either is worth polling/responding to, and that's easily something that can be decided on a per-frame basis. It could also be used to relay the device connection information to the user.

Alternatively, if it's easier to implement, there's not really anything wrong with the host always sending 1s for mouse/keyboard present regardless.

So, I think best practice for software is just that if it sees the 110 device signature at boot, it should expect that host device will be connected for the lifetime of the program, and be ready to poll and respond to either a mouse and keyboard, on any frame they report as present.

I wasn't really thinking about the interface itself disconnecting or disappearing while a program is running, but rather whether it would acceptable behavior for the keyboard or mouse present flags to change while the NES program is running.
So long as the expectation is that the program would check them per frame and either prompt the user to connect a device or whatever, then I agree the best behavior would be for this interface to have the device signature available ASAP on start up and present the devices as not present until it gets data from the USB host to say otherwise. I'll have to check when the Pico is powered by the expansion port how quickly that becomes available after boot, I assume before the PPU is ready would be ideal.

If you think the device should be permanently attached, with most games just using the controller as pass-through, it might be nice to have an on/off switch that could disable the device signature.

If powering the interface externally, that would be as simple as just not plugging it in. But if powering it from an expansion port or controller port it would be easy enough to add a switch or jumper to manage that.

[rainwarrior]

This makes me wonder if there is a suitable input device for emulator for testing in the first place. I'm not entirely sure how the SNES Mouse works in Mesen (I assume on D0), but a compatible input device might need to be made for emulators as well.

I'm not sure what you mean by input device in this case? PC emulators can implement this using the keyboard and mouse that are already attached to that computer.

If you're asking whether this can be tested in an existing emulator, I'm not aware of one that has it at the moment, but I'm sure it wouldn't be too difficult to write some support for it in Mesen. I'll take a look at your test ROM.

SNES Mouse in Mesen can be attached on D0 or D1 (on a Famicom it has to go on D1 through an expansion port adapter), though the Zapper is probably a better model for how absolute coordinates are handled, and the FamilyBasicKeyboard for keyboard input.

[bitcores]

There's not really any shared code between reading from FamilyBasicKeyboard and Zapper, and this serialized kb/m input so it wouldn't help much in implementing it, but a developer may want to support those devices as well as alternative inputs so they could at least build their program around those devices and then add support in.

On a side note, I discovered that when powering my Pico interface from the expansion port, it will become ready to respond with the 110 device signature between cycles 30,000 and 55,000. In my code this is after clearing system RAM and waiting for the second vblank, and before I have finished clearing nametable memory and loading tiles into "CHR-RAM". So at least with current code and compile options, so long as a developer doesn't poll for the device until after most of the reset maintenance it should be ready.
Of course, if powered externally it would be ready from cycle 1, same as when the console is reset. I'll look for ways to speed this up, though, because I think being powered by the console is a more elegant solution, rather than having to add more wires around. But it definitely rules out the possibility of powering my Pico driven cartridges from the console.

[maxalmond2]

Hi, this is all well over my head but I'm curious as to whether you ever got the "playing NES on a keyboard" thing working.

I play an NES game on emulator on a keyboard. From what I hear the console, CRT and controller setup has significantly less lag, but I'm unable to use a controller and was thinking the ideal setup would be keyboard, console, and CRT.

Naively I thought maybe I could just somehow rewire NES controller buttons into a keyboard - but scanning this thread it seems waaaaaay more complex than that.

[rainwarrior]

Naively I thought maybe I could just somehow rewire NES controller buttons into a keyboard - but scanning this thread it seems waaaaaay more complex than that.

If you want to use a keyboard as an NES controller, you definitely could directly wire 8 keyboard key switches to an NES controller circuit, and they would work just like the controller buttons/d-pad.

What makes the stuff discussed in this thread more complicated is not wanting to modify the keyboard, and wanting to use all of its keys.

[maxalmond2]

If you want to use a keyboard as an NES controller, you definitely could directly wire 8 keyboard key switches to an NES controller circuit, and they would work just like the controller buttons/d-pad.

Thanks. That's exactly what I was hoping to hear.

[rainwarrior]

I've created a fork of Mesen2 that implements the proposed interface.

Source code: GitHub: bbbradsmith/Mesen2/tree/kbm (code changes)
Windows Build: Mesen (Windows - net8.0 - AoT).zip
Linux Build: Mesen (Linux - ubuntu-20.04 - clang_aot).zip

You'll find "Keyboard Mouse Host" listed in the NES input expansion devices. It has some additional settings (port select, absolute/relative mouse, key/mouse on/off) that you can find just below it on the NES input panel.

One of the consequences of Mesen's keyboard interface is that it disables other keyboard shortcuts while it's active, making it difficult to pause. Edit: Mesen2 has been updated to always allow the pause shortcut, as well as gamepad-mapped shortcuts, even while a keyboard interface is active.

I have also created two suitable test ROMs:

kbmtest
The first is a dedicated test that displays everything the interface should report.

kbmtest.zip

kbmtest.png

d34test
The second is simpler, a variation of my earlier CTRLTEST ROM that just dumps data read from each data line every frame.

d34test.zip

d34test.png

Edit: Added link to code changes, note about Mesen's pause shortcut problem. Tweaked kbmtest character set.

[rainwarrior]

Looking into the USB specs, USB returns those scancodes as an array of currently pressed keys, it must be the USB host driver and OS generating press and release events (along with key repeats, sticky keys, etc.). It seems best to maybe take this USB scancode set and customize it into a synchronous PS/2-like interface. See the attached file for what I'm thinking. One byte per key press, using D7 to select press/release for up to 127 keys.

NES keyboard DRAFT.txt

I cut out stuff that one wouldn't see on most keyboards, like F12-F24 keys, copy, paste, help, find, power, etc. I kept some of the media controls because there was going to be a little room left over, and those seem to be the only "extra" keys I use sometimes. I'm open to changing it, if it can be improved. I have some other projects to complete first, but after that I'd like to work on keyboard support.

After going over this to create an emulator implementation, and the test ROMs, I feel that this sufficiently covers the needs of an NES keyboard, so I don't think there's any need to make changes to it. A brief overview of the chosen keys, trying to highlight the parts that might be worth reviewing.

• $01 - Rollover. Probably not a lot of use to most applications. Maybe could help diagnose keyclash, but I feel like the user can figure that out anyway by what key doesn't respond.
• $02 - Post Fail. Not useful. The USB host device should be dealing with this, not the NES software.
• $03 - Undefined error. Also not likely to be useful.
• $04-$64 - Matches the standard and I see no reason to change or omit any of those.
• $65 - Compose. Standard. Edit: For typing accent combinations on some language keyboards, but may also be the "context menu" key on some US keyboards?
• $66 - Ro (International1). Older 109-key Japanese keyboards have this as an extra key next to a shortened right shift.
• $67 - Kana (International2). I think this is good to have, because the Family Basic keyboard has this.
• $68 - Yen (International3). Another 109 extra, next to a shortened backspace. Is probably also Won on Korean keyboards?
• $69 - Henkan (International4). 109 extra, right of the space bar. Converts current kana to kanji.
• $6A - Muhenkan (International5). 109 extra, left of the space bar. Keeps current kana without converting.
• $6B - Kp Jp Comma (International6). Couldn't find any information about keyboards that might have this?
• $6C - Hangeul or Han/Yeong (LANG1). Toggles Hangeul entry for Korean.
• $6D - Hanja (LANG2). Converts Hangeul to Hanja characters for Korean, though often right ctrl has this function instead of a dedicated key.
• $6E - Katakana (LANG3). Dedicated Katakana key for Japanese keyboards. Uncommon/specialized?
• $6F - Hiragana (LANG4). Dedicated Hiragana key for Japanese keyboards. Uncommon/specialized?
• $70 - Zenkaku Hankaku (LANG5). Full width / half width toggle.
• $71-$77 - Media keys (play-pause, stop, prev, next, vol up, vol down, mute). These are pretty good to have. The number range and order isn't standard here, but that's OK.
• $78-$7F - Left/right control, shift, alt, win. Standard order but not in the standard range, but these are essential keys and putting them at the very top of the range like this is a great idea. Also think the symmetrical 4+4 arrangement aligned to 4 is good because it facilitates merging them in the NES ROM's software.

So $00-$65 are standard, almost all of them nearly universally present. Reading the standard it seems that the letter key codes are for the US locale, so e.g. a French keyboard might have a Z where the US W key is, but it will still use the W scancode. Though, maybe in a case like that the USB host device could be tasked with remapping it. At any rate, if your NES software has controls that require a specific physical layout (e.g. WASD) it's probably best to make it user configurable if you can.

$66-$70 feel the weakest, but among the USB HID tables, I would agree that these 11 keys are probably the most important. Everything else is basically uncommon functional extensions, extra F-keys, hexadecimal numpad, stuff like that. A lot of these have uses that probably aren't well suited for the NES anyway (it's never going to be a great word-processing platform), and they tend to have redundant modern IME methods allowing the same functionality through other keys. So... ultimately I think these are fine as-is, but they probably wouldn't get much use anyway.

$71-$77 as media control keys seems good to have. There's probably some relevant purposes for this that would work on the NES. I think the 7 you picked are probably the most commonly seen ones. I often also see a homepage key, or an e-mail key, or sometimes cut/copy/paste keys, but nothing I think is more common or useful here than the media keys chosen.

$78-$7F was a great place for these essential modifier keys.

I think in general this can be considered a "soft" standard, that someone could override if they had a need, but I think this set should cover a pretty wide range of possibilities as-is. If anything is to be overridden for some special purpose software, I would expect the $66-$70 block to be the first to fall, but it's fine to have them specified here. I don't think we need to have a reserved area or anything like that.

Similarly, the errors at $01-$03 probably aren't useful. I wouldn't expect NES software to try to do anything with them, but it's not a problem to leave them in.

One thing I noted while implementing this for Mesen is that num-lock can be tricky, because in some interfaces the numpad keys won't be recognized unless num-lock is active. So... if your NES software wants to use num-lock and the num-pad, consider giving a secondary way to toggle the NES software's num-lock state just in case the OS on the user's machine already has num-lock in the opposite state. (If your software doesn't use num-lock for anything, you don't need to worry about it. The user can toggle their own OS num-lock when they want to use the pad.)

---

Edit: Some extra notes on the international keys, after investigating by debugging Mesen, and using Windows 10's On-Screen Keyboard (system32/osk.exe) to use keys I don't have a keyboard for. Also referencing Wikipedia's Language input keys article.

• Ro (International1) - JP-106/109 works as "|" in Mesen (154 OemBackslash)
• Kana (International2) - JP-106/109 reads as "OemCopy" in Mesen (159 dbehiragana), has down event and repeat, but no keyup!
• Yen (International3) - JP-106/109 works as "\" in Mesen (150 OemPipe)
• Henkan (International4) - JP-106/109 works as "IME convert" in Mesen (14 ImeConvert)
• Muhenkan (International5) - JP-106/109 works as "IME nonconvert" in Mesen (15 ImeNonConvert)
• Kp Jp Comma (International6) - ?
• Han/Yeong (LANG1) - KO-101 works as "Kana Mode" in Mesen (9 HangulMode)
• Hanja (LANG2) - KO-101 works as "Kanji Mode" in Mesen (12 KanjiMode)
• Katakana (LANG3) - ?
• Hiragana (LANG4) - ?
• Zenkaku/Hankaku (LANG5) - JP106/109 reads as "OemEnlw" or "OemAuto" in Mesen (161 DbeDbcsChar, 160 DbeSbcsChar). This is a toggle key, so each press gives a down for one, and up for the other, repeats the same keydown while held.

So, most of these work OK. Three of them are still kinda MIA, probably not found on non-specialized keyboards? Two of them are weird and probably problematic for emulators. The "Kana" key seems to have no keyup on my system, and I suspect it might be possible for this to report 158 DbeKatakana as well, but I couldn't get Mesen to read it as such. The "Han/Zen" key reports as two keys at once with a toggle, so it's not straightforward. However, both of these can simply be remapped to another key if more useful behaviour is needed. Probably nobody needs to use these two in their NES software anyway, but just a note that emulators might have a problem with them.

---

TLDR I think Memblers' spec is perfectly fine. I'd say stick with it.

[bitcores]

Absolutely awesome updates.

One of the consequences of Mesen's keyboard interface is that it disables other keyboard shortcuts while it's active. I would recommend binding a gamepad button to the emulator Pause to work around this, because otherwise it can be difficult to do that.

Could you program in a key combo that would either release the mouse or trigger pause to help break out of this?
It's something I considered when making my example USBhost program, so I made pressing ctrl+leftshift+esc toggle the exclusive capture of the inputs.

[rainwarrior]

One of the consequences of Mesen's keyboard interface is that it disables other keyboard shortcuts while it's active. I would recommend binding a gamepad button to the emulator Pause to work around this, because otherwise it can be difficult to do that.

Could you program in a key combo that would either release the mouse or trigger pause to help break out of this?
It's something I considered when making my example USBhost program, so I made pressing ctrl+leftshift+esc toggle the exclusive capture of the inputs.

Yeah, I was thinking that what Mesen should do is never let the pause keyboard shortcut be disabled by an emulated keyboard input device. This would make the emulated escape key unreachable by default, but it would be up to the user to reassign that as they see fit. (E.g. they could move the pause shortcut to Scroll Lock similar to RetroArch, or some other key not needed by the emulated software.)

Edit: I've now made that change, it should appear in the builds linked above shortly. That one I'll definitely suggest to Sour.

[Pokun]

Emulators like Mame and several Nes emulators that emulates the Family BASIC keyboard supports a UI key (which is usually Scroll Lock) which escapes the captured state of the keyboard to allow using the keyboard keys to what they are mapped to, emulated (non-keyboard) inputs or UI hotkeys alike. Mesen might benefit from such a key too. But yeah, disallowing hotkeys when using the keyboard was not a good idea.

Though I like that I can remap emulated keys to any keys in Mesen, not all emulators does that. I think Nestopia had this particular problem where it had a default mapping that was unchangeable, which made some Family BASIC keys not have a mapping on my keyboard.

In the case of this adapter the emulated keyboard may have the equal number of keys as the physical keyboard though (Scroll Lock is present on the emulated keyboard), so I guess being able to map emulated keys and hotkeys to other USB devices at all times is extra useful.


Now what I miss is a method to paste text into the emulator which starts a typing macro, such as is possible in VirtuaNES, Nestopia and openMSX (which has the best solution for this). Sour had this on his to-do-list for a long time but other things has had priority.

[rainwarrior]

Emulators like Mame and several Nes emulators that emulates the Family BASIC keyboard supports a UI key (which is usually Scroll Lock) which escapes the captured state of the keyboard to allow using the keyboard keys to what they are mapped to, emulated (non-keyboard) inputs or UI hotkeys alike. Mesen might benefit from such a key too. But yeah, disallowing hotkeys when using the keyboard was not a good idea.

Sour actually revised this already since I suggested it. (change) So now:

• 1. The pause key is never blocked, it will always pause emulation.
• 2. Shortcuts mapped to the gamepad are no longer blocked as well. This at least gives an alternative way to map shortcuts that you want to use without a pause.

There are other ways to do it, but I think this is practical at least. I think the described Scroll-Lock approach taken by RetroArch/etc. has its own tradeoffs for usability, but it's also practical.

Anyway, now you can just tap pause to be able to use shortcuts, tap pause again to resume. Map things to the gamepad if you need to do it without pause.

[Pokun]

I see, it indeed sounds practical and is an improvement.

[bitcores]

Seeing the shortcut key can be changed, if it conflicts with use in a program it could always be changed by the user. It's not that uncommon for Scroll Lock to be missing from keyboards these days, so shifting the default shortcut there could again result it someone getting stuck in the emulator.

Anyway, I have been trying to get the mouse wheel to work on the KeyboardMouseHost input but I'm not getting any result. I've tried looking in the code and I'm not sure if the mouse wheel is even handled in Mesen.

[rainwarrior]

Anyway, I have been trying to get the mouse wheel to work on the KeyboardMouseHost input but I'm not getting any result. I've tried looking in the code and I'm not sure if the mouse wheel is even handled in Mesen.

No, Mesen has no mouse wheel support. What I added as a substitute was to map the emulated mouse's wheel up and down to a chosen button, if you hold it down it will tick once every 15 frames.