Making emulators touch-aware through breakpoint files

Discuss emulation of the Nintendo Entertainment System and Famicom.

Moderator: Moderators

Post Reply
tepples
Posts: 22738
Joined: Sun Sep 19, 2004 11:12 pm
Location: NE Indiana, USA (NTSC)
Contact:

Making emulators touch-aware through breakpoint files

Post by tepples »

I was reading through Apple's guideline for touch controls in games for iOS. It offers suggestions that could improve emulators designed for any desktop or mobile platform.

Apple's guideline for virtual gamepads recommends showing an icon for each button's primary action on the button, much as The Legend of Zelda shows the actions at the top. For example, Super Mario Bros. might show icons for "jump" and "run" on the A and B Buttons of the virtual gamepad, where the "run" changes to "fireball" after grabbing the Fire Flower. Is there a practical way for an emulator to expose some means to recognize what icon to show for each button in each situation in a particular game? As far as I can tell, it'd involve a configuration file listing breakpoints that change the icon.

Apple recommends hiding the virtual gamepad and switching to direct touch interaction when a menu is visible. Or on desktop, it could support a mouse. Is there a practical way for an emulator to expose some means to recognize a menu and feed the game button presses to operate it? As far as I can tell, it'd involve a configuration file listing breakpoints that notify the emulator where and when a menu appears or disappears.

Feasible or not?
Joe
Posts: 656
Joined: Mon Apr 01, 2013 11:17 pm

Re: Making emulators touch-aware through breakpoint files

Post by Joe »

I'm sure it could be done.

It would have to be customized for each game, though, which is a lot of work.
calima
Posts: 1756
Joined: Tue Oct 06, 2015 10:16 am

Re: Making emulators touch-aware through breakpoint files

Post by calima »

Assuming you control both the game and the emu, you could easily define a new register for each purpose.
User avatar
Ben Boldt
Posts: 1216
Joined: Tue Mar 22, 2016 8:27 pm
Location: Minnesota, USA

Re: Making emulators touch-aware through breakpoint files

Post by Ben Boldt »

I think that this could be accomplished in almost all cases with RAM watching. Particular values at particular locations would indicate the active power.

Other things to consider:
- not all games use power-ups.
- this display probably should be hidden at certain times, such as cutscenes, title screen, ending sequence, etc.
- Apple not allowing emulators in the App Store AFAIK.
- display aspect ratios giving rise to black bar areas which may be useful areas for something like this.
tepples
Posts: 22738
Joined: Sun Sep 19, 2004 11:12 pm
Location: NE Indiana, USA (NTSC)
Contact:

Re: Making emulators touch-aware through breakpoint files

Post by tepples »

Ben Boldt wrote: Tue Jun 11, 2024 6:08 am- Apple not allowing emulators in the App Store AFAIK.
The whole reason I'm asking is that Apple recently changed this policy, and one of the major Game Boy emulators got an official iOS port a couple weeks later.
Pokun
Posts: 2772
Joined: Tue May 28, 2013 5:49 am
Location: Hokkaido, Japan

Re: Making emulators touch-aware through breakpoint files

Post by Pokun »

I'm against displaying the button actions in emulators, unless maybe if the emulator is bundled with a single game to hide the fact the game is written for another hardware platform or something like that. Emulators should by default always display the names of the buttons on the official controller for the system (and use the same layout as much is possible) and it's up to the user to know what button does what in each game, there are often manuals and guides that helps with that.

Displaying game-specific button actions could be an optional feature but it's a doomed project that will never work with all games.
User avatar
Ben Boldt
Posts: 1216
Joined: Tue Mar 22, 2016 8:27 pm
Location: Minnesota, USA

Re: Making emulators touch-aware through breakpoint files

Post by Ben Boldt »

I don’t think it’s doomed if users can easily create their own mappings. I think it is a neat idea. I am pretty sure that some people would like it and some would not like it, so it is probably important to be able to turn it on or off.
mkwong98
Posts: 290
Joined: Mon May 30, 2011 9:01 pm

Re: Making emulators touch-aware through breakpoint files

Post by mkwong98 »

I'm more interested with making the menu clickable. The emulator needs to know where the window is, how are the items arranged and how the current cursor/selection is stored in RAM. Then when a click occurs, the emulator can change the value in RAM to point the cursor at the clicked item and simulate a button press. For games which use a button (usually B) to cancel, the emulator can either display the B button as a "X" on the top right corner of the menu or simulate a press of B button when the user clicks outside the menu area. Password or name entry screens would be much easier to use if the letters can be clicked. Another idea is to show the keyboard on screen or can use a real keyboard on desktop for that.
tepples
Posts: 22738
Joined: Sun Sep 19, 2004 11:12 pm
Location: NE Indiana, USA (NTSC)
Contact:

Re: Making emulators touch-aware through breakpoint files

Post by tepples »

Ben Boldt wrote: Tue Jun 11, 2024 6:08 amI think that this could be accomplished in almost all cases with RAM watching. Particular values at particular locations would indicate the active power.
This ties into a drawback that a Discord user mentioned. Anything based on breakpoints or scripting would probably halve the emulation speed. This could increase the system requirements and/or battery use of the emulator. For those games where a RAM watch is practical to identify the game's state, it would probably take less CPU.
Pokun wrote: Wed Jun 12, 2024 1:40 pmEmulators should by default always display the names of the buttons on the official controller for the system (and use the same layout as much is possible) and it's up to the user to know what button does what in each game
The point of the page I linked is that the attitude "it's up to the user to know what button does what in each game" leads to a poor player experience. In case the article disappears from Apple's website, I'll quote what I'm referring to:
Prefer designing expressive virtual controls that communicate the actions they perform. Players can more easily learn and remember what a control does when its appearance visually represents its action. For example, you might design an attack button that displays a graphic of the weapon that’s currently equipped, or a movement control that uses an arrow to show the direction of motion.

[A screenshot of a game running on iPhone in landscape. The label of the circular virtual control in the bottom-right corner of the screen uses a symbol that looks like a hand about to pick something up.]
A control that communicates its function during gameplay

[A screenshot of a game running on iPhone in landscape. The label of the circular virtual control in the bottom-right corner of the screen uses the letter A.]
A control that requires players to remember what it does

Aim to design virtual controls that dynamically change to reflect gameplay. Instead of rendering a static overlay that reproduces the buttons on a physical controller, take advantage of touch interaction to give players virtual controls that adapt to gameplay. For example, if the behavior of a virtual control changes based on the current context, its appearance can update, signaling the change to players. Using adaptive virtual controls also means that you can hide a control when it isn’t available or relevant, letting you reduce clutter and help players concentrate on what’s important.
there are often manuals and guides that helps with that.
In my experience, games can and do get separated from their manuals, and players without manuals can and do misinterpret things while playing games on stream.
Displaying game-specific button actions could be an optional feature but it's a doomed project that will never work with all games.
I didn't mean for it to work with all games. I thought I implied in my original post that it would take effort to make it work with each supported game, but that the effort per game is worthwhile.
mkwong98
Posts: 290
Joined: Mon May 30, 2011 9:01 pm

Re: Making emulators touch-aware through breakpoint files

Post by mkwong98 »

Besides RAM watching, another method is name table watching if you know what tile and where to look for.
Post Reply