Graphics in Excel (New: 'Tiled' Map Translation Tools!)

Discussion of hardware and software development for Super NES and Super Famicom. See the SNESdev wiki for more information.

Moderator: Moderators

Forum rules
  • For making cartridges of your Super NES games, see Reproduction.
User avatar
Ramsis
Posts: 341
Joined: Sun Jul 01, 2012 6:44 am
Location: Lion's den :3
Contact:

Re: Graphics in Excel

Post by Ramsis »

For MSU1 audio to play, one major prerequisite has to be met (at least on real hardware like the sd2snes), which is to unmute the SPC700, and disable echo. Also, you have to wait for the Audio Busy bit to clear before setting the audio state to play.

Here's a code snippet I posted on another forum quite some time ago (the posting is in German, but the code itself should be self-explanatory, and has English comments). It's confirmed working on both higan and sd2snes. LMK if you want the ROM and/or full source, which I'd have to dig up from somewhere. :wink:

As for the data port, it essentially works the same, except that it's up to you how to process the streamed data, of course. A good reference is e.g. ikari's video player, sources are included in these (snes/msu1 subfolder).
Some of my projects:
Furry RPG!
Unofficial SNES PowerPak firmware
(See my GitHub profile for more)
User avatar
Khaz
Posts: 314
Joined: Thu Dec 25, 2014 10:26 pm
Location: Canada

Re: Graphics in Excel

Post by Khaz »

Mmmkay, so I tried adapting ikari's video player code to my program and nothing at all happened. So I tried using your code and I'm getting somewhere because it's crashing Bizhawk now - which I've learned happens when it hits a WAI command and never gets an interrupt, which due to the way I hacked it out means it's taking the "NoMSU" branch. So, I haven't made it recognize that I'm trying to use MSU1 yet. Bizhawk is just a shell for bsnes v87, so it definitely should work, right...?

I decided to try it in higan and magically, it works. Works on my SD2SNES as well. So I guess somehow Bizhawk lacks MSU1 support despite using bsnes.

I'm grateful for the help trying to get this put together but I can't believe that the actual specification page on byuu's site that all this knowledge seems to come from is just plain gone. Without your help I'd have never figured it out. Did I miss something? I was just reading some heated conversations right on this forum no more than a year ago getting the fine details of it worked out. Seems like an exciting new development like this should have resulted in a bunch of interesting new hacks in the meantime yet MSU1 barely even brings in a handful of google results. Do I dare start relying on it for more than mere demos?

(Speaking of those conversations, did they ever reach some kind of agreement about the audio frequency/clock speed? I was extremely concerned to see the issue of video and audio de-syncing brushed aside as unimportant - a tiny desync becomes absolutely intolerable very quickly.)

Anyways. The audio is hideously out of sync the way I have it written now... It starts playing several seconds before the video begins to load. As far as I can tell from my code, the first frame should appear on screen 4-5 frames after the audio begins playing, so I'm assuming there's some kind of long buffering delay or something from loading the audio data. When looking at ikari's video player it looked like he had it set up to start the audio as the first frame is drawn, but I'm not sure that's any better if you still have a significant delay right after the audio starts.

Was trying to take a closer look in higan but I'm shocked to discover it doesn't even seem to have a frame counter so I can see how much of a delay I'm even dealing with... Why oh why doesn't it just work in Bizhawk. I think I may have to go bother adelikat about that, if he's forgiven me for the last time I wasted his time.

The video is looking amazing though. On my TV it's like 3/4 of an 8x8 tile row away from fullscreen, you can hardly tell it isn't a full picture. Frame processing is at ~200/2682 after maybe a day running full speed. Gives me time to figure out the data stream, though I am worried I won't have a processor left when it's done.

EDIT: Oh, I see:

Code: Select all

   lda #$01   ; Set audio state to play, no repeat.
   sta MSU_CONTROL
   lda #$00
-   inc a
   wai
   sta MSU_VOLUME
   cmp #$FF
   bne -
   ; The MSU1 will now start playing.
   ; Use lda #$03 to play a song repeatedly.
So it's taking a bunch of frames to gradually ramp up the volume. I guess I can build that into my normal frame loop, if I even need it.

EDIT: Audio is nicely synced and I got rid of that annoying little line of pixels at the bottom by blanking both tilemaps first. Literally all I need now is to learn to DMA frames from MSU1 and I've got it.
User avatar
Khaz
Posts: 314
Joined: Thu Dec 25, 2014 10:26 pm
Location: Canada

Re: Graphics in Excel

Post by Khaz »

Frame processing is approaching 50%. I really need to speed things up somehow but I'm already a little disturbed by the quality I'm getting out in places. Certain scenes have really large areas of almost-but-not-quite the same colour and it looks to me like those just become terrible the moment you reduce them to the SNES's maximum colour depth, as the smooth gradient of colour becomes a splotchy broken mess. This also kind of highlights the tilemap grid for those brief moments, which is less than ideal.

I haven't attempted any manual correction for this effect yet. In certain scenes it might be as easy as changing one or two palette colours, I will give it a try later.

98% of the video looks great though. I enjoy how the MSU1 just keeps on reading data after it reaches the end of the file - on emulator it does nothing but on SD2SNES it just starts playing the other ROMs on the card as though they were video, which is very very colourful and poooossibly seizure-inducing I'm not sure. I'll patch it out when I'm done but for now I'm enjoying the show.

...

Serious question: I'm pondering making use of the MSU1 for the other main project I've been working on. I don't dare to dream that it could someday be worthy of burning into a cartridge, but I'd like to keep that possibility open just in case I actually finish something for once. I'm afraid to start planning on it though without knowing how practical the MSU1 is to actually build.

It sounds straightforward enough to me, but it doesn't sound like the intended normal-cartridge implementation of MSU1 has been built yet. I got the impression it is supposed to be simpler than what ikari had to do for SD2SNES. I've recently discovered that I live near people with the means to build high quality reproduction cartridges, out of newly made parts instead of reclaimed used stock (and accurately labelled as reproductions to boot). I'm eager to discuss the possibility of MSU1 support with them, but I'd like to know what I'm getting into first from a hardware perspective.

P.S. I am aware this thread hasn't been about Excel for a long time now and I am sorry
lidnariq
Site Admin
Posts: 11635
Joined: Sun Apr 13, 2008 11:12 am

Re: Graphics in Excel

Post by lidnariq »

Real-world hardware that can ONLY stream video or audio at once is easy. Real-world hardware that can do both from the same media becomes a royal PITA.

A more practical option might be putting two compactflash cards into a single cart, one for audio, one for the CPU. Or maybe better would be a compactflash card for the CPU and one of those MP3-player-on-a-chips to stream audio.

(P.S. PM tepples a split point)
User avatar
Khaz
Posts: 314
Joined: Thu Dec 25, 2014 10:26 pm
Location: Canada

Re: Graphics in Excel

Post by Khaz »

lidnariq wrote:Real-world hardware that can ONLY stream video or audio at once is easy. Real-world hardware that can do both from the same media becomes a royal PITA.
Wait, so, the way MSU1 functions in higan and SD2SNES is not how it's intended to work? Or do I misunderstand? Are you saying that you can't DMA data to VRAM using the MSU1 while a track is playing? Or did they deliberately emulate the maximum attainable functionality regardless of how difficult it would be to implement? I mean - how bad is the SD2SNES, when it has to not only stream data and audio simultaneously, but then write SRAM back to the same card too?

I think I'm starting to understand what byuu was talking about now with the "open-ended design allowing for flexibility in deployment" or however it was worded. It makes sense but it's a bit of a pain to not just have a full blown chip design that explicitly lays it out for you. I guess nothing worth doing is easy...
93143
Posts: 1831
Joined: Fri Jul 04, 2014 9:31 pm

Re: Graphics in Excel

Post by 93143 »

The data and audio are streamed from two separate memory spaces, right?. So in a real system you could well have two different physical storage devices for them - in fact that would seem to make the most sense. The sync issue was dealt with (I think it was dealt with...) by making the audio timing track the SNES master clock, which implies that the MSU1 itself does no internal A/V synchronization, which implies that the hardware interfaces for data and audio don't have to be tightly coupled. It seems to me that this means you just need 2x"easy" instead of 1x"royal PITA".

At worst you could just copy what the SD2SNES does, with an FPGA addressing a single moderately high-bandwidth memory device. Since save games wouldn't also have to go on said device, the reset-to-save hack wouldn't be necessary any more... mind you, I have no idea what the components would cost...
Ramsis wrote:disable echo
Why?
lidnariq
Site Admin
Posts: 11635
Joined: Sun Apr 13, 2008 11:12 am

Re: Graphics in Excel

Post by lidnariq »

Khaz wrote:Or did they deliberately emulate the maximum attainable functionality regardless of how difficult it would be to implement?
That.

Basically, in order to be able to stream two independent things at the same time from the same medium requires a lot of RAM and a fairly large amount of logic glue. By promising both, it's kind of a pain to implement something similar without either
1- using two separate pieces of media, one just for audio, one just for the CPU
2- or a comparatively large FPGA with a large amount of RAM.
I think I'm starting to understand what byuu was talking about now with the "open-ended design allowing for flexibility in deployment" or however it was worded.
"flexibility in deployment" means "difficult to deploy". If the interface were instead just "Here's a CF card/deserialized SD card/whatever" it would, at absolute worst, condemn the hardware instantiation to irrelevance in a decade or two.

That's why the SegaCD / PC-Engine CD-ROM interface is hairier than the MSU1 one; it reflected the hardware rather than being abstracted away.
User avatar
Ramsis
Posts: 341
Joined: Sun Jul 01, 2012 6:44 am
Location: Lion's den :3
Contact:

Re: Graphics in Excel

Post by Ramsis »

93143 wrote:
Ramsis wrote:disable echo
Why?
Sorry, no clue whatsoever. :P
Some of my projects:
Furry RPG!
Unofficial SNES PowerPak firmware
(See my GitHub profile for more)
User avatar
Khaz
Posts: 314
Joined: Thu Dec 25, 2014 10:26 pm
Location: Canada

Re: Graphics in Excel

Post by Khaz »

So I recently learned about some functionality in Excel I wasn't previously aware of, that inspired me to actually make a more useful tilemap editor. I think what I came up with is going to save me a lot of time, so I thought I'd share it.

There are a few macros to be used in the same way as my previous spreadsheets:

ReadMap: Reads a text file of a tilemap (.dw formatted), either 32x32 or 64x64, and loads it into the MAP EDITOR tab.

WriteMap: Writes the tile map currently in the MAP EDITOR tab to a text file in .dw format.

PaletteShift: Gives three different ways to manipulate the palette numbers of every tile on the sheet. Run it and you'll get a popup window explaining how it works more specifically.

DrawEntireMap: IF you have a tile set in the TILE SET tab (the same way the tile sets are in my original spreadsheets - it reads the cell colours directly), then this macro will render the tile map according to the tile set, and output the results into MS Paint. You must have Paint open for this to work. (The image will also be drawn into another excel worksheet, but you should delete that worksheet as soon as it's done. Not every computer can seem to handle excel when the cells are at minimum size like that.)

UndoLastChange: Undoes the last edit to the map, SEE NEXT SECTION:

The fun part of this is that I discovered you can program macros to be executed on certain events within Excel. In this case, I have macros set to run when you click on cells within the MAP EDITOR worksheet!

The top part of the sheet is the 64x64 map itself. If you left-click on a cell, the value in "LEFT CLICK:" down below will be written to it. If you right-click, the value in "RIGHT CLICK:" gets written. You can drag to select a range of cells, and they will all be assigned the value in LEFT CLICK. You can't drag a range of cells with the right mouse button, but you CAN select a range with the left button (which will change it to LEFT CLICK), and then right-click that selection to change it to RIGHT CLICK.

You can change the values of LEFT and RIGHT CLICK manually using the other fields at the bottom: The box to the bottom left is your tile numbers (right now it's built for 16x16 tiles so there's no odd numbered rows or columns, sorry, can work on that). Click the boxed, numbered cells to set H/V inversion, palette and priority. Alternately, you can copy in a number that's currently on the tilemap by clicking "< COPY TILE INTO LEFT (or RIGHT) CLICK", then clicking the tile on the tilemap. The "COPY" cells will highlight themselves in red to show when they're active.

Finally, since this functionality requires disabling the normal on-click behaviour, there's also a big "DISABLE MACROS" button. When it turns red and says "MACROS DISABLED", everything returns to normal Excel until you click it again. The main thing this allows you to do is select a block of cells on the tilemap, copy it and then paste it elsewhere.

Finally, there's the UNDO STACK. Since Excel can't "undo" a macro action, and since it's very easy with this method for a stray click to screw up a map, I implemented my own undo feature. Every time you click or drag on the tilemap and make a change, it's stored in the UNDO STACK. Whenever you run UndoLastChange, it undoes the last change (the first item in the list). The keyboard shortcut is currently set up as CTRL+U, but you can change that yourself under Macros, Options...

If people have feedback I would welcome it. My continued and sincere apologies to the non-Excel crowd. I figure after a bit more touchup work, I should go back and edit the first post of this thread to put everything in one place...

EDIT: To-do list:
-8x8 tilemap support
-improve drawing (tile set/palette reading/importing, and direct file output because this method is stupid but it was just kinda proof-of-concept)
-Merge with tile editor spreadsheet?

Not that anybody is interested here, but I've realized a few more improvements on this version and I'm currently revising it. The next one should be much better in many ways.
Attachments
MapReader.xlsm
Tile Map Editor for Excel
(314.78 KiB) Downloaded 281 times
User avatar
Khaz
Posts: 314
Joined: Thu Dec 25, 2014 10:26 pm
Location: Canada

Re: Graphics in Excel (New: TileMap Editor)

Post by Khaz »

Okay. Massively overhauled the map editor since my last post. Summary of the new state of things:

ImportTileMap, ExportTileMap, PaletteShift, UndoLastChange are unchanged (aside from the names).

DrawMapToBitmap: Does direct-to-bitmap output now, but is curiously somehow slower than my previous stupid method. It's about a 3MB file so maybe excel is just really slow at one-byte-at-a-time binary writes. It takes just under an hour at worst but I can't think of any reasonable way to speed it up further.

DrawTileSet: New macro; Renders the TILE SET sheet according to one of the palettes in the Palettes sheet.

ImportPalettes, ImportTileSet: New macros; Imports palettes or a tile set from a text file. ImportPalettes requires them to be formatted specifically (.dw where each line is one palette). Tile Set just reads in data (either .db or .dw) until the sheet is full or until the end of the input file is reached.

The MAIN overhaul though was how the Map Editor functions. The annoying click-cells-to-control-stuff region at the bottom is gone, replaced by a userform! Double-click any cell in the "MAP EDITOR" worksheet to bring up the "Map Tools" window (or alternately run the "ShowMapTools" macro manually). This has all the same functionality as before but in a separate little window so you can move around the map more freely. Brief summary follows:

Each mouse button has a Tilemap-Word associated with it. LEFT and RIGHT CLICK buttons at the top determine which mouse button's Tilemap-Word you're editing with the other controls on the panel. Left click a cell on the map to assign it the value of LEFT CLICK; right-click a cell to assign it the value of RIGHT CLICK. Drag-select a range of cells to assign them all the value of LEFT CLICK. Right click that selection to change them all to RIGHT CLICK.

When you click the COPY TILE button, then select a cell on the worksheet, the contents of that cell will become the new RIGHT- or LEFT-CLICK word, depending on which button is toggled at the top. UNDO just runs UndoLastChange (as does Ctrl+U).

DISABLE MACROS works the same as before, to disable editing mode so you can copy/paste cells and interact with the sheet normally. Also note that the click-to-edit functionality on the MAP EDITOR will ONLY work while the MapTools window is open. When the window is closed, everything returns to a normal Excel spreadsheet.

I decided against merging the two spreadsheets for now, just for the sake of less work and keeping things more organized. I'm satisfied for my own purposes again, so I'm only going to take on a task like that if other people are interested. As always, let me know if you find it useful and if you have problems with it I'm happy to help.
Attachments
MapToolsV2.xlsm
Updated Tile Map Editor
(548.67 KiB) Downloaded 303 times
tepples
Posts: 22862
Joined: Sun Sep 19, 2004 11:12 pm
Location: NE Indiana, USA (NTSC)
Contact:

Re: Graphics in Excel

Post by tepples »

Khaz wrote:Code is now on Github.

Code: Select all

if bytelist[0] != b'B' or bytelist[1] != b'M':
    print ("This is not a bitmap.  Aborting.")
    quit()
Is there a reason why you parse Windows bitmaps manually rather than using Pillow (Python Imaging Library)?
User avatar
Khaz
Posts: 314
Joined: Thu Dec 25, 2014 10:26 pm
Location: Canada

Re: Graphics in Excel

Post by Khaz »

tepples wrote:
Khaz wrote:Code is now on Github.

Code: Select all

if bytelist[0] != b'B' or bytelist[1] != b'M':
    print ("This is not a bitmap.  Aborting.")
    quit()
Is there a reason why you parse Windows bitmaps manually rather than using Pillow (Python Imaging Library)?
Yes, the reason is that this program was originally written in Excel VBA, where I don't think I had a choice but to decode the bitmaps manually. When I made the transition to Python I was basically only concerned with translating what I already had to that language. Also, this is the first time I have heard of Pillow.

Considering that the parsing of the bitmap takes a trivial amount of time in the process I'm not too concerned if it's not done the most efficient way. If anyone happens to find a way to speed up the actual quantization loop without losing quality though, that would be huge and please tell me.

In general if anyone wants to take my program and improve it, please, by all means help yourself. It is my only Python project and I'm sure I did a ton of things the stupid way.
User avatar
rainwarrior
Posts: 8759
Joined: Sun Jan 22, 2012 12:03 pm
Location: Canada
Contact:

Re: Graphics in Excel (New: TileMap Editor)

Post by rainwarrior »

PIL (or Pillow) is simply a widely used image library for python. It's not part of its standard library, you have to download and install it separately.

Writing a simple BMP parser is comparable to the effort required to install PIL, so I think this is a wash. :P
tepples
Posts: 22862
Joined: Sun Sep 19, 2004 11:12 pm
Location: NE Indiana, USA (NTSC)
Contact:

Re: Graphics in Excel (New: TileMap Editor)

Post by tepples »

I imagine that it's a lot easier to \Python27\scripts\pip install Pillow (see full instructions) or sudo apt-get install python-imaging (if on Debian or Ubuntu) than to write your own parser for PNG, JPEG, and compressed BMP.
User avatar
Khaz
Posts: 314
Joined: Thu Dec 25, 2014 10:26 pm
Location: Canada

Re: Graphics in Excel (New: TileMap Editor)

Post by Khaz »

tepples wrote:I imagine that it's a lot easier to \Python27\scripts\pip install Pillow (see full instructions) or sudo apt-get install python-imaging (if on Debian or Ubuntu) than to write your own parser for PNG, JPEG, and compressed BMP.
Yeah, I'd imagine you're right. I chose simple uncompressed bitmap as an input format specifically because it's so easy to decode and I already knew how to...
Post Reply