Using window/shape masks behind background layers

Discussion of hardware and software development for Super NES and Super Famicom. See the SNESdev wiki for more information.

Moderator: Moderators

Forum rules
  • For making cartridges of your Super NES games, see Reproduction.
Pokun
Posts: 2681
Joined: Tue May 28, 2013 5:49 am
Location: Hokkaido, Japan

Re: Using window/shape masks behind background layers

Post by Pokun »

Turboxray I still find it confusing terminology, and I guess I would count as a regular enough homebrewer. Although I see your point with transparent pixels (and I guess "backdrop" and "background" are basically synonymous English words anyway), I would prefer to have separate terms for them since, as you say, the backdrop is a more limited background and not the same thing. If we would just call it the "background" people would wonder which background you are talking about whenever it isn't clear from context. And since we do have a nice well established word in homebrew communities and also used in Nintendo's and other companies' official development docs we might as well use that term.

"Backdrop colour" was used together with "border colour" in the TMS9918 datasheet, so I suppose Nintendo (though who didn't use a separate border colour), Yamaha (who made both the Sega and MSX derivatives of the TMS9918), Hudson and everyone else that made TMS9918-inspired video chips took the terms from there, although I guess the term could very likely be much older than the TMS VDP.


Inceptional I didn't find your post offensive and I doubt others did, so don't worry about that.
You do have a tendency to use personal attacks in heated discussions though ("learn to read!" etc), and to be overly dismissive to people that goes great lengths to offer advice. Those are mainly the things that resulted in those warnings earlier, so if you could avoid doing that you should probably be fine.
User avatar
rainwarrior
Posts: 8732
Joined: Sun Jan 22, 2012 12:03 pm
Location: Canada
Contact:

Re: Using window/shape masks behind background layers

Post by rainwarrior »

iNCEPTIONAL wrote: Tue Jul 19, 2022 12:26 pm Yeah, that was why I was avoiding counting the sprites as multiple additional layers for now, because once you go down that rabbit hole it's really had to quantify it all. :-o
Well, you really should consider using sprites as extra layers of parallax if your goal is to make the most interesting/impressive parallax you can on the SNES. (Example for inspiration: Lords of Thunder makes the 1-BG-layer PCe look almost like a 2-BG-layer MegaDrive.)

As for "quantifying", I dunno that anybody here cares which system has a bigger number.


From the other thread:
iNCEPTIONAL wrote: Tue Jul 19, 2022 10:34 amNow that I think about it more after having looked at rainwarrior's first reply/answer in this thread again, in the following video at 2:25 https://youtu.be/zcoU6-9_fDM?t=144, Retro Game Mechanics Explained talks of the SNES having six layers, and none of them seem to be the window/masks either, so, with a bit of expert design and coding, would it technically be possible to show what could actually look like a total of eight overlapping layers on the SNES?
No, the window applies to 6 different things, but it's not really 6 layers. (See: Wiki: Window mask settings)

There's 4 BG layers, and 1 sprite layer. Each of those can be affected by the window. The background colour isn't really a layer, and has to be shown everywhere there is nothing else on top.... so it can't be windowed.

The 6th window effect, instead of being a layer, is colour math itself. You can thus use the window to restrict colour math to a specific region of the screen.


Separately, colour math can be selectively applied to a related set of 6 things, (See: CGADSUB)

So again you have 4 BG layers and 1 sprite layer. The 6th thing is the background colour itself.

So, maybe for the purposes of colour math you can think of the background colour as its own layer. If you're using the window only for colour math, and you're using colour math only on the background colour, and the colour math is using only a fixed colour and not a sub-screen layer... then maybe in that case it's conceptually an extra 1-colour layer created by the window, superimposed on the background-colour layer.

If you're using colour math in any other way, not so much. If you want to use the sub-screen instead of a fixed colour, then there's no extra "layer". Whatever the sub-screen effect is will require the use of at least 1 of the other 5 layers.
iNCEPTIONAL

Re: Using window/shape masks behind background layers

Post by iNCEPTIONAL »

rainwarrior wrote: Tue Jul 19, 2022 11:22 pm
iNCEPTIONAL wrote: Tue Jul 19, 2022 12:26 pm Yeah, that was why I was avoiding counting the sprites as multiple additional layers for now, because once you go down that rabbit hole it's really had to quantify it all. :-o
Well, you really should consider using sprites as extra layers of parallax if your goal is to make the most interesting/impressive parallax you can on the SNES. (Example for inspiration: Lords of Thunder makes the 1-BG-layer PCe look almost like a 2-BG-layer MegaDrive.)

As for "quantifying", I dunno that anybody here cares which system has a bigger number.


From the other thread:
iNCEPTIONAL wrote: Tue Jul 19, 2022 10:34 amNow that I think about it more after having looked at rainwarrior's first reply/answer in this thread again, in the following video at 2:25 https://youtu.be/zcoU6-9_fDM?t=144, Retro Game Mechanics Explained talks of the SNES having six layers, and none of them seem to be the window/masks either, so, with a bit of expert design and coding, would it technically be possible to show what could actually look like a total of eight overlapping layers on the SNES?
No, the window applies to 6 different things, but it's not really 6 layers. (See: Wiki: Window mask settings)

There's 4 BG layers, and 1 sprite layer. Each of those can be affected by the window. The background colour isn't really a layer, and has to be shown everywhere there is nothing else on top.... so it can't be windowed.

The 6th window effect, instead of being a layer, is colour math itself. You can thus use the window to restrict colour math to a specific region of the screen.


Separately, colour math can be selectively applied to a related set of 6 things, (See: CGADSUB)

So again you have 4 BG layers and 1 sprite layer. The 6th thing is the background colour itself.

So, maybe for the purposes of colour math you can think of the background colour as its own layer. If you're using the window only for colour math, and you're using colour math only on the background colour, and the colour math is using only a fixed colour and not a sub-screen layer... then maybe in that case it's conceptually an extra 1-colour layer created by the window, superimposed on the background-colour layer.

If you're using colour math in any other way, not so much. If you want to use the sub-screen instead of a fixed colour, then there's no extra "layer". Whatever the sub-screen effect is will require the use of at least 1 of the other 5 layers.
Well, I want to do as much parallax with mainly the backgrounds and any little tricks as possible, and I might use a few sprites to add to that at times, but, really, I want to keep as many sprites free as possible because it's a shmup at the end of the day and I also want to have lots of enemies and bullets and stuff like that onscreen too, to show the SNES can do all the layers plus all the action.

The goal is to show off things that are unique to the SNES, particularly with the backgrounds, where I think it has clear advantages almost across the board, and also dispel any myths that it can't do intense, fast-paced shmup action without slowdown. Rendering Ranger R2 is my inspiration there, and I simply want to have even more technically impressive backgrounds in terms of the parallax to take things even further.

This is why I'm going to eventually need a really good programmer to do all the code behind all the design ideas.

Yeah, so I guess I can maybe use all the little tricks to fake maybe one extra layer effect at a time, be it a beam of light or a silhouette in the foreground or the typical transparency tricks or whatever, which is still all good.

I mean, and I think I could still use the window/mask plus direct colour effect together on my ice planet level to add in a solid opaque shape on top of everything else, even with the Mainsscreen and Sunscreens pkus colour math being used for the semi-transparent water, right?

I can work with that.
User avatar
rainwarrior
Posts: 8732
Joined: Sun Jan 22, 2012 12:03 pm
Location: Canada
Contact:

Re: Using window/shape masks behind background layers

Post by rainwarrior »

iNCEPTIONAL wrote: Wed Jul 20, 2022 12:21 amI mean, and I think I could still use the window/mask plus direct colour effect together on my ice planet level to add in a solid opaque shape on top of everything else, even with the Mainsscreen and Sunscreens pkus colour math being used for the semi-transparent water, right?
Only if you mean on separate scanlines. For colour math: you can use the sub-screen, or you can use the fixed colour. You can't use them both at the same time.
iNCEPTIONAL

Re: Using window/shape masks behind background layers

Post by iNCEPTIONAL »

rainwarrior wrote: Wed Jul 20, 2022 12:31 am
iNCEPTIONAL wrote: Wed Jul 20, 2022 12:21 amI mean, and I think I could still use the window/mask plus direct colour effect together on my ice planet level to add in a solid opaque shape on top of everything else, even with the Mainsscreen and Sunscreens pkus colour math being used for the semi-transparent water, right?
Only if you mean on separate scanlines. For colour math: you can use the sub-screen, or you can use the fixed colour. You can't use them both at the same time.
So I can't have an opaque window/mask shape on top of all the layers including the semi-transparent water, via the fixed colour or whatever other possible methods, without doing the two visual effects alternating on every second scanline?
Myself086
Posts: 158
Joined: Sat Nov 10, 2018 2:49 pm

Re: Using window/shape masks behind background layers

Post by Myself086 »

iNCEPTIONAL wrote: Wed Jul 20, 2022 12:21 am This is why I'm going to eventually need a really good programmer to do all the code behind all the design ideas.
I'm still down, find the money and I'm in. Really good programmers aren't going to work on your game for free.
iNCEPTIONAL wrote: Wed Jul 20, 2022 12:59 am
rainwarrior wrote: Wed Jul 20, 2022 12:31 am
iNCEPTIONAL wrote: Wed Jul 20, 2022 12:21 amI mean, and I think I could still use the window/mask plus direct colour effect together on my ice planet level to add in a solid opaque shape on top of everything else, even with the Mainsscreen and Sunscreens pkus colour math being used for the semi-transparent water, right?
Only if you mean on separate scanlines. For colour math: you can use the sub-screen, or you can use the fixed colour. You can't use them both at the same time.
So I can't have an opaque window/mask shape on top of all the layers including the semi-transparent water, via the fixed colour or whatever other methods, without doing the two visual effects alternating on every second scanline? That's how I would like to draw the beam.
Only one transparency between main and sub screens, only one main screen, only one sub screen, only one color math, only 2 windows. All per scanline.
iNCEPTIONAL

Re: Using window/shape masks behind background layers

Post by iNCEPTIONAL »

So, I just want to be real clear, the window/shape mask I would like to add on top of say all the layers in my ice level would not be using any transparency, just a solid opaque shape. Is there any reason the colour math has to be considered when using the window/shape masks only for a solid shape? Or is that literally what applying a solid colour to a window/shape mask is doing automatically, colour math, which is why it can't be applied at the same time as the colour math for the water layer?

I just want to check because it's like the other day when someone basically said you simply wouldn't use the Mainscreen/Subscreen unless doing colour math* or some variation of high-res, yet LMPunny did indeed use the Mainscreen/Subscreen to split the layers in my bridge level (all opaque) and rearrange them beyond what would normally be possible. So he actually allowed parts of BG3 to appear on top of BG2 simply by using the Mainscreen/Subscreen, but not for any typical colour math or high-res effects. Which means, if I really want to, I can indeed use the Mainscreen/Subscreen for something other than just colour math effects or variations of high-res.

Does that make sense what I'm getting at here, like, are we saying it has to be only done like this as possibly an automatic answer in this particular case, but maybe there's actually a relatively simple elusive-obvious way to do it without following the conventional rules of how to use these elements?

I just want to really confirm this is as an absolute immutable law of how the system works, which can't ever possibly be used in a slightly different way to the typical use-cases, before I go and remove some potential element of my shmup that ultimately didn't have to be removed at all.

Hopefully that doesn't come across as me being a stubborn ass who doesn't listen to other people's advice/suggestions but simply doing due diligence here.
Last edited by iNCEPTIONAL on Wed Jul 20, 2022 11:25 am, edited 4 times in total.
Myself086
Posts: 158
Joined: Sat Nov 10, 2018 2:49 pm

Re: Using window/shape masks behind background layers

Post by Myself086 »

iNCEPTIONAL wrote: Wed Jul 20, 2022 1:14 am So, I just want to be real clear, the window/shape mask I would like to add on top of say all the layers in my ice level would not be using any transparency, just a solid opaque shape. Is there any reason the colour math has to be considered when using the window/shape masks? Or is that literally what applying a solid colour to a window/shape mask is doing automatically, colour math, which is why it can't be applied at the same time as colour math in some other place?
If your windowed shapes use the color math, then the windows and the color math can't be used for anything else.
iNCEPTIONAL wrote: Wed Jul 20, 2022 1:14 am I just want to check because it's like the other day when someone basically said you simply wouldn't use the Mainscreen/Subscreen unless doing colour math or some variation of high-res, yet LMPunny did exactly that to split the layers in my bridge level (all opaque) beyond what would normally be possible. So he actually allowed parts of BG3 to appear on top of BG2, simply by using the Mainscreen/Subscreen, but not for any typical colour math effects. Which means, if I really want to, I can indeed use the Mainscreen/Subscreen beyond purely colour math effects or variations of high-res.
He used main&sub screen just to do it quick and dirty. Using main&sub screens for fixing BG priority means that you can't use main&sub screens for any other effect.
iNCEPTIONAL wrote: Wed Jul 20, 2022 1:14 am Does that make sense what I'm getting at here, like, are we saying it has to be only done like this an possibly an automatic answer, but maybe there's actually a relatively simple elusive-obvious way to do it without following the conventional rules of how to use these elements?
No, I don't get what you're getting at.
iNCEPTIONAL wrote: Wed Jul 20, 2022 1:14 am I just want to really confirm this before I go remove some potential element of my shmup that ultimately didn't have to be removed at all.
When working on a game, you should be scaling your assets down instead of up. Work on high quality assets that can be scaled down for the real hardware. Meaning you're still going to use 4 BGs but you can use more colors in your demo and then use a tool that'll automatically pick what colors to use in your palettes and apply them as best it can and then make manual adjustments (I know someone working on such tool).

What I'm saying is, you can wait until you find a programmer before removing parts of your level that you've already made. You can talk with your programmer about what can be removed or adjusted.
93143
Posts: 1717
Joined: Fri Jul 04, 2014 9:31 pm

Re: Using window/shape masks behind background layers

Post by 93143 »

You could use the window to cut a hole in all the layers and turn off colour math inside the hole, so just the main screen backdrop colour shows through.

Alternately, instead of disabling colour math inside the window, you could clip the main screen to black before math inside the window (while still also masking off all the layers, if there are any on the subscreen), which would show the subscreen backdrop colour without blending (assuming you're not using subtractive blending...).

Leaving colour math on inside the window and not clipping the main screen to black would result in the foreground shape showing up as a blend of the main and sub backdrop colours, which you may not want (any pixel transparent on both main and sub screens would be the same colour as your windowed foreground shape).

Also note that the two windows can be activated (and inverted) separately for BG1/BG2/BG3/BG4/OBJ/Color, meaning that if your shape can be drawn with one window, you can still use the other one for something else. This is not perfectly flexible (e.g. if you're clipping the main screen to black inside one Color window, you can't have another Color window that doesn't do that; if they're both active for Color they share the setting), but it's something.
User avatar
dougeff
Posts: 3079
Joined: Fri May 08, 2015 7:17 pm

Re: Using window/shape masks behind background layers

Post by dougeff »

Alternately, instead of disabling colour math inside the window, you could clip the main screen to black before math inside the window (while still also masking off all the layers, if there are any on the subscreen), which would show the subscreen backdrop colour without blending (assuming you're not using subtractive blending...).
I like this. You could draw a simple, one color, shape on the screen (Using HDMA to adjust window edges per scanline).

Since the window is cutting a hole through all the main screen layers, it would appear to be in front of everything else.

I think I will make a little demo and try it out.
nesdoug.com -- blog/tutorial on programming for the NES
iNCEPTIONAL

Re: Using window/shape masks behind background layers

Post by iNCEPTIONAL »

dougeff wrote: Wed Jul 20, 2022 8:46 am
Alternately, instead of disabling colour math inside the window, you could clip the main screen to black before math inside the window (while still also masking off all the layers, if there are any on the subscreen), which would show the subscreen backdrop colour without blending (assuming you're not using subtractive blending...).
I like this. You could draw a simple, one color, shape on the screen (Using HDMA to adjust window edges per scanline).

Since the window is cutting a hole through all the main screen layers, it would appear to be in front of everything else.

I think I will make a little demo and try it out.
Yes, this appears to be describing what I was saying, or at least the end result I was after anyway (in my case, ideally along with the semi-transparent water layer, but even without it would still be useful), so I look forward to seeing the little demo. :D

PS. And, since there's two window/shape masks, you could actually do two of those simple one colour shapes on the screen using HDMA to adjust the window edges per scanline for each shape, right?
User avatar
rainwarrior
Posts: 8732
Joined: Sun Jan 22, 2012 12:03 pm
Location: Canada
Contact:

Re: Using window/shape masks behind background layers

Post by rainwarrior »

iNCEPTIONAL wrote: Wed Jul 20, 2022 1:14 amI just want to check because it's like the other day when someone basically said you simply wouldn't use the Mainscreen/Subscreen unless doing colour math or some variation of high-res, yet LMPunny did indeed use the Mainscreen/Subscreen to split the layers in my bridge level (all opaque) and rearrange them beyond what would normally be possible. So he actually allowed parts of BG3 to appear on top of BG2 simply by using the Mainscreen/Subscreen, but not for any typical colour math or high-res effects. Which means, if I really want to, I can indeed use the Mainscreen/Subscreen for something other than just colour math effects or variations of high-res.
No, LMPunny is using colour math to do this. The main screen area has a black background, and the sub-screen is doing additive blending against the black.
iNCEPTIONAL

Re: Using window/shape masks behind background layers

Post by iNCEPTIONAL »

rainwarrior wrote: Wed Jul 20, 2022 10:16 am
iNCEPTIONAL wrote: Wed Jul 20, 2022 1:14 amI just want to check because it's like the other day when someone basically said you simply wouldn't use the Mainscreen/Subscreen unless doing colour math or some variation of high-res, yet LMPunny did indeed use the Mainscreen/Subscreen to split the layers in my bridge level (all opaque) and rearrange them beyond what would normally be possible. So he actually allowed parts of BG3 to appear on top of BG2 simply by using the Mainscreen/Subscreen, but not for any typical colour math or high-res effects. Which means, if I really want to, I can indeed use the Mainscreen/Subscreen for something other than just colour math effects or variations of high-res.
No, LMPunny is using colour math to do this. The main screen area has a black background, and the sub-screen is doing additive blending against the black.
OK, this is one of those examples where a clear miscommunication is happening between what I'm trying to convey and what is coming across and how it is being interpreted, so I'm not going to say anymore on it. I accept that you are correct that it is using colour math to achieve the result I am talking about. And I will potentially use LMPunny's particular solution to my layer issue in the future again for something that isn't necessarily an obvious semi-transparency effect, some version/variation of high-res, a more typical version/variation of a window/shape mask, or any of the other more standard and accepted uses for Main-screen/Sub-screen/colour-math. Cool?
SNES AYE
Posts: 201
Joined: Mon Nov 07, 2022 11:28 am

Re: Using window/shape masks behind background layers

Post by SNES AYE »

dougeff wrote: Wed Jul 20, 2022 8:46 am
Alternately, instead of disabling colour math inside the window, you could clip the main screen to black before math inside the window (while still also masking off all the layers, if there are any on the subscreen), which would show the subscreen backdrop colour without blending (assuming you're not using subtractive blending...).
I like this. You could draw a simple, one color, shape on the screen (Using HDMA to adjust window edges per scanline).

Since the window is cutting a hole through all the main screen layers, it would appear to be in front of everything else.

I think I will make a little demo and try it out.
Yes, this could have some interesting uses. Did you ever get around to creating that little demo?
Post Reply