Goddammit! I don't know if 3D graphics designer use Photoshop or not, and if they do, how they manage to put with it all day, but talk about a product not being intuitive... On the other hand, it's true that I know a thing or two about how Adobe doesn't care that much about any of the software they design to be either customer friendly or intuitive...
So, today, in our new "What we thought we'd do in about 30 seconds in Photoshop CS4 but ended up doing in hours" saga, we'll see how to convert a shadow effect layer into an alpha channel. The purpose here is to display a nice transparent text overlay texture, with a blending shadow against a non uniform background in a 3D animation (eg. DirectX, OpenGL, ...).
And because I'm in a generous mood today, I'm even gonna drop some illustrations here in there.
Let's say that you are creating an animated "motivational poster", like the one on the left, and your goal is to do some nice animation of the motto like a zoom out for instance, while making sure the motto has a shadow for added impact.
You will of course break down your animation between a static background and a text layer, loaded as a texture, and animate the latter.
Now, if you merely save the text layer with a shadow effect in Photoshop and use that as your texture, it will not blend nicely at all, as the shadow will be a solid colour, and eat part of the the nice aperture green logo. Therefore you want to add an alpha component to your texture, which you'll use for shadow blending.
At this stage, I'll suppose that you have your nice text layer with its shadow effect. The good part about all this ordeal is that you won't have to create a separate layer out of the effect, or play with adding/substracting selections, because a black text display with its shadow is pretty much exactly what you need to use for the alpha channel. The only thing is, if you use coloured or texturized text, you'll want to convert it to solid black before the next operations, through a duplicate layer or something.
Now, if you're a Photoshop novice and you've done your research, you should be aware that the alpha components of an image is dealt with as a channel in Photoshop, and that there is a tab for that (again non-intuitive as hell: where is my "right click -> new channel" option, Adobe? And how am I supposed to know what the small icons at the bottom represent when there is no bleeping contextual help in your product?!?). So you'd think that, with all the graphics people dealing with channels, depths masks and alpha all day long, there'd be a bloody "Convert Layer to Channel" function readily available by default in the interface. Well, if it was straightforward to work with alpha channels in Photoshop, that would make the life of graphics designers too easy, and obviously this is not something that you would ever want.
So, after cursing for about 10 minutes about those non-intuitive menus, we have to fallback to looking on the internet for some helpful souls (and I got to give it to CreativeCow there, for providing all the actual solution steps):
- Search into your Photoshop installation directory for .atn files (AcTioNs) and locate the one labelled "Video Actions.atn". Mine was in "D:\Program Files\Adobe\Adobe Photoshop CS4 (64 Bit)\Presets\Actions"
- In Photoshop, go to Window -> Actions. You will see a set of "Default Actions", but of course, not all the Actions that come with Photoshop have been loaded by default in one of the more obscure menu of the product, where noone would mind if they had been.
Then click on what has to be the most useful button for advanced users, and therefore the one that's been made as easy to overlook as possible, namely that bleeping contextual dropdown menu.
And I really have to wonder what is it with this late redesign of UIs by Adobe and Microsoft (eg. Office 2007), whose only purpose seems to make the power users' life a living hell by hiding the most powerful feature in a place where the sun don't shine no more...
- In that menu, you will find a "Load Action" option, which you should use to navigate to the folder where you spotted the "Video Actions.atn" file, and open that file.
Alternatively, you can skip step 1 and directly select "Video Actions" in that menu to make the Video Actions appear. Why "Video Actions" is in that menu but not in the main action window by default is really beyond me here...
- OK, now we're talking: In the "Video Actions" section, the first 2 actions are: "Alpha Channel from Visible Layers" and "Alpha Channel from Visible Layers (inverted)".
The Ark of the Covenant at last! And all we had to do was get our face and soul melted for it - a bargain! Merrily we go then by setting only our "black text with shadow" layer visible (make sure you don't select the background) and then clicking the little "Play" button in the Actions menu with "Alpha Channel from Visible Layers" selected. You should end up with a new channel that looks like the one on the left. In this image, white is for pixels with no alpha (completely opaque) whereas black is for pixels with 100% alpha (completely transparent).
- Now, before we save our image with its newfound alpha channel, we want to lay it over black solid background for the shadow, so just create one and save only the text layer (with or without the shadow effect - it shouldn't matter) over the black background as your image with alpha. You might want to enable the alpha channel (set the eye) after creating the black background to have a better idea of what exactly you're saving.
- Now because there does not appear to be clear conventions as to what a zero alpha value is meant to represent (this can vary from application to application), you might find that the alpha mask you get in your application does the opposite effect of what you wanted. If that's the case, rather than redo the steps using the "Inverse" function for layer to alpha we've seen in the menu, you can simply double click on your Alpha channel in Photoshop and toggle between "Color Indicates Masked Area" / "Color Indicates Selected Area", as this will invert your Alpha channel
And that, sir, is how you do it.
Of course, if you look at the actions, you can probably figure out how to achieve the same from the various menus (haven't really figured that one just yet, especially where a on/off selection is converted to a grayscale range). I also haven't figured out how you can insert one of those actions into the standard Photoshop menus (wouldn't it sit nicely in the layer menu?). Oh and finally, if like me you've cursed Photoshop's stupid and uninformative handling of undo (Why is it that a 10 year old program like Paint Shop Pro 5 does a much better job at undo than Photoshop?), note that you can access the undo history from the Window menu as well, where you will actually get a description of the actions you have taken.
