Method 1 — Alpha Channel as a Clipping Channel

Digital artists often think of compositing in this way: where the alpha channel is white, use one image. Where the alpha channel is black, use another image. Where it is gray, use a blend of the two. In effect the alpha channel "clips back" the overextend RGB image.

This method works fine when one wishes to create a transitional effect between two scenes called a wipe.

This technique also serves when one needs to isolate a foreground element and extract it from a photographic background.

For more information on this method, see my additional page on Using the Alpha Channel as a Clipping Channel.

Using the alpha channel as a clipping channel fails when the foreground object is an already isolated CG object rendered against black with an alpha channel. In that case we need another approach . . .

Method 2 — Alpha Channel as a Holdback Matte

Consider the computer-generated sphere shown below. In this case, when you use the alpha channel to unnecessarily extract the foreground from its black background, you pull in gray around the edge, creating a matte line.

Rather than solve the problem, many compositors opt to live with this effect, or "fix it in post" by choking in on the matte.

One might tolerate or fix the matte lines around a sharp-edged object because the problem area is only one pixel wide, but notice how problematic the hazy gray artifact becomes around soft-edged or motion-blurred objects.

A compositor who attempts to fix these matte lines by "choking in"on the alpha channel will only "solve" the matte line problem shown above at the expense of the motion blur. He soon finds himself forced to accept either significant matte lines or spoiled motion blur — a poor choice one way or the other.

A CG animator can anticipate this dilemma and take steps to compensate for the compositor's technique by using, for example, Maya's "composite" option (a check box under Render Options), or perhaps by using Shake's "RGB over A" MDiv node to divide the CG image's foreground RGB values by its own alpha channel, thereby nullifying the compositing software's unnecessary multiplication with an equally unnecessary preemptive division ...

... notice along the edges of the sphere pictured above how the division "unpremultiplies" the antialiasing, causing the normally smooth, antialiased RGB values to become jaggy, and to overshoot the alpha region.

This effect occurs because 100% foreground over 100% alpha is, of course, 100% RGB. But 50% foreground over 50% alpha is also 100% RGB. In fact, 17.265% foreground over 17.265% alpha is, once again, 100% RGB. In fact, any percentage color value divided by the same percentage alpha value causes that color to be "normalized" to its 100% value. (Mathematical purists note: to avoid division-by-zero errors, these methods skip black alpha pixels. It may sound kludgey, but it works well in practice)

In many cases one can force naive compositing software to "get it right" by simply supplying "pre-divided" foregrounds as a matter of policy. (see above)

---

But sometimes the sophisticated CG artist wishes to create RGB values that are intentionally decoupled from the alpha values — as sometimes happens in the case of reflections or glows. (see below)

Feeding the compositor pre-divided RGB values won't work here, because the RGB values of the image are neither premultiplied by the alpha, nor are they intended to be post-multiplied by it. The alpha channel is simply a holdback matte for the background image. To use such an alpha channel, one must emulate the behavior of a film optical printer — one must run the black core matte as a holdback for the background, and then add the foreground as a burn-in, or a "DX" (a double exposure).

For more information on this method, see my additional page on Using the Alpha Channel as a Holdback Matte

In other words:

Invert (~) the alpha channel and multiply ( * ) it by the background. Then sum ( + ) in the foreground with simple addition. The foreground doesn't need to be multiplied by its own alpha — it's already extracted — and it's the perfect shape to drop right into that hole you just made with the black core matte. Look at the sequence of images above to see how well this works. Notice how pixels associated with zero alphas just "DX" on top of undiminished background pixels, an effect which offers a nice added benefit for those who choose to avail themselves of it.

— and an effect that looks a whole lot better than what you see below: