Alpha Channel as a Clipping Channel

A key mix can be a useful technique for many compositing tasks, particularly composites known as wipes.

The key mix technique is poorly suited for compositing 3D CG elements that have been rendered over black backgrounds. When used for this purpose, it produces matte lines, those areas of gray haze that surround the foreground element.

Matte lines are caused by an unneccesary multiply that occurs in the key mix compositing process. The multiply occurs between the foreground element and its own alpha channel. One way to improve the results produced by a key mix used in this manner is to introduce a preemptive divide into the process - dividing the foreground by its own alpha channel in anticipation for the multiply which occurs in the key mix.

Many programs (such as Photoshop) do not offer the ability to perform image division. Many other programs, such as Maya, offer division, but perform poorly when called upon to execute the mathematically undefined operation of "division by zero" (which occurs when the image in the dividend contains black pixels). The program Shake is smart enough to skip black pixels. Shake produced the image you see above. Notice how dividing a beauty pass by its own matte (to use "optical printer" terminology) causes the beauty pass to overshoot its matte.

By inserting this preemptive "divide by alpha" into the key mix process, you can force key mixing to be an acceptable procedure for compositing 3D CG. If you must.

Look ma! No matte lines!

I'm not a big fan of this technique. I prefer to treat the alpha channel as a
holdback matte for the background - in Shake this is achieved by using the over node for compositing.

More on that other technique here:

http://www.digitalartform.com/archives/2004/10/alpha_channel_a_1.html

And here's a related page on matte lines and transparency mapping:

Transparency Mapping and MATTE LINES:
http://www.digitalartform.com/archives/2004/12/transparency_ma.html