Stencil buffer

In this program the stencil buffer is filled with 1s wherever a white stripe is drawn and 0s elsewhere. Two versions of each oval, square, or triangle are then drawn. A black colored shape is drawn where the stencil buffer is 1, and a white shape is drawn where the buffer is 0.

A stencil buffer is an extra buffer, in addition to the color buffer and depth buffer (z-buffering) found on modern graphics hardware. The buffer is per pixel, and works on integer values, usually with a depth of one byte per pixel. The depth buffer and stencil buffer often share the same area in the RAM of the graphics hardware.

In the simplest case, the stencil buffer is used to limit the area of rendering (stenciling). More advanced usage of the stencil buffer makes use of the strong connection between the depth buffer and the stencil buffer in the rendering pipeline. For example, stencil values can be automatically increased/decreased for every pixel that fails or passes the depth test.

The simple combination of depth test and stencil modifiers make a vast number of effects possible (such as stencil shadow volumes, Two-Sided Stencil,^[1] compositing, decaling, dissolves, fades, swipes, silhouettes, outline drawing or highlighting of intersections between complex primitives) though they often require several rendering passes and, therefore, can put a heavy load on the graphics hardware.

The most typical application is still to add shadows to 3D applications. It is also used for planar reflections.

Other rendering techniques, such as portal rendering, use the stencil buffer in other ways; for example, it can be used to find the area of the screen obscured by a portal and re-render those pixels correctly.

The stencil buffer and its modifiers can be accessed in computer graphics APIs like OpenGL and Direct3D.

OpenGL

glEnable(GL_STENCIL_TEST); // by default not enabled
glStencilMask(stencilMask); // allow writing to stencil buffer, by default (0xFF) no mask.
glClearStencil(clearStencilValue); // clear stencil value, by default = 0
glStencilFunc(func, ref, mask); // by default GL_ALWAYS, 0, 0xFF, always pass stencil test
glStencilOp(fail,zfail,zpass); // by default GL_KEEP, GL_KEEP, GL_KEEP, dont change stencil buffer
glClear(GL_STENCIL_BUFFER_BIT); // clear stencil buffer, fill with (clearStencilValue & stencilMask)

Test: ( ref & mask ) func (stencilValue & mask)

Depending on the three possible conditions of stencil function/depth function.

1. Stencil Test Function fails:

   If say func is GL_NEVER, the stencil test will always fail. 
   Neither Color/Depth buffers are modified. Stencil buffer is modified as per glStencilOp fail.
   If say glStencilOp(GL_REPLACE, GL_KEEP, GL_KEEP) then GL_REPLACE takes place and 
   stencilValue = (ref & stencilMask) // will become ref

2. Stencil Test Function passes/Depth Test Function fails:

  If say func is GL_ALWAYS, the stencil test will always pass, but depth test may fail.
  Neither Color/Depth buffer are modified. Stencil buffer is modified as per glStencilOp zfail.
  If say glStencilOp(GL_KEEP, GL_INCR, GL_KEEP) then GL_INCR takes place and
   stencilValue = (stencilValue+1) // will become 1

3. Stencil Function passes/Depth Function passes:

  If say func is GL_ALWAYS, the stencil test will always pass. If depth test also passes.
  Both Color/Depth buffer are modified. Stencil buffer is modified as per glStencilOp zpass.
  If say, glStencilOp(GL_KEEP, GL_KEEP, GL_KEEP) then Stencil values are not changed, only Color and Depth buffers are modified.

Typically Stencil buffer is initialized by setting depth buffer and color buffer masks to false. and then setting appropriate ref value to stencil buffer by failing the stencil test every time.

  // disable color and depth buffers
  glColorMask(GL_FALSE, GL_FALSE, GL_FALSE, GL_FALSE);
  glDepthMask(GL_FALSE);

  glStencilFunc(GL_NEVER, 1, 0xFF); // never pass stencil test
  glStencilOp(GL_REPLACE, GL_KEEP, GL_KEEP);  // replace stencil buffer values to ref=1
  glStencilMask(0xFF); // stencil buffer free to write
  glClear(GL_STENCIL_BUFFER_BIT);  // first clear stencil buffer by writing default stencil value (0) to all of stencil buffer.
  draw_stencil_shape(); // at stencil shape pixel locations in stencil buffer replace stencil buffer values to ref = 1

Now use the initialized stencil buffer and stencil test to write only in the locations where stencil value is 1

  // enable color and depth buffers.
  glColorMask(GL_TRUE, GL_TRUE, GL_TRUE, GL_TRUE);
  glDepthMask(GL_TRUE);

  // no more modifying of stencil buffer on stencil and depth pass.
  glStencilMask(0x00);
  // can also be achieved by glStencilOp(GL_KEEP, GL_KEEP, GL_KEEP);

  // stencil test: only pass stencil test at stencilValue == 1 (Assuming depth test would pass.) 
  // and write actual content to depth and color buffer only at stencil shape locations.
  glStencilFunc(GL_EQUAL, 1, 0xFF);
 
  draw_actual_content();

See also

• Z-buffering (depth buffer)
• Shadow volume (a stencil buffer technique creating shadows)

References

1. https://msdn.microsoft.com/en-us/library/windows/desktop/bb206123(v=vs.85).aspx