A layer mask cuts a layer with vector shapes of its own — a circle that frames a face, a hand-drawn outline around a product, a rectangle that wipes open over a second. It is the After Effects mask: several shapes per layer, combined with Add, Subtract, Intersect or Difference, each with its own feather, expand, opacity and an animated Grow, and every one of them keyframable, the outline included. Layer masks are free.
Masks work on video, image, text, shape and MotionFast layers, and on adjustment layers, where they limit the grade to an area instead of cutting pixels. A sequence, grid or 3D model layer has no mask of its own — open the sequence or grid and mask the layers inside it.
Where masks live
Select the layer and open the Mask tab on the inspector’s rail (the dashed-circle icon). It holds the LAYER MASK section. The tab shows a green dot whenever the layer has a mask, so a masked layer is visible at a glance.
The section header carries an On/Off switch and, while masks are on, a count of how many there are. Clicking On switches the section off again and removes the layer’s masks — ⌘Z brings them back.
Below the mask list sits the RotoScope block on a video layer, or Magic Mask on an image layer. Those trace a subject for you and write the result as a mask; they have their own page: RotoScope and Magic Mask.
Add a mask
Click Off in the section header. The section switches on and the canvas editor opens straight away with the Pen live and an empty stack, so the first mask is whatever you make:
- A preset — the Masks row adds + Circle, + Rectangle, + Rounded or + Triangle, centred on the layer.
- A drag — pick Rect or Ellipse in the Tool row and drag on empty canvas. The tool hands back to Select as soon as the shape is drawn, ready to reshape it.
- A drawing — use the Pen (below).
Edit masks on canvas opens and closes the editor later; while it is open the button reads Done editing masks and the Tool row offers Select, Pen, Rect and Ellipse. A layer holds up to 16 masks.
Draw with the Pen
- Click the canvas to place points. A placed point is smooth — the outline curves through the points you give it, so four clicks around something round make a round shape.
- Drag out of a click to pull that point’s tangent yourself, or Alt-click to place a sharp corner.
- While drawing, every placed point stays live: click one to select it, drag it, or pull its handles (Alt-drag a handle to move one side only). Alt-click a placed point to switch it between smooth and corner.
- Close the mask by clicking the first point again or pressing Enter. Backspace removes the last point and Escape drops the drawing.
A closed mask is an ordinary mask from that moment — same box, same settings, same keyframes — and the editor switches to Select.
On a finished mask the Pen keeps working: click an anchor of the selected mask to select it and drag its handles, click the outline to add a point, Alt-click an anchor to toggle a corner and press Delete to remove it, all without switching tools.
Undo, one point at a time
While a drawing is open, ⌘Z / ⌘⇧Z (Ctrl+Z / Ctrl+Shift+Z on Windows) step it back and forward one point at a time instead of undoing the project underneath. Undoing a mask you have just closed brings the drawing back with every point and its own history, with the Pen live again; redo closes it once more.
Edit points and the mask box
With the Select tool:
| To | Do this |
|---|---|
| Reshape | Drag an anchor, or drag one of its tangent handles (Alt breaks the mirrored pair). |
| Add a point | Double-click the outline (with the Pen, a single click on the outline). |
| Toggle smooth / corner | Alt-click an anchor. |
| Select several points | Drag a marquee over empty canvas, Shift– or ⌘-click anchors, or press ⌘A for all of them. |
| Move, scale or turn a group | Drag inside the group box (Shift keeps to one axis), drag a grip to scale (Alt about the centre, Shift keeps proportions on a corner), or drag the knob to rotate (Shift snaps to 15°). |
| Nudge | Arrow keys move the selected points 1 px, 10 px with Shift. |
| Delete points | Delete or Backspace. An outline keeps at least three points; a delete that would leave fewer does nothing. |
| Delete the mask | Delete with no point selected removes the selected mask. |
| Move or resize the whole mask | Drag inside the selected mask to move it, drag the box grips to resize it, drag the rotation knob to turn it (Shift snaps to 15°). |
| Pick another mask | Click its outline on the canvas, or its card in the panel. |
Each mask’s outline is drawn in its own colour, matching the dot on its card, and a feathered mask shows a second, fainter outline where the feather reaches.
The point magnifier
While you drag an anchor, a tangent handle or a Pen point, a round magnifier appears beside the pointer. It shows the spot under the point at 2.5× with the outline drawn through it, so you can put an edge exactly on the pixels your cursor or finger would otherwise hide. It sits on whichever side has room and disappears when you let go. The same magnifier follows region points in RotoScope and Magic Mask.
On a tablet or phone
Anchors, tangent handles and the Pen all follow a finger. While you drag, the point rides a little above your fingertip so you can see it, and the magnifier shows what is underneath. Two quick taps on an anchor switch it between smooth and corner (Alt-click on a mouse), and a still press-and-hold on an anchor deletes it (the Delete key on a keyboard). A feathered mask renders soft on tablets and phones exactly as on the desktop. See Web Version, Tablets and Phones.
Several masks, modes and order
Each mask is a card in the list. Its head has an enable dot (in the mask’s colour — click it to switch that mask off without deleting it), the name, the mode, move up / move down arrows and a remove button that takes the mask’s keyframes with it. Under the head you see the point count, a keyed badge when the mask is animated, and disabled when it is switched off. The open card is the selected mask — the one the canvas edits.
| Mode | What the mask does to the result so far |
|---|---|
| Add | Adds its area. |
| Subtract | Cuts its area away. |
| Intersect | Keeps only where it overlaps what is already there. |
| Difference | Keeps where either one covers, but not both. |
The stack is combined from the top card down, so order matters — swapping an Intersect and a Difference changes the result. If the first mask is Subtract or Intersect, the stack starts from the full layer instead of from nothing, so a lone Subtract mask cuts a hole rather than blanking the layer. A circle with a smaller Subtract circle under it is a ring.
Mask settings
Open a mask’s card to set it:
| Setting | What it does | Range / default |
|---|---|---|
| Mode | Add, Subtract, Intersect or Difference (above). | Add |
| Invert | Swaps inside and outside for this mask. | Off |
| Opacity | How strongly this mask counts in the stack. | 0–100 %, 100 |
| Feather | Softens the edge. | 0–500 px, 0 |
| Expand | Pushes the edge outward; negative pulls it in. | −500 to 500 px, 0 |
| Grow | Moves the edge at a steady speed over the clip (below). | −10000 to 10000 px/s, 0 |
| X / Y | Where the mask’s box sits, measured from the layer’s centre. | −300 to 300 %, 0 |
| Width / Height | The mask box’s size. | 2–400 %, 100 |
| Rotation | Turns the mask box inside the layer. | −1080° to 1080°, 0 |
X, Y, Width and Height are percentages of the layer’s own box, so a mask follows the layer when you resize, zoom, rotate, parent or keyframe it — and stays exact on a layer tilted in 3D. Feather, Expand and Grow are in composition pixels, so a 20 px feather is 20 px in the final frame whatever the layer’s scale.
The Shape group reshapes the selected mask from a preset — Circle, Rectangle, Rounded, Triangle, Diamond, Pentagon, Hexagon, Star, Burst, Blob or Squircle. The Path row below shows the point count, the outline’s keyframe stopwatch and Smooth.
Two settings apply to the whole stack and sit above the list:
- Amount (0–100 %, default 100, keyframable) fades the whole stack. At 0 the layer is untouched, at 100 the masks cut fully — keyframe it to bring a mask in or out.
- Feather quality — Fast or High. High rasterises the mask more finely, which smooths a very wide feather and keeps a hard edge crisp when the layer is scaled up.
Grow: a reveal without keyframes
Grow is a speed in composition pixels per second, added to Expand as the clip plays. Give a small circle a Grow of 800 px/s and it opens across the frame on its own; a negative Grow closes a mask instead. Keyframe Grow for a reveal that speeds up or slows down — the speed is accumulated over time, so a change in rate accelerates the edge smoothly rather than making it jump. At the top of the range a mask opens across a 4K frame in well under a second.
Animate a mask
Every number in a mask card, and Amount, has a keyframe stopwatch next to its label. Click it and the property gets a keyframe track holding its current value, ready in the Keyframe Builder, and the field becomes read-only — the keyframes own the value from then on. Once a property is keyed, dragging the mask box or its points on the canvas writes a key at the playhead. Click the stopwatch again to remove the track.
The Path row’s stopwatch animates the outline itself — Animate the outline (morph). The first key holds the outline on screen at that moment; move the playhead, reshape the points, and the mask morphs between the two shapes. Adding or removing points on an animated outline changes every key at once, so the morph always matches point for point. Picking a Shape preset on an animated outline writes it as a key at the playhead.
On the timeline, a mask’s tracks are named after the mask and the property — Mask 1 · Feather, Roto · Path — and they work in the Keyframe Builder and the graph editor like any other property.
Smooth a mask outline
A traced or hand-drawn outline often carries more points than you want to fine-tune. Smooth on the Path row opens the Smooth mask popover, which previews on the canvas as you tune it and reports the point count before and after, the Max deviation from the original and the Length change:
| Method | What it does |
|---|---|
| Fit | Refits the curve with as few points as the tolerance allows. Fewest points, but they land in new places. |
| Reduce | Deletes only the redundant points; the ones that stay keep their exact position. |
| Even | Rebuilds the outline with an exact number of points, evenly spaced. |
| Relax | Keeps every point and averages out the shake — for traced and tracked outlines. |
Keep corners over protects turns sharper than the angle you set. Apply writes the result as one undo step; Cancel or Escape puts the outline back. On an animated outline, Fit is unavailable (it picks its own point count per shape); Reduce drops the same points from every key, Even rebuilds every key with the same count and Relax smooths each key in place.
Copy a path into a mask
The copy button in the Shape header opens Copy a path, which lists every other path in the project — motion paths, camera paths, shape outlines and other layers’ masks. Pick one and this mask takes on that outline. It works the other way too: masks appear as sources wherever another path can be copied from. See Motion Paths.
Masks on adjustment layers
On an adjustment layer a mask does not cut pixels — the layer has none of its own. It limits where the adjustment applies: inside the mask the layers below get the grade, the LUT and the Overlay Effects chain, and outside it the untouched stack shows through. That is a local grade: brighten one face, cool one corner of the sky, blur only the background.
Everything above carries over — several masks and modes, feather, expand, grow, keyframes and the canvas editor. Amount at 0 fades the mask away so the grade covers the whole frame again. The adjustment layer’s box spans the frame by default and follows the layer’s own transform. Masked adjustment layers work at the top level and inside sequences.
How masks meet the effect chain
A mask cuts the layer’s own pixels before almost everything else on the layer runs — the After Effects order, masks first and the track matte last. In practice:
- Chroma Key runs before the mask, so a mask trims the keyed subject. Use a loose mask as a garbage matte to cut away light stands and the edges of the screen that the key cannot clean. See Chroma Key.
- Running Glow traces the mask’s edge rather than the layer’s rectangle, and camera depth of field defocuses the masked silhouette.
- An overlay effect with Clip to layer shape switched on stays inside the mask. See Overlay Effects.
- On video, image and MotionFast layers, the layer’s border and shadows follow what the mask leaves visible instead of the rectangle it cut away, and Shape Rulers, Neon Glow and Lines Draw frame the visible area. With Amount below 100 the cut-away part still shows, so these go back to the whole layer.
- A Track Matte gets the last word. A mask cuts with the layer’s own shapes; a track matte cuts with another layer’s alpha or brightness, after the effects. A layer can have both. See Track Matte.
The full order is on the Layer Effects page.
Where to go next
- RotoScope and Magic Mask — trace a moving subject into a mask instead of drawing it
- Motion Tracking — make a layer follow a detail in the footage
- Track Matte — mask a layer by another layer
- Layer Effects — the full per-layer order of operations
- Keyframe Builder — timing and easing for mask keyframes
Leave a Reply