Definitely do this in AE instead of Premiere. It's possible in premiere but the workflow in AE is going to be a LOT faster/easier (especially if you want to do more complex things with the mask and effects). The exception is if it's just a one-time thing or you just need a very simple mask for the effect; then it might be worth it if you want to keep all of your workflow in premiere.
Steps for Premiere:
■ Apply any video effect to a video clip.
■ Open the Effect Controls menu and select/expand the effect you just applied.
■ Use the mask tools located directly under the name of the effect to create a mask for that effect.
You can create multiple masks for each effect. By default, a mask defines an area in which the effect will be applied; in other the effect will be applied to the areas contained in the masks and ONLY those areas. However, you can change this behavior by toggling the
inverted flag for each individual mask.
■ If all masks on an effect have inverted set to off (the default), the behavior is as described above.
■ If all masks on an effect have inverted set to on, then the effect will be applied everywhere EXCEPT the areas contained within the masks.
■ If at least one mask has inverted enabled and at least one mask has inverted disabled, then the effect is applied everywhere EXCEPT the areas contained within inverted masks but not within un-inverted masks.
You can also animate masks by selecting the "Track mask forward/backward" options, which will create keyframes for the mask on the appropriate frames.
I'll also include steps on how to do the masking in AE using a method that I prefer over THM01's for its flexibility.
Steps for AE:
■ Create a new adjustment layer in your comp and move it directly above the layer you want to modify. I will refer to this layer as "FX".
■ Apply any effects you wish on the FX layer. This will apply the effects to every (visible) layer below FX.
■ Create a new solid white layer and move it directly above FX. Then turn this new layer into a luma matte for the adjustment layer by selecting the "Luma" option in the TrkMat dropdown menu for FX. I will refer to this matte layer as "Matte".
■ Create masks in Matte to define the areas that you want your effects to apply to. You can do this by right-clicking Matte in the timeline panel and selecting Masks...New Mask from the pop-up menu. Or you can just start drawing on the preview panel (making sure Matte is selected) with the shape or pen tools and it will automatically create a new mask from the shape you create.
Here's an example of what the timeline panel looks like after going through the above steps, taken from one of my projects. In it, I apply a CC radial blur to the bottom source footage layer using the
Sword Blur adjustment layer, which uses
Mask as a luma matte.
Mask contains a single mask ("Mask 1") that defines the area where
Sword Blur, and therefore also where the radial blur, is visible in the frame.
You can animate masks in AE by enabling Time-Vary (the stopwatch thing) for the "Mask Path" property for a given Mask. You can also apply multiple masks to your luma matte layer if you want your effects to be visible in mutliple, discrete areas. You can add, subtract, and blend multiple masks on the same layer in more complex ways than in premiere.
If you want to apply the same mask(s) to multiple effects, just apply all of the effects onto a single adjustment layer and define the masks once in a luma matte layer. This definitely beats the process in premiere, where you have to copy-paste masks between different effects if you want to sync masks over two or more effects.