Fear is an affliction; a negative status effect that handicaps the target.
A creature under the effect of Fear is unable to take any action while they are feared, though they can still move. They also can't dodge or parry any physical attacks directed at them while feared. AI-controlled creatures will usually run away while under the effect of Fear.
Players are able to inflict Fear on a target using the fear spell from the Defensive Affliction Magic skill.
Players can increase their chances of resisting a Fear Affliction spell by wearing jewelry that raises their Prime Roots resistance.
Vorax are also able to induce Fear in their enemies, but this is not a magical effect and thus Prime Roots resistance will not help you resist it. Creatures that are immune to Fear spells similarly are not immune to this effect.