Binary spells are spells which can only have full effect (hit) or no effect (resist); partial resists are not possible. Normally, damage spells are only binary if they have an additional non-damage effect. Some examples are 
[Death Coil], and 
[Mind Flay]. 
[Frostbolt] is no longer a Binary Spell.
The Binary Spell distinction is no longer important in later expansions, as damage resistances were removed.
For binary spells, both spell penetration and spell hit rating can be used to overcome resistances.
DoTs used to be binary spells, but this was changed.