An instance reset is a way of fully or partially restarting an instance. This way, mobs will come back or, in some cases, unbug.
Soft reset[]
A soft reset is when all party members leave the instance, and then come back in. This puts people out of combat, making mobs run back to their original places.
The term "soft reset" can also mean having all party/raid members leave the instance for at least 30 minutes. This causes any trash mobs that have been killed to respawn. The most common reason to do this (deliberately) is to farm trash mobs in raid instances, for reputation or rare drops.
Hard reset[]
A hard reset is when all party members leave the instance, and the instance leader right clicks over his own portrait and the clicks the "Reset Instance" button. This restarts the whole instance, and all mobs will be back.
Heroic dungeons and Raid instances cannot be hard reset manually. All heroic mode instances (and thus, the IDs) will hard reset at the same time region-wide every day. The date of the hard reset of raid instances depends on the instance.
If you are in such an instance at the time of the reset, you will be forcibly returned to your hearthstone's home. At 60, 45 and 15 minutes before the reset there is a warning that the instance will reset. At 10 minutes before the reset you will be informed that you will be forcibly returned to your hearthstone's home.
When resetting a normal dungeon, some restrictions apply. You cannot enter more than five instances per hour, even if they are of the same dungeon. This was added to prevent farming an instance's trash mobs over and over again without killing a boss.
Resetting in Heroic Instances will reset trash but not bosses.