Guild perk

A guild perk is one of several passive bonuses and active abilities granted by being a member of a guild. All are "quality of life" improvements that do not affect combat or the capacity to progress through game content. Therefore, players are encouraged to join a guild without being massively punished for abstaining from that part of the game. Several perks focus on play with other guild members, but solo and non-guilded group play benefit as well. Perks were first added as part of an extensive guild advancement system in which each of 25 guild levels granted a perk (or a second level of one), but ultimately the system was simplified to just grant a small fixed set for any member of any guild with no advancement required.

Guild perks can be viewed on the "Perks" tab in the guild window, as well as in the spellbook's general tab. The active abilities can be dragged to action bars or used directly from either location, as well as used in macros like any other ability.

While access to guild rewards is certainly a benefit of joining a guild, they are not considered "guild perks".

List of perks
The following four passive bonuses become active upon joining a guild.
 * Mount Up: Faster mounted movement.
 * Hasty Hearth: More frequent hearthstone availability.
 * The Quick and the Dead: Faster recovery after dying.
 * Guild Mail: Instantaneous mail within the guild.

These following is available to all members of a guild.
 * Mobile Banking: Summon remote access to the guild bank.

Removed
The list of guild perks has seen a severe reduction since their introduction in the Cataclysm expansion. Warlords of Draenor in particular saw the dissolution of the guild advancement leveling system, and with it most perks.

Patch 5.0.4
The developers chose not to continue with cauldron-based raid consumables after Cataclysm, so related perks were removed:
 * Chug-A-Lug
 * Happy Hour

Additionally and controversially, it was felt that the ability for one person to summon an entire raid to an instance discouraged exploration and involvement in the wider world, leading to its removal :
 * Have Group, Will Travel

Patch 6.0.2
The dismantling of the guild advancement system was part of a paradigm shift in the developers' thinking of how guild benefits should work. In most cases, it had become the norm for the majority of players to be in maximum-level guilds, and developers were designing systems with those bonuses in mind, making them mostly pointless. The tuning of numbers (experience, reputation, honor, trade skill, justice, and harvesting gains, durability losses, flight path speed, and vendor prices) was therefore changed to simply use a "baseline" as if the perk was in effect, and the perks themselves were removed:
 * Fast Track
 * Mr. Popularity
 * Reinforce
 * The Doctor Is In
 * Honorable Mention
 * Working Overtime
 * For Great Justice
 * Ride Like the Wind
 * Bountiful Bags
 * Bartering

Finally, one perk was removed due to its abuse by people who would create guilds and spam invites to unsuspecting new players simply to take all the gold generated from their gameplay via the perk: Blizzard stated that it would pursue other options to help guilds recoup costs from raiding, such as increasing the number of desirable Bind on Equip items to sell.
 * Cash Flow

This post by Blizzard's Lead Encounter Designer, Ion Hazzikostas, explains the changes in detail.

Patch 7.0.3
With the release of Legion, the Mass Resurrection (guild perk) perk was removed in favor of adding a baseline spell with the same functionality to all healing specializations (Absolution, Ancestral Vision, Mass Resurrection, Reawaken, and Revitalize).