The Cannon Master himself is not a very difficult foe to defeat, but reinforcements (Crimson Riflemen) continually arrive, stand in one spot, and fire on your party. They can be killed by loading and firing the cannons beside the Cannon Master with ammo from the piles beside the Cannon Master.
Before the fight, designate someone to be the "Cannon Boy". Healers or ranged DPS are the best candidates for this. Periodically, this person should fire the cannon to kill all the reinforcements that spawn at the end of the hall, though conservation of ammo is important. Always hold on to at least one cannon ball, as Willey's death spawns a bunch of the Riflemen, which are very hard to take out by more traditional methods.