Moderators can refund bounties, the primary purpose of this ability is to allow moderators to close questions that are bountied. If you encounter a question that should be closed, but is bountied, just flag for moderator attention.
In general this case should be very rare. You can only add a bounty after 2 days, so the question has to evade closure for that long. For that reason this is rarely an issue, but sometimes a problematic question does evade closure long enough. This is rare enough that it is sufficient for moderators to deal with it.
The big issue with allowing the community to handle this case is the question of what happens to the reputation. This is not a simple decision, refunding a bounty on a question that was featured for several days pretty much gives that user the advantages of a bounty for free. Closing without refunding a bounty could remove that reputation without the user getting the benefit of the bounty. These options are just a bit too dangerous to give to the community, who might not even be aware of the subtleties when closing a question.