Kampot has a lot Wats. These are Buddhist Temples that are found throughout Cambodia. They house the monks that you will see most mornings, walking barefoot down the street, and stopping at homes and businesses along the way. They are collecting food or money to support the Wat, and give a short prayer to the giver.
There are several Wats, however, the main Wat is on the other side of the river from downtown Kampot. It's about a 5 minute ride from the downtown, and worth a visit. Kampot's main Wat is on the Riverside, and you'll have to cross a bridge, (or take a boat) to get to it.
As with any Wat, be quiet and respectful while in the Wat. You shouldn't touch the Monks, wear a hat, or dress inappropriately. Other than that, you're welcome to visit any of the Wats in town. The Monks are always glad to have foreigners visit, and will, if possible, try to speak English with you. A great place for photographs also.