Port Townsend's one time Customs House is on the waterfront on Water Street, and is now home to The Lighouse Cafe, which serves breakfast and lunch. I have never eaten there, so no comment on the food, but it is a very interesting building which is in great shape for it's age. 
