Brewster’s Tavern was the site of the government seat and the town’s social center, and George Washington often visited it from 1765 to 1799.