The Osterby House was a large 18th century house that served as a boardinghouse for brickyard workers and was the site of illegal liquor sales, knifefights, threats, and a deadly fire. The current house was built in 1900.