The original house was built in the late 1600s by Nathaniel Oakes. Sold to Northborough's first two ministers, Rev. John Martyn in 1747 and Rev. Peter Whitney in 1770, it became one of the first places for religious ceremonies. After a fire in 1780, Whitney rebuilt the house in its present form. Whitney, an author and forceful advocate of independence from England, died in 1816.