Old Parsonage was the house of Shirley's first minister, Phinehas Whitney. Dorothy Quincy and Madam Lydia Hancock visited here, and in 1773 John Hancock came here to accompany them home. Moved to this site in 1906, it became the Rectory of St. Anthony's Church.