On the brow of the hill overlooking the meadow stood the Old Stone Fort, or Stoughton House, built in two portions, probably an older stone section and a wooden section. At the north end was a door of heavy oak timbers studded with iron spikes that bore the hacks and cuts of Indian tomahawks. The house was torn down in 1809.