During the War of 1812, an American fort stood on this hill under the command of Major John Jermain and was manned by the Fourth Regiment of the New York Artillery, protecting the port of Sag Harbor from foreign invaders. In the early morning hours of 11 July 1813, a contingent of British marines attempted a hostile occupation of the surrounding area. Cannon fire from the fort, together with American militia and the citizens of the village of Sag Harbor, drove them back and they never returned.