The house was built on land owned by William Alexander Stirling, a Brigadier General in the Continental Army, and was then purchased by Elias Boudinot. Boudinot was President of the Continental Congress, was elected to the first US Congress as a New Jersey representative after Constitution ratification, and served as Director of the US Mint. Although the house had many owners, it has traditionally been named after Elias Boudinot.