Stagwell was patented to Thomas Stagwell in 1649 and acquired by Richard Bennett in 1706, when he was one of the largest landowners in Maryland. His descendant Judge Richard Bennett Carmichael built the house about 1805. He later presided over the convention of 1867 for a new constitution for the State of Maryland.