The Point House was built in 1804 by John Payne, Jr. His son Charles W. Payne, master of the ship "Fanny," died at sea in 1838, and the Broken Mast Monument in Oakland Cemetery bears his name.