The cemetery began with a first burial in the late 1800s on land owned by the McMullen family and continued in use as a Black cemetery after the Coachman family bought the property in 1902. The Coachman family lost the land during the Great Depression, and in 1944 Alfred Ehle bought part of it from the State of Florida and subdivided it for residential development. In 1951 Ehle transferred the cemetery lot to the St. Paul Home Helping Hand Society of Safety Harbor, which deeded the site two years later to the Safety Harbor Colored Community. Although some lists show nearly 130 interments, only 20 markers are visible today, with 14 still readable, including 3 for U.S. veterans. The earliest legible marker is dated 1896, the next legible date is 1940, and the latest is 1971. A 2018 ground-penetrating radar survey of most of the site by the University of South Florida and Florida Public Archaeology Network confirmed about 100 potential grave sites. Descendants and relatives informally maintained the cemetery throughout the 20th century. In published obituaries, it was called either Safety Harbor Colored Cemetery or Helping Hands Cemetery. In 2017 volunteers organized to restore the site, gave the property the formal name Whispering Souls African American Cemetery, and in 2018 a non-profit formalized plans for its long-term care. On June 24, 2021, Whispering Souls African American Cemetery, Inc. secured clear legal title to the site.