Holly Oaks was the home of Dr. Robert Emmett Bledsoe Baylor (1791-1873), where he resided for many years and where he died, and Baylor University bears his name.