The Wayside Inn was a Victorian Gothic structure built from 1869 to 1938 by Col. Benjamin C. Butler, and its famous guests included President U.S. Grant, the Astors, the Vanderbilts, and the Pullmans.