Building houses right on the edge of the water generally isn't wise. Coastal erosion is not your friend, and it will wear down any substance over time. 360 people once called Holland Island home, and back in the early 20th century it was a rapidly growing community.
Unfortunately, the effects of erosion were becoming an issue in the Chesapeake Bay as early as 1914. No amount of masonry and sea defense building did anything to fix the problem, and Holland Island had to be abandoned. By 1922 it was completely empty.