Often there are beautiful minerals left behind at abandoned mines. The mine shown above is an old copper mine in New Mexico. The blue and green stone seen in the walls is chyrosoprase and malachite. The interior walls of the mine were a fantastic blue and green and there were large mineral samples strewn all around the area.
Going into an old mine often isn't safe or advisable but there are usually dump heaps near mines which may have good mineral samples. If you can find an old but safe mine that is open to the public the chance to be in the Earth, surrounded by rock, is a treat in itself.