Named for the geological concept of a mineral vein, a vast region of a particular valuable mineral which may lie hidden inside solid rock until a miner or digger inadvertently punches through and discovers it.
A YouTube vein is a huge collection of similar and interesting videos on YouTube, which you had never been aware existed, and which you discovered through the "Related Videos" list on some other video. The Related Videos list on each video within the vein will similarly lead to a huge number of other videos inside the vein, so that once you enter the vein, you can be exploring it for hours via the Related Videos links.
It may be difficult to get out of the vein, because if it's large enough, all of the related videos on almost all of the videos within the vein will lead to more videos within the vein, in an inescapable closed circuit. Conversely, once out it can be difficult to get back in (without using the browser's Back button) because by definition, the videos that lead into the vein are obscure and don't come up often in searches and related-videos lists. It's only a vein if its entry points are in that way hard to come by or obscure, but once inside it proves to be enormous.
I looked at this Japanese music video that came up in a YouTube search and uncovered an enormous youtube vein of old japanese pop music in the related videos!