A YouTube vein is a huge collection of similar and interesting videos on YouTube, which you had never been aware existed, and which you discovered through the "Related Videos" list on some other video. The Related Videos list on each video within the vein will similarly lead to a huge number of other videos inside the vein, so that once you enter the vein, you can be exploring it for hours via the Related Videos links.
It may be difficult to get out of the vein, because if it's large enough, all of the related videos on almost all of the videos within the vein will lead to more videos within the vein, in an inescapable closed circuit. Conversely, once out it can be difficult to get back in (without using the browser's Back button) because by definition, the videos that lead into the vein are obscure and don't come up often in searches and related-videos lists. It's only a vein if its entry points are in that way hard to come by or obscure, but once inside it proves to be enormous.