Humans are however very bad at retrospectively telling which memes they were exposed to in the past and which not. That is basically the reason why highly specialized people in academia are often bad teachers if they didn't practice teaching a lot. The inferential distance between expert and novice (i.e. the difference in knowledge) simply becomes so large and complex, that the expert forgets what the actual gap is.
I do agree, but at ~14 years of age, in a small village, during the days when only the privileged had even 28k dial-up access, I'd had rather limited opportunity to be exposed to such things!
It would be enough for a friend having bragged about how they hallucinated about certain things. Words can convey very complex imagery like that and these experiences are encoded everywhere (in scripture, folk wisdom, idioms, songs, jokes, tales); in a web of distributed representations of the noosphere that we share via language. It is pretty much impossible to escape it and it seems conceivable that it might strongly bias our experiences under influence of hallucinogens because the mind is basically always concerned with interpreting inputs by the most suitable explanation (which will likely stem from stories and memories). Of course, we are only talking about a certain extent to which these experiences are determined by shared concept spaces, because there is no indication this is the only sensible hypothesis. The similarity of these experiences by different people may as well be explained by our shared cognitive architecture (e.g. modules for recognition of agency or venomous bugs) as other comments in this thread have suggested.