Lisp will hardly ever make it into the corporate hierarchies, where non-coding bosses want you to write in the dumbest possible language. And unfortunately without conquering this market a true boom is impossible.
Perhaps for this reason Lisp became an individualist, niche language.
Instead of "dumbest possible", I'd say "most constrained".
And, just to play devil's advocate: There may actually be some benefits to coding in a constrained, regimented manner. Sure, you need to write more code to do the same thing, so an individual programmer isn't able to tackle as large a project as he would be able to in a less-constrained language. But a more constrained language is more amenable to automated analysis and may make it easier for people to work together or maintain one another's code: Lisp may allow an individual programmer to express his thoughts in a manner that is clearest to him, but it might not be as clear to others. But if you use a language that forces everyone to think the same way, then you end up with a less-concise program that perhaps more people can understand.
And, for the vast majority of projects, a highly constrained language is all you need. For most workplaces, even a language like C++ is too powerful; they will add a host of self-imposed constraints like "RAII", "no global variables" or "don't overload operators", and they may use lint-like tools to enforce these constraints.
...Or, so the argument goes. Whether or not it bears weight depends on the relative value of having many people on a project versus having a few people who are more productive... or whether it is easier for other programmers to understand more concise code versus more constrained code. I'd argue that the lack of Lisp companies dominating their competitors favors constrained code, but there is a circularity in that argument that I don't like.
Agreed, and looks like the competition between bigger groups of replaceable people vs. smaller groups of brighter people will keep going in this industry for decades, until something changes from the roots, if at all.
Granted, DSLs are easily expressible in Lisp, but pretty much anything is expressible in Lisp. If you want some sort of guarantee that your DSL is limited in some way, you would need to write additional Lisp code to enforce those constraints. But if the constraints map nicely to the constraints in a more limited language, you get the constraints effectively for free by choosing the limited language, while trading off the power of Lisp.
The prevalence of DSLs in Lisp code poses an interesting counterpoint to Greenspun's Tenth Rule: Any sufficiently complicated Lisp program contains an ad-hoc, informally specified subset of a declarative language. :)
If and only if the constraints map nicely. Otherwise you're out of luck.
DrScheme's language levels, where constrained languages help beginners avoid the strange error messages one gets by accidentally invoking an advanced language feature, are unique, as far as I know. Correct me if a learning tool for any non-Lisp language has succeeded in implementing the same sort of thing. Otherwise, I think it's obvious that if you have specific desired constraints for a limited language, some kind of Lisp is your best bet.
My impression is that non-coding bosses don't care that much, or, they care enough to recommend what they read in trade journals. Solution: Talk up Lisp (etc) in trade journals. That java is a standard because it is "dumb" is a myth. Not one CTO in a million went through a catalog of languages and said, "Nope, this has closures. Nope, this has eval()..." Otherwise, they would have come to, "Nope, this has garbage collection... We're sticking with COBOL."
pg said it best "The pointy-haired boss miraculously combines two qualities that are common by themselves, but rarely seen together: (a) he knows nothing whatsoever about technology, and (b) he has very strong opinions about it."
From my experience in different companies, big and small, decisions are made based on the number of factors:
* Trade journals, as you said, which appeal solely to the corporate bosses; although I'd give this a lower importance among others.
* Wages and availability of programmers in the job market.
* A chance for the boss to understand the language and/or to talk about architecture on the meetings easily, if needed. This one is interesting, as, I think, that's the main reason Java is popular. Smarter bosses can read some semi-technical journals, where "patters", "methodologies" and "common practices" are explained well for them -- just for them.
So, Java clearly wins in all 3 points, while Lisp loses in all 3.
Perhaps for this reason Lisp became an individualist, niche language.