I went through Stanford in the 1980s, just as it was becoming clear that logic-based AI had hit a wall. That was the beginning of the "AI Winter", which lasted about 15 years. Then came machine learning.
AI used to be a tiny field. In the heyday of McCarthy and Minsky,
almost everything was at MIT, Stanford, and CMU, and the groups weren't that big. There were probably less than 100 people doing anything interesting. Also, the total compute power available to the Stanford AI lab in 1982 was about 5 MIPS.
Part of what makes machine learning go is sheer compute power. Training a neural net is an incredibly inefficient process. Many of the basic algorithms date from the 1980s or earlier, but nobody could hammer on them hard enough until recently. Back in the 1980s, John Koza's group at Stanford was trying to build a cluster out of a big pile of desktop PCs. Stanford got a used NCube Hypercube with 64 processors (1 MIPS, 128KB each). The NCube turned out to be useless. There was a suspicion that with a few more orders of magnitude in crunch power, something might work, but with the failure of AI, nobody was going to throw money at the problem.
At last, there are profitable AI applications, and thus the field is huge. Progress is much faster now, just because there's more effort going in. But understanding of why neural nets work is still poor. Things are getting better; the trick of using an image recognition neural net to generate canonical images of what it recognizes finally provided a tool to get some insight into what was going on. At last there was a debug tool for neural nets. Early attempts in that direction determined that the recognizer for "school bus" was recognizing "yellow with black stripe", and that some totally bogus images of noise would be mis-recognized. Now there are somewhat ad-hoc techniques for dealing with that class of problems.
The next big issue is to develop something that has more of an overview than a neural net, but isn't as structured as classic predicate-calculus AI. One school tries to do this by working with natural language; Gary Markus, the author of the parent article, is from that group. There's a long tradition in this area, and it has ties to semantics and classical philosophy.
The Google self-driving car people are working on higher-level understanding out of necessity. They need to not just recognize other road users, but infer their intent and predict their behavior. They need "common sense" at an animal level. This may be more important than language. Most of the mammals have that level of common sense, enough to deal with their environment, and they do it without much language. It makes sense to get that problem solved before dealing with language. At last, there's a "killer app" for this technology and big money is being spent solving it.
There seems to be a mismatch indeed in incorporating common sense aspects and the proposed symbolic methods to solve them. Common sense seems "fuzzy". It doesn't look like rules and symbols are rich enough to cover its essence.
On the other hand, context has to do with a slot-like thinking architecture. If I wake up in the weekend in the same bed, I'm acting differently than during the week. Moreover I am easily think of a scenario in which I wake up in my bed on a raft on the ocean.
I'm not aware of any connectionist framework that comes close. 1) Polychronization doesn't tell how to connect groups. 2) Global workspace theory is "homogeneous" connectionist. As is Copycat. 3) Alternatives such as ARTMAP are not grammar like either. 4) Bayesian methods or dynamic field theories care even less about representation.
We really have a lot to figure out in the coming decennia! :-)
> it was becoming clear that logic-based AI had hit a wall... Then came machine learning.
In that context, this advance is a sign that there's still momentum going in the current ML paradigm - the hardest board game was at last solved! But as many have pointed out, there's a time in the near future where this approach too will stall short of ambitions. Will this lead to another winter?
For the algorithm ideologues probably yes. The key to current interest was that NN do vision extremely well, which can be built into image search, etc, which can be commercialized. So I think you are right that R&D for self-driving cars will be the sustaining investment in this field for the future, and this will drag AI research towards "animal common sense" as a goal. If they build a hell of safe car and the people buy into it, there could a Cambrian explosion in AI. But its failure or rejection could break the field too.
I finished a MSCS in 1985. Ed Feigenbaum was still influential then, but it was getting embarrassing. He'd been claiming that expert systems would yield strong AI Real Soon Now. He wrote a book, "The Fifth Generation", [1] which is a call to battle to win in AI. Against Japan, which at the time had a heavily funded effort to develop "fifth generation computers" that would run Prolog. (Anybody remember Prolog? Turbo Prolog?) He'd testified before Congress that the "US would become an agrarian nation" if Congress didn't fund a big AI lab headed by him.
I'd already been doing programming proof of correctness work (I went to Stanford grad school from industry, not right out of college), and so I was already using theorem provers and aware of what you could and couldn't do with inference engines. Some of the Stanford courses were just bad philosophy. (One exam question: "Does a rock have intentions?")
"Expert systems" turned out to just be another way of programming, and not a widely useful one. Today, we'd call it a domain-specific programming language. It's useful for some problems like troubleshooting and how-to guides, but you're mostly just encoding a flowchart. You get out what some human put in, no more.
One idea at the time was that if enough effort went into describing the real world in rules, AI would somehow emerge. The Cyc project[3] started to do this in 1984, struggling to encode common sense in predicate calculus. They're still at it, at some low level of effort.[4] They tried to make it relevant to the "semantic web", but that didn't seem to result in much.
Stanford at one time offered a 5-year "Knowledge Engineering" degree. This was to train people for the coming boom in expert systems, which would need people with both CS and psychology training. They would watch and learn how experts did things, as psychology researchers do, then manually codify that information into rules.[2] I wonder what happened to those people.
AI used to be a tiny field. In the heyday of McCarthy and Minsky, almost everything was at MIT, Stanford, and CMU, and the groups weren't that big. There were probably less than 100 people doing anything interesting. Also, the total compute power available to the Stanford AI lab in 1982 was about 5 MIPS.
Part of what makes machine learning go is sheer compute power. Training a neural net is an incredibly inefficient process. Many of the basic algorithms date from the 1980s or earlier, but nobody could hammer on them hard enough until recently. Back in the 1980s, John Koza's group at Stanford was trying to build a cluster out of a big pile of desktop PCs. Stanford got a used NCube Hypercube with 64 processors (1 MIPS, 128KB each). The NCube turned out to be useless. There was a suspicion that with a few more orders of magnitude in crunch power, something might work, but with the failure of AI, nobody was going to throw money at the problem.
At last, there are profitable AI applications, and thus the field is huge. Progress is much faster now, just because there's more effort going in. But understanding of why neural nets work is still poor. Things are getting better; the trick of using an image recognition neural net to generate canonical images of what it recognizes finally provided a tool to get some insight into what was going on. At last there was a debug tool for neural nets. Early attempts in that direction determined that the recognizer for "school bus" was recognizing "yellow with black stripe", and that some totally bogus images of noise would be mis-recognized. Now there are somewhat ad-hoc techniques for dealing with that class of problems.
The next big issue is to develop something that has more of an overview than a neural net, but isn't as structured as classic predicate-calculus AI. One school tries to do this by working with natural language; Gary Markus, the author of the parent article, is from that group. There's a long tradition in this area, and it has ties to semantics and classical philosophy.
The Google self-driving car people are working on higher-level understanding out of necessity. They need to not just recognize other road users, but infer their intent and predict their behavior. They need "common sense" at an animal level. This may be more important than language. Most of the mammals have that level of common sense, enough to deal with their environment, and they do it without much language. It makes sense to get that problem solved before dealing with language. At last, there's a "killer app" for this technology and big money is being spent solving it.