I guess. A lot of people already did.

http://lukemuehlhauser.com/if-youre-an-ai-safety-lurker-now-...

No. It simply does not outside of the realm of science fiction. Understanding HGI will help us understand enough about ourselves and our motivations and ethics that it won't be an issue.

Humans augmented by machine learning systems, on the other hand, are here now and a wholly different question. Institutions empowered by AI research have a power / person ratio unlike any of those of the past.

If you haven't read Bostrom's book yet, I'd really recommend it. http://www.amazon.com/Superintelligence-Dangers-Strategies-N...

It's about as likely as a meteor hitting the planet and wiping out the entire human life. Possible? Sure. Should I be panicking about this right now? Nah.

The book is nonsense. I'll start paying attention when someone who has real experience in the field of AI research (and I'm not talking charlatans like Yudkowsky here), but someone like say Norvig come out and say it's a reasonable concern today.

Many computer science professors have publicly said they think AI poses significant risks. There's a list here: http://slatestarcodex.com/2015/05/22/ai-researchers-on-ai-ri...

Also see this open letter, signed by hundreds of experts: http://futureoflife.org/ai-open-letter/

Some experts in the second link you gave are concerned, sure. But I can probably find an equal number who dismiss it as well. There isn't a clear consensus over AGI. I still remain skeptical. Same with your first link, which tries to "forecast" AGI. People can't forecast next month's weather correctly, so forgive me for not believing in a 10% chance in 10 years.

Actually, I take back my appeal to authority argument in its entirety, because I just remembered the first thing I saw in my AI class was a video of experts claiming the exact same thing. The video was from the 50s.

EDIT: Found it: https://www.youtube.com/watch?v=rlBjhD1oGQg

Also, no-one is worried about a skynet scenario. The worrying scenario is just any powerful system that optimises for something that's different from what humans want.

Second, the point is that even uncertainty is enough for action. For AI to not be a problem, you'd need to be very confident that it'll occur a long way in the future, and that there's nothing we can do until it's closer. As you've said, we don't have confidence in the timeline. We have large uncertainty. And that's more reason for action, especially research.

Consider analogously:

"We've got no idea what the chance of run-away climate change is, so we shouldn't do anything about it."

Seems like a bad argument to me.

I'm 100% with Torvalds on this when he laughs at the prepostorous notion that AI will become a doomsday scenario. I think it'll become more and more specialised, branch out to other fields and become reasonably good. But there's a huge leap to go from there to HGI.

> any powerful system that optimises for something that's different from what humans want.

Except this notion rests on the premise that humans will not be in full control, which leads to the exponential growth argument which leads back to the Skynet like scenario.

> the point is that even uncertainty is enough for action

And that action is...what exactly? People won't stop building intelligent systems. There is no real path that we have from where we are to HGI, so it's not like researchers have a concrete path. Just what exactly does this research look like?

> "We've got no idea what the chance of run-away climate change is, so we shouldn't do anything about it."

Extremely Poor analogy. We have decades worth of concrete data that tells us the nature and reality of climate change. We demonstrably know that it's a threat. Can you say the same about AI?

Also, I'll take the time to re-iterate how heavily skeptical I remain of groups like MIRI that are spearheaded by people who don't believe in the scientific method, believe in stuff like cryogenics, have history of trying to profit off of someone else's copyrighted material and have someone managed to convince a whole lot of people that donating them is the best way to fight off the AI doomsday scenario. People should do their research before linking to stuff like that. :(

In general, if there's a poorly understood but potentially very bad risk, then (a) more research to understand the risk is really high priority (b) if that research doesn't rule out the really bad scenario, we should try to do something to prevent it.

With AI, unfortunately waiting until the evidence that it's harmful is well established is not possible, because then it could be too late.

What AI risk research could involve is laid out in detail in the link.

I would guess A. the development of "known safe" high-level primitives, and B. a coupling of education in the craft of AI with education in the engineering of AI.

The "profession" of developing strong, general-purpose AI should basically look like a cross between cryptography and regular old capital-e-Engineering: like crypto, you'd constantly hear important things about "not rolling your own" low-level AI algorithms; and, like Engineering, the point of the job would be making sure the thing you're building is constructed so that it doesn't "exceed tolerances."

The research goals in the field of "friendly AI" are thus twofold:

• work in understanding possible computational models for minds, to understand what sort of tolerances there can be—what knobs and dials and initial conditions each type of mind comes with;

• and work in development of safe high-level primitives for constructing minds.

Both of these are the subjects of active papers. Note that these can both also be described as plain-old "AI research"; they're not just abstract philosophy, these papers are steps toward something people can build. The research within the subfield of "friendly" AI just has different criteria for what makes for a "promising avenue of research", e.g. ignoring black-box solutions because there are no knobs for humans to tweak.

> MIRI ... spearheaded by

MIRI is two things, and it's best to keep them mentally separate. It's a research organization, like Bell Labs—and it's a nonprofit foundation that funnels money into that research organization.

Yudkowsky is the director (head cheerleader) of the nonprofit. He doesn't really touch the research organization. The research org will succeed or fail on its merits (mostly whether it hires good researchers), but the leadership of the nonprofit has not-much to do with that success or failure, any more than AT&T had any impact on the success or failure of Bell Labs. You can believe in the research org known as MIRI even if you actively distrust Yudkowsky.

That's quite a strong claim - could you provide evidence for it? Are you referring to HPMOR? HPMOR has always been given away for free, both online and in print, and J.K. Rowling has explicitly allowed non-commercial Harry Potter fanfics.

The Singularity Institute for Artificial Intelligence, the nonprofit I work at, is currently running a Summer Challenge to tide us over until the Singularity Summit in October (Oct 15-16 in New York, ticket prices go up by $100 after September starts). The Summer Challenge grant will double up to $125,000 in donations, ends at the end of August, and is currently up to only $39,000 which is somewhat worrying. I hadn't meant to do anything like this, but:

I will release completed chapters at a pace of one every 6 days, or one every 5 days after the SIAI's Summer Challenge reaches $50,000, or one every 4 days after the Summer Challenge reaches $75,000, or one every 3 days if the Summer Challenge is completed. Remember, the Summer Challenge has until the end of August, after that the pace will be set. (Just some slight encouragement for donors reading this fic to get around to donating sooner rather than later.) A link to the Challenge and the Summit can be found in the profile page, or Google "summer singularity challenge" and "Singularity Summit" respectively.

Like AI based-HFT? Which humans want what? I'm pretty sure, that if there's one constant, that there always will be some humans on either side of the argument.

So Transcendence in leu of The Terminator. It's still hollywood fiction.

My point is that the entire idea of an buggy AI destroying humanity is absolutely divorced from the reality of AI research. It's not unimaginable to the lay-person, but there is simply no way of getting there from the neural networks we're making now.

To begin, can you go ahead and explain what self-awareness means in terms of a neural network you are training?

Here's a comparison: wouldn't it be great if we had started thinking about climate change way back at the beginning of the industrial revolution, before we decided to create tons of open-air coal plants?

There are reasons to solve problems before they're problems.

So is it worth exploring? Sure. But I'm not going to be concerned about something for which there is zero evidence to support it.

There are no reasons to solve problems before there is foolproof evidence that they are indeed problems.

- The horse population declined dramatically after their work abilities were exceeded on every dimension by machines

- Homo Neanderthalensis disappeared after the appearance of Homo Sapiens.

I'm not saying these are perfect analogies. But you are saying they're completely irrelevant. Why is that?

At the very least, that future is too complicated to form such a confident opinion that the risks of machine learning are in the realm of science fiction.

I think it's important to separate those real developments from the movie-plot-threat of a super-AI escaping human control.

Why?

But humans did make artificial flying machines.

I don't mean to be unkind, but your argument doesn't have close to the level of certainly that we should bet our species on.

No one knows where the roadblocks to AGI are, or what the timelines might be; but there is what seems like huge progress happening recently in an area which might eventually lead there. While no one knows the path, many intelligent people have thought about this without finding any theoretical roadblocks. Please don't publicly dismiss the concerns as 'science fiction' without a little more thought.

You're not getting it.

The effective argument of the danger seer is akin to "humans will learn to fly but then our cities will be covered with feathers" -

Humans have incentive to produce flexible, learning, language-understanding machines. Humans don't an incentive to produce a thing that will suddenly go rogue and decide to kill us.

It is entirely reasonable to argue that the tendency of humans themselves to go rogue and decide kill other humans comes from the evolutionary processes that produced humans rather than from the quality of human intelligence. We can see other relatively intelligent animals that live with in more (and in less) harmony than humans. No reason intelligence implies entity X won't fully cooperate with the people who intentionally engineered it.

I would argue that we human imagine that AGIs would have the same "downsides" as ourselves simply because we don't have any conception of intelligent things besides ourselves.

Maybe a carefully human-engineered intelligence would for reasons unknown turn into the things we see on movies. Maybe the next generation of supercolider will create an subatomic chain reaction that destroys the earth. But it's premature to worry about these particular hypothetical, especially given humans with ordinary AIs do pose problems.

So: Yes, we might all get along fine; ideally the more intelligent entity would be even more enlightened that we are; etc - those are all valid possibilities, and no one is saying they are implausible.

All people are saying is that some bad possibilities are also plausible (Eliezer Yudkowsky writes some great stuff about all the accidental ways we could screw up trying to program an AGI to be benevolent), and that, as the downside of the bad possibilities is potentially so great, then, in the absence of any argument why they definitely won't happen, we need to think carefully about them.

Some would say that, because the potential consequences of a bad outcome are so great, we need to be able to prove the bad outcomes won't happen before we build this.

In my opinion, Eliezer Yudkowsky and those who follow his argument are not intellectually credible and should dismissed with zero consideration.

In looking at this issue, I would note that Yudkowsky's (Bostrone's) ultimate argument is exactly homologous to Pascal's Wager. [1]

IE, since there's a hypothetical small probability that some entity [A Future AGI/Christian God/Flying Spaghetti Monster] could exist and the stakes for ignoring the possibility are "infinite" then we must "rationally" labor now to deal with possibility regardless of not understanding the mechanism involved in that entity.

And the thing is that modern science and modern statistics pretty hinge on ignoring, uh, "guff", demanding that extraordinary claims submit their extraordinary evidence and so-forth.

Further, the other fallacious argument pushed by Yudkowsky et al involves treating an AGI as a wish-granting Genie - human make overt logical requests and the AGIs "twist" these in a world-changing/destroying fashion. The problem is the over-request-format is identical to "logical specification" approach of gofai, the generally carded view that AI can be achieved with only the symbolic specification of how the world works (notably Yudkowsky and friend simultaneously acknowledge AI won't be based on symbolic specification/gofai and ground their argument on AGIs operating on explicit orders only, which boil down to symbolic specification. The Bostrone example is you tell the Genie "make me a thousand paper clips" and it turns the planet into paper clips "just to be sure").

[1] https://en.wikipedia.org/wiki/Pascal's_Wager

[2]https://en.wikipedia.org/wiki/Wish

[3]

I myself am on fence, on the one hand it is a bit like worrying about over population on mars, and there really is nothing we can productively do right now to research AI safety, the best thing we have are convolutional long/short term memory nets, and there's not much you can say about it.

But at the same time worrying is a great move if the alternative is to just wait until we can do something.

We haven't unintentionally produced a program that had a whole range of unexpected behaviors. An intelligence having human-like "survival instincts" and etc would be a big variation on a tool.

>In our journey to achieving HGI, we must first achieve an understanding of HGI ethics, motivations, and behavior.

You mean that it's impossible to make a human level general intelligence ('HGI', to use your term), without understanding those things?

If so, I disagree with your assertion: it is not obvious that those things have to be done before a human level AGI could be made.

It might be a good idea to do so, but its not clear that they are a necessary requirement. Some people would say this is why AGI is dangerous.

Maybe there's a path to bootstrapping an AGI, where we build something from relatively simple bits we can understand, that then becomes very much more complex than we can understand.

This happens all the time: for example, we get very complex classification behaviour from deep nets: [simple pieces] + [lots of data] doing something more complex than we have explicitly trained them to.

If we were convinced that "In our journey to achieving HGI, we must first achieve an understanding of HGI ethics, motivations, and behavior" was a necessary thing to build before we could build a HGI, people would be less worried. But its not; and that's partially why people are worried. In fact, some of the people working on AI risk could be said to be basically racing to understand these things before other folk build an AGI.

It is entirely logical to argue that a high level intelligence could not be constructed without a broad and deep understanding of it's motivations and behaviors.

That may or may not be ethics as such but if something we make on purpose, we'd need to understand it. And considering it's difficulties, making it on purpose seems necessary, contrary to movies etc.

Instead, andreyf relied on it as a fact. I pointed out we can't assume that.

As such: its reasonable to fear AGI coming before we have that stuff sorted out.

Hence, as such: there is a real risk here, which the safety folks are trying to mitigate; hence they aren't worried about a threat scenario easily dismissable by andreyf's argument.

>It is entirely logical to argue that a high level intelligence could not be constructed without a broad and deep understanding of it's motivations and behaviors.

Ie. If you, or anyone had convincingly argued this, then it would be reasonable to dismiss (or at least greatly reduce) the concerns of the safety folk. But no one has, so its not reasonable to.

My claim is that I have seen no evidence that it is reasonable to fear a fictional self-aware autonomous AI (what I call HGI). The evidence people point to seems to have a very tenuous relationship with the reality of AI research.

Once we have even an inkling of an idea of how one might approach creating an artificial HGI, we can discuss how we can prevent it from being evil and taking over the world. Until there's actually a proposal for how one might construct one, though, any such discussion is worse than useless.

How can you talk about the risk of something when you literally have no idea what it is you are describing or how it's built, embodied with magical properties imagined by science fiction authors? To me, this makes about as much sense as talking about stopping aliens which could very well exist and could very well wipe out our civilization, i.e. great subject for wrapping up a party or going to the movies, but not exactly material for peer reviewed journals.

Feral: Yes - but no one provided any argument for that.

Me: I should rephrase, I sort-of do above but still, negation of "it requires understanding" is "AGI could happen without understanding, at random, in some fashion".

And I'd say that is the position that's no has provided to support for - again, akin to the worries that bring a giant supercollider online could destroy the earth. I mean, given that an AGI is an unknown, maybe bringing the supercollider online, another unknown, could create an AGI, which would then destroy the earth!

Plus, human beings have generally not succeed in producing complex engineering achievements by accidents. The atomic bomb required a massive engineering effort, heavier-than-air flight required considerable effort, alchemists never synthesized CPUs by baking minerals at random, even largest pieces of modern software have not threatened to "achieve consciousness" even as they malfunction repeatedly.

My argument is "consciousness by accident" is the extraordinary claim which requires extraordinary evidence. And AndreyF also adds - the dangers of ill-intentioned humans using AIs is here today, why worry about the hypothetical "getting out of control" problem when you humans who have historically inflicted vast amounts of misery on others of their species - oh, and consider several madmen fighting each other, that couldn't happen, not in Syria or whatever place one might name.

I'd be interested in your take on it.

More specifically, there are an infinite number of reasons for and against doing anything. Once we understand how humans weigh these reasons and choose between them and model them in an HGI system, we will be able to give it the values and morals that its creators choose.

1. https://news.ycombinator.com/item?id=10724108

"1. The utility function may not be perfectly aligned with the values of the human race, which are (at best) very difficult to pin down.

"2. Any sufficiently capable intelligent system will prefer to ensure its own continued existence and to acquire physical and computational resources – not for their own sake, but to succeed in its assigned task."

The first of those is what Bostrom calls "perverse instantiation" and Dietterich and Horvitz call the "Sorcerer's Apprentice" problem (http://cacm.acm.org/magazines/2015/10/192386-rise-of-concern...). The second of these is what Bostrom calls "convergent instrumental goals" and Omohundro calls "basic AI drives."

The first of these seems like a fairly obvious problem, if we think AI systems will ever be trusted with making important decisions. Human goals are complicated, and even a superintelligent system that can easily learn about our goals won't necessarily acquire the goals thereby. So solving the AI problem doesn't get us a solution to the goal specification problem for free.

The second of these also has some intuitive force; https://intelligence.org/?p=12234 shows Omohundro's idea can be stated formally, so it's not purely sci-fi. Averting the "Sorcerer's Apprentice" problem in full generality would mean averting this problem, since we'd then simply be able to give AI systems the right goals and let them go wild. Absent that, if AI systems become much more cognitively capable than humans, we'll probably need to actively work on some approach that violates Omohundro's assumptions (and the assumptions of the formalism above). Bostrom and MIRI both talk about a lot of interesting ideas along these lines.

The first problem is not new. We have a similar problem with some corporations, for example.

"A sufficiently capable intelligent system" is as real as "sufficiently hostile aliens". It's hard to argue and reason about a fictional system with a assortment of properties picked by someone aiming to spreading fear.

I would say that the central concern is with notional systems that can form detailed, accurate models of the world and efficiently search through the space of policies that can be expected to produce a given outcome according to the model. This can be a recommender system that tells other agents what policies to adopt, or it can execute the policies itself.

If the search process through policies is sufficiently counter-intuitive and opaque to operator inspection, the "Sorcerer's Apprentice" problem becomes much more severe than it is in ordinary software. As the system becomes more capable, it can look increasingly safe and useful in its current context and yet remain brittle in the face of changes to itself and its environment. This is also where convergent instrumental goals become more concerning, because systems with imperfectly understood/designed policy selection criteria (introducing an element of randomness, from our perspective) seem likely to converge on adversarial policies due to the general fact of resource limitations.

There's no reason to think this kind of system is inevitable, but it's worth investigating how likely we are to be able to develop superhuman planning/decision agents, on what timescale, and whether there are any actions we could take in advance to make it possible to use such systems safely. At this point not enough research-hours have gone into this topic to justify any strong conclusions about whether we can (or can't) make much progress today.

http://givewell.org/labs/causes/ai-risk gives a good summary of this topic.

Most people that worry about this topic don't worry about human level AI, it is the vastly greater than human level AI that is the concern.

There are some caveats here. The cost of AI labor needs to be below that of the human minimum. Given there is no effective limit to how much AI labor you can create, the likely result is the cost of capital will fall to near zero making AI labor near free. No human will be able to compete against this except in providing personal services to the owners of capital in fluff jobs.

The more interesting thing is that with the cost of capital and labour at near zero we should get massive deflation to the point that it will be very cheap to provide all people with a very high basic income that will make the need for employment unnecessary. This would be a great world to live in, but it would have a profound effect on the society unlike anything since the rise of agriculture.

I don't see how that follows. How does a computer program do manual labor?

Actually most manual labor is done by machines controlled to a greater or less extent by a human brain. AI just removes the human brain out of the production loop.

I'm not sure I really understand the economics either.

Is AI going to be free? Who are the owners of capital going to sell their production to if people have no income? What are people who don't own machines going to do? Curl up in a ball and let themselves starve to death?

The best way to think about production is that everything produced is a combination of four things: labor, capital (past production), energy, and matter (land and raw materials). If you have labor that can be used to build itself (AI) then you can expand labour and capital to a very large number. You can also build new sources of energy very cheaply so the only limiting resource is matter. If you have the energy, capital and labor you can recycle most raw materials very cheaply. This leaves land as the only factor in production that does not scale.

What would all this mean in practice. Physical objects and services would become very cheap to make (effectively free) and all wealth would be tied to the ownership of land. Given the state owns most land, it should be very cheap for the state to provide all the material objects and services that any citizen needs. Unless the state actively wants to stave to death its citizens then there is not need for anyone to go hungry - if the state does want to stave to death people it does not need AI.

If you have links to more detailed ressources on that topic, I'd be interested in reading them.

Once you have one human level AI you can make more replacing each human in the production of the human level AI - you basically start at the end point and working backwards replacing each job performed by a human with an AI. Once you get to the final job you will have a production system that can build AIs without any human labor. The system can make as many extra AI’s as wanted without any human labor - it is detached from the human labor market.

Unfortunately I don’t have any links discussing this. I think not too many people dwell on human level AI’s because the expectation is with AI’s if they can get to human level they will not stay at human level. Once you have AI’s that can make AI’s then they can make better AI’s until they far surpass human level. Understanding what these AI’s will be like is impossible for us.

Again, I don't see how that follows. Once you know how to build something, you can do it again. Of course. But what if building a functioning AI that can replace a human costs $100 bn?

Certainly it's not going to be economically viable to replace tasks that do not produce high ROI, even if you assume AIs are paired with indestructible robot bodies made out of unobtainium that are as versatile as the human body (have we even built one of those ever?).

I'm familiar with the idea that AIs could make better AIs and lead to some intelligence singularity, etc. Again, I think there's a lot of handwaving behind those ideas. We have no idea what the complexity of building more complex intelligence is, or what it even means.

* I don't see how "Understanding HGI" prevents a "human-level artificial intelligence" from doing anything it pleases.

I've expounded on the denial: we will need to first figure out what motivates humans before we can replicate their levels of intelligence. So in order to solve HGI, we will need to understand HGI, which means we will be able to control HGI. We will understand personalities and values in a systematic way, and we will be able to choose the personalities and values of any one instance of HGI.

I would not assume this with certainty.

There are many systems in nature that consist of simple pieces, when composed, yield enormous complexity. Replicating intelligences might be like black boxes which we don't understand but can generate, or grow, if you will, from simpler primitives. We might get lucky and discover the means to generate intelligences without understanding them.

As an example, "Machine learning works spectacularly well, but mathematicians aren’t quite sure why." https://www.quantamagazine.org/20151203-big-datas-mathematic...

On that premise, the risk of accidentally discovering the right bit-string that morphs into a all powerful evil AI seems to be minuscule. Since at least human intelligence consists of multiple dimensions/factors and only some combinations thereof can be described as "intelligent", given a certain environment. Accidentally discovering something "thinkable": maybe, brute forcing the right combination for human-surpassing intelligence? Most likely not.

Biology doesn't have any need for "the right bit-string" (modulo DNA, different purpose) to make intelligence work, and one shouldn't have such an expectation of an emergent strong AI, either. Instead of "the right bit-string", think more "the right model"

I agree with the problem being more one of finding the 'right model', as I'd hope that this will reduce the parameter space greatly. That's why I agree with gggp's post that we'd need to know more about human intelligence, thus also human motivation.

There will have to be more true breakthroughs to get to AI, yes; I do agree though that it will take a "long" time, however before this happens.

That title's wording is clickbait, not an argument. Mathematicians know exactly why neural networks work – because they contain a model of the data, and they try to develop ways to build (train) those models so that they do it better. The model may not have human concepts associated with it, but that doesn't mean it's somehow "out of control".

In order to achieve HGI, one would have to train models of quite a few domains (the same way schools create models in people's minds), as well as tons of ethical and practical heuristics to weigh an infinite number of reasons for or against doing something.

I strongly doubt that.

>As an example, "Machine learning works spectacularly well, but mathematicians aren’t quite sure why." https://www.quantamagazine.org/20151203-big-datas-mathematic....

If you read the HN comments on that very article, I recall posting some links to papers giving theoretical grounding for why deep learning works so well.

If you could prove that, you'd put a lot of minds at ease. By "prove" I mean not necessarily a full mathematical proof, but certainly more than a few paragraphs of English text.

I'm skeptical myself, seems like the more we learn about biology the more it is doing that we did not realize, but skepticism is at best the first step to proof.

The fear and idea of somehow creating an HGI without understanding how to control it came from The Terminator. It's pure fantasy.

You're failing to sell this argument to someone generally inclined to agree with you about the outcome.

However, until one proposes a concrete methodology for modeling the world which includes the method that is doing the modeling (self-awareness), a decision making process, and a way to take action autonomously and without intervention, it's very hard to talk generally about preventing catastrophic bugs.