When would be a good time to start worrying? Let's say two hundred years from now, give or take a few decades. I imagine that by that time we might have made some progress on the subject of intelligence, what it is and how to reproduce it. At that point, it might be prudent to worry about an AI getting out of control.

We have no way of knowing what external indicators we might get. We have no idea what AGI would look like. That is often offered up as the very reason to "act now!" by Singularitarians, but if there is a threat that you don't know anything about, there is also nothing you can do about it. Unless you know what you're trying to stop, your actions are as good as random.

In this sort of discussion, you can replace "AGI" with "alien invasion". The threat from a Superintelligence is as serious and as predictable as that from an alien civilisation. We can prepare against AGI as much as we can prepare against an alien invasion. And we have exactly as much to fear from a hostile AGI as we have from a hostile alien species.

Ok, so what makes you say 200 years? And more importantly, what would make you change your mind about that number (in either direction)?

> [...] if there is a threat that you don't know anything about, there is also nothing you can do about it.

I disagree with this, at least somewhat. For one thing, even without knowing the specific threat, there's lots of stuff you can do which is just generally helpful - e.g., try to spread to other planets. Will it necessarily help? No. But there are a lot of threats it will protect against.

More importantly, people actually working on the problem of AI safety say they are doing things that appear, at least to them, to be useful. What makes you so sure they're wrong? From the little I know of their research, it certainly seems like stuff that will likely be pretty helpful.

Last point - considering just how big a problem AGI could be if they're right, just how many resources would you want to devote to it? Literally zero?

200 year is an entirely arbitrary period of time. The idea is that there's no way to predict when it will be possible to develop AGI but, if it happens at all, it will happen at some point in the future distant enough that nobody alive today will be around to say "I told you so".

You can ask how I know this. Obviously, I don't because I can't see into the future. But I can see the state of the art in the present and it's been baby steps for the last 70 years or so - and our capabilities have remained entirely primitive, industry hype nonwithstanding.

>> More importantly, people actually working on the problem of AI safety say they are doing things that appear, at least to them, to be useful. What makes you so sure they're wrong?

There's all sorts of opinions about how AI performance is accelerating ("exponentially"). In truth however, what has actually accelerated and in fact, plateaued in recent years is the performance in very specific tasks -object and speech recognition- and not in the general intelligence of AI systems. In fact, if you want to be more precise, we can only talk about advances in the context of very specific benchmarks, which is to say, specific datasets (like ImageNet) and according to specific metrics (say, F-score).

The problem is that all those benchmarks are arbitrarily chosen (iish; see below) and research teams spend a great deal of time tuning their systems to beat them. Which means, good performance in a benchmark tells us nothing about the extrinsic quality of a system: how it does in the real world, outside the lab and when the central assumption of PAC learning, that training and unseen data can be expected to have the same distribution (so that a system's training performance on the former predicts that on the latter) is not guaranteed. And then, performance in one type of task (e.g. classification) tells us nothing about the general capabilities of the system (i.e. general intelligence).

Singularitarians, like the people at MIRI (which your previous comment linked to) have focused on the performance of AI systems on modern benchmarks and the increase in compute, but modern benchmarks were essentially invented to allow some progress in machine learning, when mathematical results showed that progress was impossible. These previous results include Gold's famous result about learning in the limit (from the literature on Inductive Inference). The relaxation of assumptions was suggested in Leslie Valiant's paper "A theory of the learnable", which introduced PAC learning, the paradigm under which modern machine learning operates.

And to clarify my point above- modern machine learning benchmarks are not exactly chosen arbitrarily, rather they are justified by PAC learning assumptions about the learnability of propositional functions (and those, only; in fact, modern machine learning systems are propositional in nature, which severely restricts the expressive power of their models, making it much harder to realise the promise of Turing-complete learning of many of them).

... that probably got a bit too technical. My point is that just because we see imrpovement in performance today, in the field of research that we call machine learning, that doesn't mean that there is actual progress in the understanding of what intelligence is, or our ability to reproduce it. In a way, we might have changed our metrics, but we haven't necessarily improved our performance.

>> (...) try to spread to other planets.

So we have a science fiction problem and we're looking for science fiction solutions to it? :)

>> Last point - considering just how big a problem AGI could be if they're right, just how many resources would you want to devote to it? Literally zero?

Well, again that depends on what we can do about the problem, which in turn depends on what we know about it. I guess, like you say, if we know nothing about the problem, we can try random things like spreading to other planets or genetically enhancing the whole human race's intelligence until we ourselves are superintelligent and our risk to be taken over by an artificial superintelligence is 0.

But, having 0 certainty about the nature of AGI, we have exactly the same chances to avert the danger from it by sitting on our hands, as we have by migrating to other planets. I believe Bostrom's superintelligence scenario involves a machine that eventually colonises Mars with secret minining robots? If you're prepared to entertain the possibility of truly-Super intelligence, there's probably nothing you can do about it anyway.

To fully answer your last point. Sorry- this has grown extremelly long. I'm actually doing work right now, waiting for a long experiment to run so I got some time to burn :)

So, my pet formula for deciding whether taking a risk is worth it is to multiply the probability of some adverse event occurring, let's call this event X, by the cost associated with that event, let's call it Y; so R = p(X) * Y, where R the risk and you can then decide on some risk threshold, T, where if R < T you can justify taking the actions that you believe might lead to event X.

Now, when it comes to the Singularity (or, indeed, an Alien Invasion) we can accept the cost of the event to be infinite, Y = ∞, under the assumption that Superintelligence means game over for the species. But our knowledge of the event is nonexistent so the probability of the event, X = Singularity is... undefined. You can't plug that in my formula. You can't guess at the probability of X because nothing like X has ever happened before. You can try to extrapolate it from the development of human intelligence, but that took billions of years to evolve in a manner completely different than what we can reasonably expect for artificial intelligence (i.e. involving computers).

Basically, there's no way to calculate the risk from Superintelligence as long as we know nothing about it- 0 information means undefined risk. And therefore, no amount of resources can be justified to be spent to mitigate this risk.

In other words, I really couldn't answer your question- 0 resources, an infinite amount of resources, they're essentially the same.

Bottom line: to make decisions you need information even more than you need the resources to implement them.

Or in other words: you can't prepare for the unknown.

___________

OK, experiment's done cooking :)

Do you sincerely think that we are totally unable to predict when--or if ever--we might create general intelligence, something which stupid evolution has done through trial and error, but humans can attempt with striking brilliance and worldwide motivated research efforts? You can't make any rough prediction on whether we will be able to reach that point, or when it will happen?

>> Do you sincerely think that we are totally unable to predict when--or if ever--we might create general intelligence, something which stupid evolution has done through trial and error, but humans can attempt with striking brilliance and worldwide motivated research efforts?

Yes.

>> You can't make any rough prediction on whether we will be able to reach that point, or when it will happen?

No.

So you wrote a rather long post, I'll try to hone in on our disagreements. There are, I believe, two main disagreements:

1. I'm more confident than you in our ability to do something now and to predict things about AGI. Not much more confident, mind you, I just think the numbers are not literally 0, which makes most of your points above regarding amount of resources moot.

Basically, I think there are things we'll likely need to solve one way or another to figure out AI safety, and these are things we can work on. I also think that we can make some very rough estimates on timelines, and some very rough estimates on things we should do to mitigate.

2. I believe you misunderstand some of the points that people at MIRI make. You appear to classify them together with people like Ray Kurzweil, who appeals to exponential growth, etc, to make his arguments. The people at MIRI, afaict, don't really agree with this line of reasoning - in fact, I believe MIRI has all but ignored the progress on current AI for most of its existence (barring the last year or so), at least in terms of its research.

I mean, I agree that modern machine learning is not AGI or close to it. And I have no idea what AGI will look like - just a scaled up version of ML? A completely different take on things? The merge of a few different concepts together that will suddenly "tip over" to being capable? I have no idea. No one really does.

I am mindful, however, of a pretty long history of humans being terrible at predictions, in both directions - classic cases including people saying flight was impossible, a few years after the Wright brothers had already flown.

I'm not saying we're 2 years away from AGI - I'm saying we're not really sure, and even if we were 200 years away, I think it's worth spending some time and resources on trying to think about what we should do to prepare. The worldwide amount of resources spent on AGI and other existential risks are, what, 0.000001% of the amount of resources the world spends on sport? Does that really seem like a reasonable distribution to you?

>> (...) try to spread to other planets.

> So we have a science fiction problem and we're looking for science fiction solutions to it? :)

:)

It was just an example of something we can do right now. There are other examples if you want the other end of the spectrum - like stopping all technological decvelopment. I just don't think that will happen.

And let me point out that while AGI is a large threat IMO, lots of other things are large threats too - our technology is advancing on all fronts, and in many areas, we will soon be at a place where we can cause humanity to cease existing. E.g.:

1. Creation of weapons, much more powerful than atom bombs, that can effectively kill everyone.

2. Creation of super viruses that can destroy all humanity.

3. Creating the ability to "mind-upload" or similar, turning some people into much more powerful /faster thinkers/ whatever than others. It doesn't need to be an artificial super-intelligence to be dangerous.

And yes, these are all sci-fi scenarions, precisely because this is what's new in our world - scientific advancement! That's why the argument of "we've survived so far" is wrong - because the variable that's changing is the abilities that humanity has access to.