He sets up a definition where "real intelligence" requires consciousness, then argues AI lacks consciousness, therefore AI lacks real intelligence. This is somewhat circular.
The argument that consciousness can't be computable seems like a stretch as well.
Consciousness is not a result, it cannot be computed. It is a process, and we don't know how it interacts with computation. There are only two things I can really say about consciousness, and both are speculation: I think it isn't observable, and I think it is not a computation. For the first point, I can see no mechanism by which consciousness could affect the world so there is no way to observe it. For the second, imagine a man in a vast desert filled only with a grid of rocks that have two sides, a dark and light side and he has a small book which gives him instructions on how to flip these rocks. It seems unlikely that the rocks are sentient, yet certain configurations of rocks and books could produce the thought computation of the human mind. When does the sentience happen? If the man flips only a single rock according to those rules, would the computer be conscious? I doubt it. Does the consciousness exist between the flips of rock when he walks to the next stone? The idea that computation creates consciousness seems plainly untenable to me.
Indeed, I also think consciousness cannot be reduced to computation.
Here is one more thing to consider. All consciousness we can currently observe is embodied; all humans have a body and identity. We can interact with separate people corresponding to separate consciousnesses.
But if computation is producing consciousness, how is its identity determined? Is the identity of the consciousness based on the set of chips doing the computation? It is based on the algorithms used (i.e., running the same algorithm anywhere animates the same consciousness)?
In your example, if we say that consciousness somehow arises from the computation the man performs itself, then a question arises: what exactly is conscious in this situation? And what are the boundaries of that consciousness? Is the set of rocks as a whole? Is it the computation they are performing itself? Does the consciousness has a demarcation in space and time?
There are no satisfying answers to these questions if we assume mere computation can produce consciousness.
There are indeed no good answers, but that doesn't imply that the premise is wrong - it can just as well be that we assume that notions like "identity" are fundamental when it is in fact a property that we ascribe based on factors not directly related to consciousness (i.e. things like bodies), simply because our day-to-day experience always includes those factors and places them front and center.
Even with humans there are cases where this breaks down to some extent. E.g. in the case of multiple personality disorder, how many distinct streams of consciousness are there, and should we consider them distinct identities?
Just wanted to point out that I absolutely share your view here. I would like to add that the concept of virtualization and the required representation of computation makes substrate-independent consciousness rather absurd.
To me the only explanation for consciousness I find appealing is panprotopsychism.
I think to argue usefully about consciousness you've got to be able to define what you mean by it. If you use in the sense of a boxer is knocked unconscious as he's not aware of anything much versus conscious where he knows what's going on and can react and punch back, then AI systems can also be aware or not and react or not.
If you say it's all about the feelings and machines can't feel that way then it gets rather vague and hard to reason about. I mean they don't have much in the way of feelings now but I don't see why they shouldn't in the future.
I personally feel both those aspects of consciousness are not woo but the results mechanisms built by evolution for functional purposes. I'm not sure how they could have got their otherwise unless you are going to reject evolution and go for divine intervention or some such.
Consider a universe purely dictated by the mathematical laws of physics. It would be indistinguishable from our own to an observer, but such a universe would effectively be a fixed 4D structure, a statue incapable of experience. You have experience, yes? You think therefore you are. There exists something beyond maths and physics, experiencing the our universe, and you are that thing. How could such an entity develop from physical processes?
> such a universe would effectively be a fixed 4D structure, a statue incapable of experience
This claim requires proof. When I consider such a universe, it is obvious to me that such a universe would contain entities capable of experiencing it.
Why? We know that there are things which can experience but not be conscious, for instance simple robots. Do you think it would be impossible to have a universe full of people with no souls, who simply react to inputs like a robot? I think such a thing could exist and would be indistinguishable from out own.
A simple robot has neither consciousness nor experiences because it lacks the notion of self-identity. On the other hand, "soul" is itself a concept that is begging the question. If you consider consciousness an emergent property of sufficiently organized matter, then a world you describe is literally impossible.
AIs have self-concept, but I don't think you'd describe them as conscious. Likewise many animals do not recognise themselves in mirrors but still seem to experience the world.
> If you consider consciousness an emergent property of sufficiently organized matter, then a world you describe is literally impossible.
You are applying the word "impossible" to reject a perfectly valid consideration. It seems to be the case that in our universe, sufficiently organised matter can become conscious, but we can easily imagine one without this property. If you imagine a universe that only follows the laws of physics as we currently understand them, then it would not be the case. The universe would just be atoms bouncing off eachother. The idea that these atoms can come together to produce something which is more than a soulless automaton is an addition you are making to the laws of physics, not something present within them. We can imagine a world without this addition, just as we can imagine a world where gravity repels instead of attracting. We can see examples all over the place of things which act to approximate consciousness but are not conscious. It is no stretch to imagine a universe where this is a continuous scale, and consciousness doesn't magically emerge at some point along it.
There is a good reason why most people consider consciousness arising from the laws of physics as we currently understand them to be totally untenable. According to the known physics, the human mind is essentially just an advanced computer. To say that consciousness arises in any advanced computer is highly problematic. Take for instance this[1] popular comic. Do you honestly believe that this endless field of rocks could experience the world as you do if a man were to move them around in the right way?
> There is a good reason why most people consider consciousness arising from the laws of physics as we currently understand them to be totally untenable.
It is only "totally untenable" if you have a preconceived notion that humans are somehow special and not subject to the laws of physics. In other words, if you really want non-material souls to exist.
And yes, I do honestly believe that this endless field of rocks would "experience the world" if someone were to move them around in the right way. Although that is not entirely correct - whatever it is that is moving them in this way should also be considered a part of the overall system, and it is that system that would experience consciousness.
> and it is that system that would experience consciousness
This is not a common view, I'll just say that much. What happens if the man stops moving the rocks? Does the universe die? And if he starts again later? Is the universe conscious only in the instants where the rocks move or only when the man observes the outcome? My experience of consciousness doesn't seem compatible with something a bunch of rocks could feel if you moved them in the right way. And note importantly that this is something the rocks are doing on their own. The man could be moving them according to some set of rules which he doesn't understand, therefore the simple act of rocks moving creates life. I'm honestly astounded that you don't have anything in your experience of the world that you think wouldn't be felt by rocks that shift around in the right way.
> In other words, if you really want non-material souls to exist.
I don't want them to exist, I simply observe them existing. My experience of consciousness is not explained by any physical process. Physics can explain being which say they are conscious, but who hears me say it? A universe could be inhabited solely by chat bots, talking to eachother and claiming to be conscious, but there would be no one to observe that happening.
If you think of an emotional scene in a movie, say "frankly my dear I don't give a damn" in Gone With the Wind or whatever, there is obviously emotion in the scene if you watch it and you see the characters/actors experience it but the film is a stationary thing when it's sitting on the shelf, similar to a fixed 4D structure. I figure reality is like that.
It seems clear to us that the film is playing, but that's the magic isn't it? There is nothing in physics to suggest that the universe is anything other than a film on a shelf, yet we experience it playing. We are the characters in the movie playing out frames in a fixed sequence one after the other (as far as we know according to physics), yet we experience it where the characters in a film do not. We have a souls, or at least you do. For all you know you could be the only one.
Penrose believes that consciousness originates from quantum mechanics and the collapse of the wavefunction. Obviously you couldn't (effectively) simulate that with a classical computer. It's a very unconventional position, but it's not circular.
This is just another form of "god of the gaps" - Penrose desperately wants an interpretation that would allow for freedom of will, and so he constructs a theory around a physical process that allows for randomness, despite there being no evidence that this process is actually relevant to consciousness.
The argument that consciousness can't be computable seems like a stretch as well.