|
|
Close Help | ||||||||||||||
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Hello, and welcome to Coding a Transhuman AI (version 2.0 alpha), or "CaTAI" for short. CaTAI is the first serious attempt to sketch out the work needed and the principles involved in programming a general intelligence which is, or has the potential to become, smarter than human. (The project is presently in the design/conceptualization stage and no code has yet been written.)
The permanent address of this document is http://singinst.org/CaTAI.html.
©2000 by Eliezer S. Yudkowsky. All rights reserved.
 |
At present, CaTAI 2.0a contains only the topics "Paradigms" and "Mind",
although the published sections are complete and self-contained.
Some discussion of further topics (such as those which will eventually
be included under "Cognition" and "Design") can be found in the previous
version of this document,
CaTAI 1.0
(346K) at http://singinst.org/AI_design.temp.html.
However, I strongly advise that you read this version first.
Words defined in the glossary look like °this, but the glossary hasn't been compiled yet. |
°Ve, °vis, and °ver are gender-neutral
pronouns.
As a storyteller, I dislike giving away the climax in the opening pages. But a true mind is too tangled to be told as a linear story; everything depends on everything else. In these pages, I try not to rely on ideas that haven't been formally introduced. Even so, I think it'll be easier on my reader if I summarize some of the basic ideas in "Coding a Transhuman AI". Please bear in mind that this a summary only.
Previous AI has tended to suffer from exaggeration caused by academic polarization. I hope that you'll view the ideas that follow as gentle suggestions, rather than absolute injunctions; as pieces of the answer, which may or may not answer the entire question, but do count as progress. Just because a design feature is necessary doesn't make it sufficient. Too many failures have come of the trophy-hunting mentality, asking which buzzwords the code can be described by, and not asking what the code actually does.
This document is about what a general intelligence is, and how to build one. The desired end result is a self-enhancing mind or "seed AI". The term derives from the idea that, rather than trying to build a mind immediately capable of human-equivalent or transhuman reasoning, the goal is to build a mind capable of enhancing itself, and then re-enhancing itself with that higher intelligence, until the goal point is reached. "The task is not to build an AI with some astronomical level of intelligence; the task is building an AI which is capable of improving itself, of understanding and rewriting its own source code. The task is not to build a mighty oak tree, but a humble seed." (From 1.1: Seed AI.)
General intelligence itself is huge. The human brain, created by millions of years of evolution, contains more than a hundred neurologically distinguishable areas, composed of a hundred billion neurons connected by a hundred trillion synapses. We should not expect the problem of AI to be easy. Subproblems of cognition include attention, memory, association, abstraction, symbols, causality, subjunctivity, expectation, goals, actions, introspection, caching, and learning, to cite a non-exhaustive list. These features are not "emergent". They are complex functional adaptations whose functionality must be deliberately duplicated within the AI. If done right, cognition can support the thoughts implementing abilities such as analysis, design, understanding, invention, self-awareness, and the other facets which together sum to an intelligent mind. An intelligent mind with access to its own source code can do all kinds of neat stuff, but we'll get into that later.
Different schools of AI are distinguished by different kinds of underlying "mindstuff". Classical AI consists of "predicate calculus" or "propositional logic", which is to say suggestively named LISP tokens, plus directly coded procedures intended to imitate human formal logic. Connectionist AI consists of neurons implemented on the token level, with each neuron in the input and output layers having a programmer-determined interpretation, plus intervening layers which are usually not supposed to have a direct interpretation, with the overall network being trained by an external algorithm to perform perceptual tasks. (Although more biologically realistic implementations are emerging.) Agent-based AI consists of hundreds of humanly-written pieces of code which do whatever the programmer wants, with interactions ranging from handing data structures around to tampering with each other's behaviors.
Seed AI inherits connectionism's belief that error tolerance is a good thing. Error tolerance leads to the ability to mutate. The ability to mutate leads to evolution. Evolution leads to rich complexity - "mindstuff" with lots of tentacles and interconnections. However, connectionist theory presents a dualistic opposition between stochastic, error-tolerant neurons and the crystalline fragility of code or assembly language. This conflates two logically distinct ideas. It's possible to have crystalline neural networks in which a single error breaks the chain of causality, or stochastic code in which (for example) multiple, mutatable implementations of a function point have tweakable weightings. Seed AI strongly emphasizes the necessity of rich complexity in cognitive processes, and mistrusts classical AI's direct programmatic implementations.
However, seed AI also mistrusts the connectionist position which holds higher-level cognitive processes to be sacrosanct and opaque, off-limits to the human programmer, who is only allowed to fool around with the neuron behaviors and training algorithms, and not the actual network patterns. Seed AI does prefer learned concepts to preprogrammed ones, since learned concepts are richer. Nonetheless, it's permissible, if risky, to preprogram concepts in order to bootstrap the AI to the point where it can learn. More to the point, it's okay to have an architecture where, even though the higher levels are stochastic or self-organizing or emergent or learned or whatever, the programmer can still see and modify what's going on. And it is necessary that the designer know what's happening on the higher levels, at least in general, because cognitive abilities are not emergent and do not happen by accident. Both classical AI and connectionist AI propose a kind of magic that avoids the difficulty of actually implementing the higher layers of cognition. Classical AI states that a LISP token named "goal" is a goal. Connectionist AI declares that it can all be done with neurons and training algorithms. Seed AI admits the necessity of confronting the problem directly.
In the human brain, there's at least one multilevel system where the higher levels, though stochastic, still have known interpretations: the visual processing system. Feature extraction by the visual cortex and associated areas doesn't proceed in a strict hierarchy with numbered levels (seed AI mistrusts that sort of thing), but there are definitely lower-level features (such as retinal pixels), mid-level features (such as edges and surface textures), and high-level features (such as 3D shapes and moving objects). Together, the pixels and attached interpretations constitute the cognitive object that is a visual description. It's also possible to run the feature-extraction system in reverse, activate a high-level feature and have it draw in the mid-level features which draw in the low-level features. Such "reversible patterns" are necessary-but-not-sufficient to memory recall and directed imagination. Memory and imagination, when implemented via this method, can hold rich concepts that mutate interestingly and mix coherently. A mental image of a red sausage can mutate directly to a mental image of a blue sausage without either storing the perception of redness in a single crystalline predicate or mutating the image pixel by pixel. David Marr's paradigm of the "two-and-a-half dimensional world", multilevel holistic descriptions, is writ large and held to apply not just to sensory feature extraction but to categories, symbols, and other concepts. If seed AI has a "mindstuff", this is it.
Seed AI also emphasizes the problem of sensory modalities (such as the visual cortex, auditory cortex, and sensorimotor cortex in humans), previously considered a matter for specialized robots. A sensory modality consists of data structures suited to representing the "pixels" and features of the target domain, and codelets or processing stages which extract mid-level and high-level features of that domain. Sensory modalities grant superior intuitions and visualizational power in the target domain, which itself is sufficient reason to give a self-modifying AI a sensory modality for source code. Sensory modalities can also provide useful metaphors and concrete substrate for abstract reasoning about other domains; you can play chess using your visual cortex, or imagine a "branching" if-then-else statement. Finally, a sensory modality provides intuitions for understanding concrete problems in a training domain, such as source code. This makes it possible for the AI to learn the art of abstraction - moving from concrete problems, to categorizing sensory data, to conceptualizing complex methods, and so on - instead of being expected to swallow high-level thought all at once. Sensory modalities are the foundations of intelligence - a term carefully selected to reflect necessity but not sufficiency; after you build the foundations, there's still a lot of house left over.
Sensory modalities - visual, spatial, codic - are the bottom layer of the AI, the layer in which representations and behaviors are specified directly by the programmer. (Although avoiding the crystalline fragility of classical AI is still a design goal.) The next layer is concepts. Concepts are pieces of mindstuff, which can either describe the mental world, or can be applied to alter the mental world. (Note that successive concepts can be applied to a single target, building up a complex visualization.) Concepts are contained in long-term memory. Categories, symbols, and most varieties of declarative memory are concepts. Concepts are more powerful if they are learned, trained, or otherwise created by the AI, but can be created by the programmer for bootstrapping purposes. (If, of course, the programmer can hack the tools necessary to modify the concept level.) The underlying substrate of the concept can be code, assembly language, or neural nets, whichever is least fragile and is easiest to understand and mutate; this issue is discussed later, but I currently lean towards code.
Concepts, when retrieved from long-term memory, built into a structure, and activated, create a thought. The archetypal example of a thought is building words - symbols - into a grammatical sentence and "speaking" them within the mind. Thoughts exist in the RAM of the mind, the "working memory" created by available workspace in the sensory modalities. During their existence, thoughts can modify that portion of the world-model currently being examined in working memory. (Not every sentence spoken within the mind is supposed to describe reality; thoughts can also create and modify subjunctive ("what-if") hypotheses.) Thoughts are identified with - supposed to implement the functionality of - the human "stream of consciousness".
The three-layer model of intelligence is necessary, but not sufficient. Building an AI "with sensory modalities, concepts, and thoughts" is no guarantee of intelligence. The AI must have the right sensory modalities, the right concepts, and the right thoughts.
Evolution is the cause of intelligence in humans. Intelligence is an evolutionary advantage because it enables us to model, predict, and manipulate reality, including that portion of reality consisting of other humans and ourselves. In our physical Universe, reality tends to organize itself along lines that might be called "holistic" or "reductionistic", depending on whether you're looking up or looking down. "Which facts are likely to reappear? The simple facts. How to recognize them? Choose those that seem simple. Either this simplicity is real or the complex elements are indistinguishable. In the first case we're likely to meet this simple fact again either alone or as an element in a complex fact. The second case too has a good chance of recurring since nature doesn't randomly construct such cases." (Robert M. Pirsig, "Zen and the Art of Motorcycle Maintenance", p. 238.)
Thought takes place within a causal, goal-oriented, holistic world-model, and seeks to better understand the world or invent solutions to a problem. Methods include: Holistic analysis: Taking a known high-level characteristic of a know high-level object, and using learned heuristics to try and construct an explanation for the characteristic; an explanation consists of a low-level structure which gives rise to that high-level characteristic in a manner consistent with all known facts about the high-level object. Causal analysis: Taking a known fact and using heuristics to construct a causal sequence which results in that fact. Holistic design: Taking a high-level characteristic as a design goal, using heuristics to reduce the search space by reasoning about constraints on possible designs, and then testing ideas for specific low-level structures that attempt to satisfy the goals.
Both understanding and invention are fundamentally and messily recursive; whether a wheel works in a bicycle depends on whether that wheel consists of steel, rubber or tapioca pudding. Hence the need for heuristics (thoughts learned from experience) that bind high-level characteristics to low-level properties, and the need to recurse on finding new heuristics or more evidence or better tools or greater intelligence or higher self-awareness before the ultimate task can be solved.
When a sufficiently advanced AI can bind a high-level characteristic like "word-processing program" through the multiple layers of design to individual lines of code, ve can write a word-processing program given the verbal instruction of "Write a word-processing program." (This also assumes speech recognition and language processing - not to mention a very detailed knowledge of what a word-processing program is, what it does, what it's for, how humans will use it, and why the program shouldn't erase the hard drive.) When the AI, perhaps given a sensory modality for atoms and molecules, can understand all the extant research on molecular manipulation, ve can work out a sequence of steps which will result in the construction of a general nanotechnological assembler, or tools to build one. When the AI can bind a high-level characteristic like "useful intelligence" through the multiple layers of designed cognitive processes to individual lines of code, ve can redesign vis own source code and increase vis intelligence.
Developing such a seed AI may require a tremendous amount of programmer effort and programmer creativity; it is entirely possible that a seed AI is the most ambitious software project in history, not just in terms of the end result, but in terms of the sheer depth of internal design complexity. To bring the problem into the range of the humanly solvable, it is necessary that development be broken up into stages, so that the first stages of the AI can assist with later stages. The usual aphorism is that 10% of the code implements 90% of the functionality. Seed AI adds the distinction between learned concepts and programmer-designed concepts. If so, the first stage might be an AI with simplified modalities, preprogrammed simple concepts, low-level goal definitions, and perhaps even programmer-assisted development of the stream-of-consciousness reflexes needed for coherent thought. Such an AI would hopefully be capable of manipulating code in simple ways, thus rendering the source code for concepts (and in fact its own source code) subject to the type of flexible and useful mutations needed to learn rich concepts or evolve more optimized code. The skeleton AI helps us fill in the flesh on the skeleton.
...
..
.
Have you got all that?
Good.
Take a deep breath.
We're ready to begin.
It is probably impossible to write an AI which is immediate possession of human-equivalent abilities in every field; transhuman abilities even more so, since there's no working model. This is not necessary. The task is not to build an AI with some astronomical level of intelligence; the task is building an AI which is capable of improving itself, of understanding and rewriting its own source code. The task is not to build a mighty oak tree, but a humble seed.
As the AI rewrites itself, it moves along a trajectory of intelligence. The task is not to build an AI at some specific point on the trajectory, but to ensure that the trajectory is open-ended, reaching human equivalence and transcending it. Smarter and smarter AIs become more and more capable of rewriting their own code and making themselves even smarter. When writing a seed AI, it's not just what the AI can do now, but what it will be able to do later. And the problem isn't just writing good code, it's writing code that the seed AI can understand, since the eventual goal is for it to rewrite its own assembly language.
If "recursive self-enhancement" is to avoid running out of steam, it's necessary for code optimization or architectural changes to result in an increment of actual intelligence, of smartness, not just speed. Running an optimizing compiler over its own source code (1) may result in a faster optimizing compiler. Repeating the procedure a second time accomplishes nothing, producing an identical set of binaries; the same algorithm is being run, only faster. A human who fails to solve a problem in one year (or solves it suboptimally) may benefit from another ten years to think about the problem; even so, an individual human may eventually run out of ideas. An individual human who fails to solve a problem in a hundred years may, if somehow transformed into an Einstein, solve it within a short time. Faster simple algorithms accomplish little or nothing; faster intelligent thought can make a small difference; better intelligent thought makes the problem new again.
If each rung on the ladder of recursive self-enhancement involves a leap of sufficient magnitude, then each rung should open up enough new vistas of self-improvement for the next rung to be reached. If not, of course, the seed AI will optimize itself and use up all perceived opportunities for improvement without generating the insight needed to see new kinds of opportunities. In this case the seed AI will have stalled, and it will be time for the human programmers to go to work nudging it over the bottleneck. Ultimately, the AI must cross, not only the gap that separates the mythical average human from Einstein, but the gap that separates homo sapiens sapiens from homo sapiens neanderthalis. The leap to true understanding, when it happens, will open up at least as many possibilities as would be available to a human researcher with access to vis own neural source code.
A surprisingly frequent objection to self-enhancement is that intelligence, when defined as "the ability to increase intelligence", is a circular definition - one which would, they say, result in a sterile and uninteresting AI. Even if this were the definition (it isn't), and the definition were circular (it wouldn't be), the cycle could be broken simply by grounding the definition in chess-playing ability or some similar test of ability. However, intelligence is not defined as the ability to increase intelligence; that is simply the form of intelligent behavior we are most interested in. Intelligence is not defined at all. What intelligence is, if you look at a human, is more than a hundred cytoarchitecturally (2) distinct areas of the brain, all of which work together to create intelligence. Intelligence is, in short, modular, and the tasks performed by individual modules are different in kind from the nature of the overall intelligence. If the overall intelligence can turn around and look at a module as an isolated process, it can make clearly defined performance improvements - improvements that eventually sum up to improved overall intelligence - without ever confronting the circular problem of "making myself more intelligent". Intelligence, from a design perspective, is a goal with many, many subgoals. An intelligence seeking the goal of improved intelligence does not confront "improved intelligence" as a naked fact, but a very rich and complicated fact adorned with less complicated subgoals.
Presumably there is an ultimate limit to the intelligence that can be achieved on a given piece of hardware, but if the seed AI can design better hardware, the cycle continues. To be specific, if a seed AI is smart enough to chart a path from modern technological capabilities to the hardware described in K. Eric Drexler's Nanosystems, this should be enough computing power to provide thousands or millions of times the raw capacity of a human brain. (3). Whether the cognitive and technological trajectory beyond this point continues forever or tops out at some ultimate physical limit is basically irrelevant from a human perspective; nanotechnology plus thousands of times human brainpower should be far more than enough to accomplish whatever you wanted a transhuman for in the first place.
This scenario often meets with the objection that a lone AI can accomplish nothing; that technological advancement requires an entire civilization, with exchanges between thousands of scientists or millions of humans. This actually understates the problem. To think a single thought, it is necessary to duplicate far more than the genetically programmed functionality of a single human brain. After all, even if that functionality were duplicated perfectly, the AI might do nothing but burble for the first year - that's what human infants do.
Perceptions have to coalesce into concepts. The concepts have to be strung together into thoughts. Enough good thoughts have to be repeated often enough for the sequences to become "cached", for the often-repeated subpatterns to become reflex. Enough infrastructural reflexes must accumulate for one thought to give rise to another thought, in a connected chain, forming a stream of consciousness. Unless we want to sit around for years listening to the computer go ga-ga, the functionality of infancy must be either encapsulated in a virtual world that runs in computer time, or bypassed using a skeleton set of preprogrammed concepts and thoughts. (Hopefully, the "skeleton thoughts" will be replaced by real, learned thoughts as the seed AI practices thinking.)
Scientific thought relies on millennia of accumulated knowledge, the how-to-think heuristics discovered by hundreds of geniuses. While a seed AI may be able to absorb some of this knowledge by surfing the 'Net, there will be other dilemnas, unique to seed AIs, that it must solve on its own.
Finally, the autonomic processes of the human mind reflect millions of years of evolutionary optimization. Unless we want to expend an equal amount of programming effort, the functionality of evolution itself must be replaced - by the seed AI's self-tweaking of those algorithms, or by replacing processes that are autonomic in humans with the deliberate decisions of the seed AI.
That's a gargantuan job, but it's matched by equally powerful tools. The traditional advantages of prehuman AI are threefold: The ability to perform repetitive tasks without getting bored; the ability to perform algorithmic tasks at greater linear speeds than our 200 hz neurons permit; and the ability to perform complex algorithmic tasks without making mistakes (or rather, without making those classes of mistakes which are due to distraction or being unable to store all the interim results in short-term memory). All of which, of course, has nothing to do with intelligence.
The toolbox of seed AI is yet unknown; nobody has built one. This page is more about building the first stages, the task of getting the seed AI to say "Hello, world!" But, if this can be done, what advantages would we expect of a general intelligence with access to its own source code?
The ability to design new sensory modalities. In a sense, any human programmer is a blind painter - worse, a painter born without a visual cortex. Our programs are painted pixel by pixel, and are accordingly sensitive to single errors. We need to keep track of each line of code as an abstract object, floating above our visual cortex or auditory cortex. A seed AI could have a "codic cortex", a sensory modality devoted to code, with intuitions and instincts devoted to code, and the ability to abstract higher-level concepts from code and intuitively visualize complete models detailed in code. A human programmer is very far indeed from vis ancestral environment, but an AI can always be at home.
The ability to blend conscious and autonomic thought. Combining Deep Blue with Kasparov doesn't yield a being who can consciously examine a billion moves per second; it yields a Kasparov who can wonder "How can I put a queen here?" and blink out for a fraction of a second while a million moves are automatically examined. At a higher level of integration, Kasparov's conscious perceptions of each consciously examined chess position may incorporate data culled from a million possibilities, and Kasparov's dozen examined positions may not be consciously simulated moves, but "skips" to the dozen most plausible futures five moves ahead. (5).
Freedom from human failings, and especially human politics. The tendency to rationalize untenable positions to oneself, in order to win arguments and gain social status, seems so natural to us; it's hard to remember that rationalization is a complex functional adaptation, one that would have no reason to exist in "minds in general". A synthetic mind has no political instincts (6); a synthetic mind could run the course of human civilization without politically-imposed dead ends, without observer bias, without the tendency to rationalize. The reason we humans instinctively think that progress requires multiple minds is that we're used to human geniuses, who make one or two breakthroughs, but then get stuck on their Great Idea and oppose all progress until the next generation of brash young scientists comes along. A genius-equivalent mind that doesn't age and doesn't rationalize could encapsulate that cycle within a single entity.
Overpower - the ability to devote more raw computing power, or more efficient computing power, than is devoted to some module in the original human mind; the ability to throw more brainpower at the problem to yield intelligence of higher quality, greater quantity, faster speed, even difference in kind. Deep Blue eventually beat Kasparov by pouring huge amounts of computing power into what was essentially a glorified search tree; imagine if the basic component processes of human intelligence could be similarly overclocked...
Self-observation - the ability to capture the execution of a module and play it back in slow motion; the ability to watch one's own thoughts and trace out chains of causality; the ability to form concepts about self based on fine-grained introspection.
Conscious learning - the ability to deliberately construct or deliberately improve concepts and memories, rather than entrusting them to autonomic processes; the ability to tweak, optimize, or debug learned skills based on deliberate analysis.
Self-improvement - the ubiquitous glue that holds a seed AI's mind together; the means by which the AI moves from crystalline, programmer-implemented skeleton functionality to rich and flexible thoughts. In the human mind, stochastic concepts - combined answers made up of the average of many little answers - leads to error tolerance; error tolerance lets concepts mutate without breaking; mutation leads to evolutionary growth and rich complexity. An AI, by using probabilistic elements, can achieve the same effect; another route is deliberate observation and manipulation, leading to deliberate "mutations" with a vastly lower error rate. A blind search can become a heuristically guided search and vastly more useful; an autonomic process can become conscious and vastly richer; a conscious process can become autonomic and vastly faster - there is no sharp border between conscious learning and tweaking your own code. And finally, there are high-level redesigns, ones which require too many simultaneous, non-backwards-compatible changes to be implemented by evolution.
If all of that works, it gives rise to self-encapsulation and recursive self-enhancement. When the newborn mind fully understands vis own source code, when ve fully understands the intelligent reasoning that went into vis own creation - and when ve is capable of duplicating that reason independently, so that the mind contains its own design - the cycle is closed. The mind causes the design, and the design causes the mind. Any increase in intelligence, whether sparked by hardware or software, will result in a better mind; which, since the design was (or could be) generated by the mind, will propagate to cause a better design; which, in turn, will propagate to cause a better mind. (7). And since the seed AI will encapsulate not only the functionality of human individual intelligence but the functionality of evolution and society, these causes of intelligence will be subject to improvement as well. We might call it a "civilization-in-a-box", with more "hardware" intelligence than Einstein (8) and capable of codifying abstract thought to run at the linear speed of a modern computer.
A successful seed AI would have power. A genuine civilization-in-box, thinking at a millionfold human speed, might fold centuries of technological progress into mere hours. I won't belabor the point. I've done so in my other writings - Staring into the Singularity, for example. It's just important to realize that the fundamental purpose of transhuman AI differs from that of traditional AI.
The academic purpose of prehuman AI is to write programs that demonstrate some aspect of human thought - to hold a mirror up to the brain. The commercial purpose of prehuman AI is to automate tasks too boring, too fast, or too expensive for humans. It's possible to dispute whether an academic implementation actually captures an aspect of human intelligence, or whether a commercial application performs a task that deserves to be called "intelligent".
In transhuman AI, if success isn't blatantly obvious to everyone except trained philosophers, the effort has failed. My own purpose in creating transhuman AI is to safely develop nanotechnology and any subsequent ultratechnologies that may be possible, to use these technologies to provide an "operating system" for all the matter in the Solar System (and any other accessible matter), to divide up ownership of that matter fairly among all six billion inhabitants of Earth, and to provide a user-friendly API (including facilities whereby users may choose to be upgraded to superintelligence). It's just a high-tech implementation of the Standard Human Quest: To convert the Solar System from a collection of disorganized quarks into a collection of organized quarks; to bring all higher-level patterns under absolute human control; to erase death, pain, coercion, and stupidity from the human condition (or at least make them utterly voluntary); and to fulfill to the maximum possible extent whatever greater destiny or higher goals exist, if any do. (9).
To return to Earth: There will undoubtedly be many stages, many interim subgoals and interim successes, along the path to superintelligence. The key point is that while embodying some aspect of cognition may be useful or necessary, it is not an end in itself. Treating facets of cognition as ends in themselves has led traditional AI to develop a sort of "trophy mentality", a tendency to value programs according to whether they fit surface descriptions. (One gets the impression that if certain AI researchers were asked to write the next Great English Novel, they'd write a compiler manual and then tear off through the streets, shouting: "Eureka! It's in English! It's in English!") My hope is that the lofty but utilitarian goals of seed AI will lead to the habit of looking at every piece of the design and saying: "Sure, it's neat, but how does it contribute materially to general intelligence?" After all, if an aspect of cognition is duplicated faithfully but without understanding its overall purpose, it's a matter of pure faith to expect it to contribute anything.
But that gets us into the next section, "Thinking About AI".
AI has, in the past, failed repeatedly. The shadow cast by this failure falls over all proposals for new AI projects. The question is always asked: "Why won't your project fail, like all the other projects? Why did the previous projects fail? Does your theory of general intelligence explain the previous failures while predicting success for your own efforts?" Actually, anyone can explain away previous failures and predict success; all you have to do is assert that some particular new characteristic is the One Great Idea, necessary and sufficient to intelligence. The real question is whether a new approach to AI makes the failure of previous efforts seem massively inevitable, the predictable result of historical factors; whether the approach provides a theory of previous failures that is satisfyingly obvious in retrospect, makes earlier errors look like natural mistakes that any growing civilization might make, and thus "swallows" the historical failures in a new theory which leaves no dangling anxieties.
Okay. I won't go quite that far. Still, AI has an embarassing tendency to predict success where none materializes, to make mountains out of molehills, and to assert that some simpleminded pattern of suggestively-named LISP tokens completely explains some incredibly high-level thought process. Why?
Consider the symbol your mind contains for 'light bulb'. In your mind, the sounds of the spoken words "light bulb" are reconstructed in your auditory cortex. A picture of a light bulb is loaded into your visual cortex. Furthermore, the auditory and visual cortices are far more complex, and intelligent, than the algorithm your computer uses to play sounds and MPEG files. Your auditory cortex has evolved specifically to process incoming speech sounds, with better fineness and resolution than it displays on other auditory tasks. Your visual cortex does not simply contain a 2D pixel array. The visual cortex has specialized processes that extract David Marr's "two-and-a-half dimensional world" - edge detection, corner interpretation, surfaces, shading, movement - and processes that extract from this a model of 3D objects in a 3D world. "About 50 percent of the cerebral cortex of primates is devoted exclusively to vusual processing, and the estimated territory for humans is nearly comparable." (°MITECS (The MIT Encyclopedia of the Cognitive Sciences), "Mid-Level Vision".)
In the semantic net or Physical Symbol System of classical AI, a light bulb would be represented by an atomic LISP token named light-bulb.
| NOTE: | I say "LISP tokens", not "LISP symbols", despite convention and accepted usage. Calling the lowest level of the system "symbols" is a horrifically bad habit. |
Some of the problem may be explained by history; back when AI was being invented, in the 1950s and 1960s, researchers had tiny little machines that modern pocket calculators would sneer at. These early researchers chose to believe they could succeed with "symbols" composed of small LISP structures, cognitive "processes" with the complexity of one subroutine in a modern class library. They were wrong, but the need to believe produced approaches and paradigms that sank AI for decades.
Previous AI has been conducted under the Physicist's Paradigm. The development of physics over the past few centuries - at least, the dramatic, stereotypical part - has been characterized by the discovery of simple equations that neatly account for complex phenomena. In physics, the task is finding a single bright idea that explains everything. Newton took a single assumption (masses attract each other with a force equal to the product of the masses divided by the square of the distance) and churned through some calculus to show that, if an apple falls towards the ground at a constant acceleration, then this explains why planets move in elliptical orbits. The search for a similar fits-on-a-T-Shirt unifying principle to fully explain a brain with hundreds of cytoarchitecturally distinct areas has wreaked havoc on AI.
"Heuristics are compiled hindsight; they are judgemental rules which, if only we'd had them earlier, would have enabled us to reach our present state of achievement more rapidly." (Douglas Lenat, 1981.) The heuristic learned from past failures of AI might be titled "Necessary, But Not Sufficient". Whenever neural networks are mentioned in press releases, the blurb always includes the phrase "neural networks, which use the same parallel architecture found in the human brain". Of course, the "neurons" in neural networks are usually nothing remotely like biological neurons. But the main thing that gets overlooked is that it would be equally true (not very) to say that neural networks use the same parallel architecture found in an earthworm's brain. Regardless of whether neural networks are Necessary, they are certainly Not Sufficient. The human brain requires millions of years of evolution, thousands of modules, hundreds of thousands of adaptations, on top of the simple bright idea of "Hey, let's build a neural network!"
The Physicist's Paradigm lends itself easily to our need for drama. One great principle, one bold new idea, comes along to overthrow the false gods of the old religion... and set up a new bunch of false gods. As always when trying to prove a desired result from a flawed premise, the simplest path involves the Laws of Similarity and Contagion. For example, the "neurons" in neural networks involve associative links of activation. Therefore, the extremely subtle and high-level associative links of human concepts must be explained by this low-level property. Similarly, any instance of human deduction which can be written down (after the fact) as a syllogism must be explained by the blind operation of a ten-line-of-code process - even if the human thoughts blatantly involve a rich visualization of the subject matter, with the results yielded by direct examination of the visualization rather than formal deductive reasoning.
In AI, the one great simple idea usually operates on a low level, in accordance with the Physicist's Paradigm. Reasoning from similarity of surface properties is used to assert that high-level cognitive phenomena are explained by the low-level phenomenon, which (it is claimed) is both Necessary and Sufficient. This cognitive structure is a full-blown fallacy; it contains the social drama (one brilliant idea, new against old) and the rationalization (reasoning by similarity of surface properties, sympathetic magic) necessary to bear any amount of emotional weight. And that's how AI research goes wrong.
There are several ways to avoid making this class of mistake. One is to have the words "Necessary, But Not Sufficient" tattooed on your forehead. One is an intuition of causal analysis - I'll talk about how to build one later on - that says "This cause does not have sufficient complexity to explain this effect." One is to be instinctively wary of trying to implement cognition directly on the token level. (One is learning enough evolutionary psychology to recognize and counter ideology-based thoughts, but that's moving off-topic...)
One is introspection. Human introspection currently has a bad reputation in cognitive science, looked on as untrustworthy, unscientific, and easy to abuse. This is totally true. Still, you can't build a mind without a working model. It is necessary to know, intuitively, that classical-AI propositional logic - syllogisms, property inheritance, et cetera - is inadequate to explain your deduction that dropping an anvil on a car will break it. You should be able to see, introspectively, that there's more than that going on. You probably visualized an anvil smashing into your car's hood, the metal crumpling, and the windshield shattering. (10). Clearly visible is vastly more mental material, more cognitive "stuff", than classical-AI propositional logic involves.
The revolt against the Physicist's Paradigm can be formalized as the Law of Pragmatism:
| The Law of Pragmatism |
| Any form of cognition which can be mathematically formalized, or which has a provably correct implementation, is too simple to contribute materially to intelligence. |
The key words are "contribute materially". An architecture can be necessary to thought without accounting for the substance of thought. The Law of Pragmatism says that if a neural network's rules are simple enough to be formalized mathematically, than the substance of any intelligent answers produced by that network will be attributable to the specific pattern of weightings. If the pattern of weightings is created by a mathematically formalizable learning method, then the substance of intelligence will lie, not in the learning method, but in the intricate pattern of regularities within the training instances.
We can't be certain that the Law of Pragmatism will hold in the future, but it's definitely a heuristic in the Lenatian sense; if only we'd known it in the 1960s, so much error could have been avoided. The Law of Pragmatism is one of the tools used to determine whether an idea is Necessary, But Not Sufficient.
°CaTAI (11) proposes a mind which contains modules vaguely analogous to human sensory modalities (auditory cortex, visual cortex, etc.). This does not mean that you can design any old system which can be described as "containing modular sensory modalities" and then dash off a press release about how your company is composing an AI containing modular sensory modalities. That's the tropy mentality I was talking about earlier. A modular, modality-based system is Necessary, But Not Sufficient; it is also necessary to have the right modules, in the right sensory modalities, using the right representation and the right intuitions to process the right base of experience to produce the right concepts that support the right thoughts within the right larger architecture.
When you think of a light bulb, the syllables and phonemes of "light bulb" are loaded into your auditory cortex; if you're a visual person, a generic picture of a light bulb - the default prototype - appears in your visual cortex. Let's suppose that some AI has reasonably sophisticated analogues of the auditory cortex and visual cortex, capable of perceiving higher-level features as well as the raw binary data. This is clearly necessary; is it sufficient to understand light bulbs in the same way as a human?
No. Not even close. When you hear the phrase "triangular light bulb", you visualize a triangular light bulb.
| NOTE: | Please halt, close your eyes, and visualize a triangular light bulb. Please? Pretty please with sugar on top? |
How do these two symbols combine? You know that light bulbs are fragile; you have a built-in comprehension of real-world physics - sometimes called "naive" physics - that enables you to understand fragility. You understand that the bulb and the filament are made of different materials; you can somehow attribute non-visual properties to pieces of the three-dimensional shape hanging in your visual cortex. If you try to design a triangular light bulb, you'll design a flourescent triangular loop, or a pyramid-shaped incandescent bulb; in either case, unlike the default visualization of "triangle", the result will not have sharp edges. You know that sharp edges, on glass, will cut the hand that holds it.
Look at all that! It requires a temporal, four-dimensional understanding of the light bulb. It requires an appreciation, a set of intuitions, for cause and effect. It requires that you be capable of spotting a problem - a conflict with a goal - which requires means for representing conflicts, and cognitive reflexes derived from a goal system.
Look at yourself "looking at all that". It requires introspection, reflection, self-perception. It requires an entire self-sensory modality - representations, intuitions, cached reflexes, expectations - focused on the mind doing the thinking.
For you to read this paragraph, and think about it, requires a stream of consciousness. For you to think about light bulbs implies that you codified your past experiences of actual light bulbs into the representation used by your long-term memory. The visual image of the light bulb, appearing in your visual cortex, implies that a default prototype for "light bulb" was abstracted from experience, stored under the symbol for "light bulb", and triggered by that symbol's auditory tag of 'light bulb'. And this prototype can even be combined with the learned symbol for "triangle". You have formed an adjective, "triangular", consisting of characteristics which can be applied to modify the visual and design substance of the light-bulb concept. For you to visualize a light-bulb smashing, with an accompanying tinkling noise, requires synchronization of recollection and reconstruction across multiple sensory modalities.
I've mentioned many features in the last paragraphs; none of them are emergent. None of them will magically pop into existence on the high level "if only the simple low-level equation can be found". In a human, these features are complex functional adaptations, generated by millions of years of evolution. For an AI, that means you sit down and write the code; that you change the design, or add design elements (special-purpose low-level code that directly implements a high-level case is usually a Bad Thing), specifically to yield the needed result.
In short, the design in CaTAI is simply far larger, as a system architecture, than any design which has been previously attempted. It's large enough to resemble systems of the complexity described in the 471 articles in The MIT Encyclopedia of the Cognitive Sciences. (12). You'll appreciate this better after reading the rest of the document, of course, but when you have done so, I expect that seed AI will look too different from past failures for one to reflect on the other. Fish and fowl, apples and oranges, elephants and screwdrivers. There is still the possibility that any given seed AI project will fail, or even that seed AI itself will fail - but if so, it will fail for different reasons.
Intelligence is an evolutionary advantage because it enables us to model, predict, and manipulate reality. This includes not only Joe Caveman (or rather, Pat Hunter-Gatherer) inventing the bow and arrow, but Chris Tribal-Chief outwitting his (13) political rivals and Sandy Spear-Maker realizing that the reason her spears keep breaking is that she's being too impatient while making them. That is, the "reality" we model includes not just things, but other humans, and the self. (14).
A chain of reasoning is important because it ends with a conclusion about how the world works, or about how the world can be altered. The "world", for these purposes, includes the internal world of the AI; when designing a bicycle, the hypothesis "a round object can traverse ground without bumping" is a statement about the external world. The hypotheses "it'd be a good idea to think about round objects", or "the key problem is to figure out how to interface with the ground", or even "I feel like designing a bicycle", are statements about the internal world.
From an external perspective, cognitive events matter only insofar as they affect external behavior. Just so, from an internal perspective, the effect on the world-model is the punchline, the substance. This is not to say that every line of code must make a change to the world-model, or that the world-model is composed exclusively of high-level beliefs about the real world. The thought sequences that construct a what-if scenario - a °subjunctive fantasy world - are altering a world-model, even if it's not the model of the world. A "vague feeling that there's some kind of as-yet unnamed similarity between two pictures" is part of the content of the AI's beliefs about the world. The code that produces that intuition may undergo many internal iterations, acting on data structures with no obvious correspondence to the world-model, before producing an understandable output.
What makes a pattern of bytes - or neurons - a "model"? And what makes a particular statement in that model "true" or "false"? (15). The best definition I've found is derived from looking at the cause of our intelligence: "Intelligence is an evolutionary advantage because it enables us to model, predict, and manipulate reality." Models are useful because they correspond to external reality.
<glossary a=SPDM g=SPDM r=SPDM>I distinguish four levels of binding:
(17).
The "world-model" for an AI living in that world consists of everything the AI knows about that world - the positions, velocities, radii, and masses of the billiard balls. More abstract perceptions, such as "a group of three billiard balls", are also part of the world-model. The prediction that "billiard ball A and billiard ball B will collide" is part of the world-model. If the AI imagines a situation where four billiard balls are arranged in a square, then that imaginary world has its own, subjunctive world-model. If the AI believes "'imagining four billiard balls in a square' will prove useful in solving problem X", then that belief is part of the world-model. In short, the world-model is not necessarily a programmatic concept - a unified set of data structures with a common format and °API. (Although it would be wonderfully convenient, if we could pull it off.) The "world-model" is a cognitive concept; it refers to the content of all beliefs, the substance of all mental imagery.
Returning to the billiard-ball world, what is necessary for an AI to have a "model" of this world?
Suppose that a cue ball travelling south at 4 meters/second, bumping into a billiard ball travelling south at 2 meters/second, results in the cue ball and the billiard ball travelling south at 3 meters/second. Suppose, furthermore, that these rules are contained within the AI's internal model of the environment, so that if the AI visualizes a cue ball at {8.2, 6} of radius 1 travelling south at 4 m/s, and a ball at {8.2, 10} of radius 1 going south at 2 m/s, the AI will visualize the balls bumping one second later at {8.2, 11}, and the two balls then travelling south at 3 m/s. It's a long way from there to knowing - consciously, °declaratively - that two balls in general bumping at 4 m/s and 2 m/s while going in the same direction will travel on together at 3 m/s. It's an even longer way to knowing that "if billiard ball X bumps into billiard ball Y, then they will continue on together with the average of their velocities". And it's a still longer way to reversing the rule and knowing that "to get a group of two balls travelling together with velocity X, given billiard ball A with velocity Y, bump it with billiard ball B having velocity (2X - Y)". Finally, to close the loop, this last high-level rule must be applied to create a particular hypothesized action in the world-model, and the hypothesized action needs to be taken as a real action in external reality.
Without jumping too far ahead, there are a number of properties that a world-model needs to support high-level thought. It needs to support time - multiple frames or a temporal visualization - with accompanying extraction of temporal features. It needs to support predictions, or expectations (and an expectation isn't real unless the AI notices when the expectation is fulfilled, and especially when it is violated). The world-model needs to support hypotheses, subjunctive frames of visualization, which are distinct from "real reality" and can be manipulated freely by high-level thought. (By "freely manipulated", I mean a direct manipulative binding; choosing to think about a billiard ball at position {2, 3} should cause a billiard ball to materialize directly within the representation at {2, 3}, with no careful sequence of actions required.) And for the visualization to be useful once it exists, the high-level thought must refer to the particular billiard ball visualized... and the reference must run both ways, a two-way linkage.
Time, expectation, comparision, subjunctivity, visualization, introspection, and reference. I haven't defined any of these terms yet. (Most are discussed in the section on "Cognition", which has not yet been published.) Nonetheless, these are some of the basic attributes that are present in human world-models, and which are Necessary (But Not Sufficient) for the existence of high-level features such as causality, intentionality, goals, memory, learning, association, focus, abstraction, categorization, and symbolization.
| NOTE: | I mention that list of features to illustrate what will probably be one of the major headaches for AI designers: If you design a system and forget to allow for the possibility of expectation, comparision, subjunctivity, visualization, or whatever, then you'll either have to go back and redesign every single component to open up space for the new possibilities, or start all over from scratch. Actualities can always be written in later, but the potential has to be there from the beginning, and that means a designer who knows the requirements spec in advance. |
A human has a visual cortex, an auditory cortex, a sensorimotor cortex - areas of the brain specifically devoted to particular senses. Each such "cortex" is composed of neural modules which extract important mid-level and high-level features from the low-level data, in a way determined by the "laws of physics" of that domain. The visual cortex and associated areas (20) are by far the best-understood parts of the brain, so that's what we'll use for an example.
Visual information starts out as light hitting the retina; the resulting information can be thought of as being analogous to a two-dimensional array of pixels (although the neural "pixels" aren't rectangular). "Low-level" feature extraction starts right in the retina, with neurons that respond to edges, intensity changes, light spots, dark spots, et cetera. From this new representation - the 2D pixels, plus features like edges, light spots, and so on - the lateral geniculate nucleus and striate cortex extract mid-level features such as edge orientation, movement, direction of moving features, textures, the curvature of textured surfaces, shading, and binocular perception. This information yields °David Marr's two-and-a-half-dimensional world, which is composed of scattered facts about the three-dimensional properties of two-dimensional features - this is a continuous surface, this surface is curving away and to the left, these two surfaces meet to form an edge, these three edges meet to form a corner.
Finally, a 3D representation of moving objects is constructed from the 2.5D world. Constraint propagation: If the 3D interpretation of one corner requires an edge to be convex, then that edge cannot be concave in another corner. Object assembly: Multiple surfaces that move at the same speed, or that move in a fashion consistent with rotation, are part of a single object. Consistency: An object (or an edge, or a surface) cannot simultaneously be moving in two directions.
The resulting 3D representation, still bound to the 2.5D features and the 2D pixels, is sent to the temporal cortex for object recognition and to the parietal cortex for spatial visualization.
The visual cortex is the foundation of one of the seven senses. (Yes, seven. In addition to sight, sound, taste, smell, and touch, there's proprioception (the nerves that tell us where our arms and legs are) and the vestibular sense (the inner ear's inertial motion-detectors). (21).) The neural areas that are devoted solely to processing one sense or another account for a huge chunk of the human cortex. In the modular partitioning of the human brain, the single most common type of module is a sensory modality, or a piece of one. This demonstrates a fundamental lesson about minds in general - a lesson that AI has yet to learn.
Classical AI programs, particularly "expert systems", are often partitioned into microtheories. A microtheory is a body of knowledge, i.e. a big semantic net, e.g. propositional logic, a.k.a. suggestively named LISP tokens. A typical microtheory subject is a human specialty, such as "cars" or "childhood diseases" or "oil refineries". The content of knowledge typically consists of what would, in a human, be very high-level, heuristic statements: "A child that is sick on Saturday is more likely to be seriously ill than a child who's sick on a schoolday."
How do the microtheory-based modules of classical AI differ from the sensory modules that are common in the human mind? How does a "microtheory of vision" differ from a "visual cortex"? Why did the microtheory approach fail?
There are two fundamental clues that, in retrospect, should have alerted expert-system theorists ("knowledge engineers") that something was wrong. First, microtheories attempt to embody high-level rules of reasoning - heuristics that require a lot of pre-existing content in the world-model. The visual cortex doesn't know about butterflies; it knows about edge-detection. The visual cortex doesn't contain a preprogrammed picture of a butterfly; it contains the feature-extractors that let you look at a butterfly, parse it as a distinct object standing out against the background, remember that object apart from the background, and reconstruct a picture of that object from memory. We are not born with experience of butterflies; we are born with the visual cortex that gives us the capability to experience and remember butterflies. The visual cortex is not visual knowledge; it is the space in which visual knowledge exists.
The second, deeper problem follows from the first. All of an expert system's microtheories have the same underlying data structures (in this case, propositional logic), acted on by the same underlying procedures (in this case, a few rules of °Bayesian reasoning). Why separate something into distinct modules if they all use the same data structures and the same functions? Shouldn't a real program have more than one real module?
I'm not suggesting that data formats and modules be proliferated because this will magically make the program work better. Any competent programmer knows better than to use two data formats where one will do. But if the data and processes aren't complex enough to seize the programmer by the throat and force a modular architecture, then the program is too simple to give rise to real intelligence.
And a single-module architecture certainly isn't the way the brain does it. Maybe there's some ingenious way to represent auditory and visual information using a single underlying data structure. If we can get away with it, great. But if no act of genius is required to solve the very deep problem of getting domain-specific representations to interact usefully, if the problem is "solved" because all the content of thought takes the form of propositional logic, if all the behaviors can fit comfortably into a single programmatic module - then the program doesn't have enough complexity to be a decent video game, much less an AI. (22).
We shouldn't be too harsh on the classical-AI researchers. The idea of building an AI that operates on "pure logic" - no sensory modalities, no equivalent to the visual cortex - was certainly worth trying. As Ed Regis would say, it had a certain hubristic appeal. Why does human thought use the visual cortex? Because it's there! After all, if you've already evolved a visual cortex, further adaptations will naturally take advantage of it. It doesn't mean that an engineer, working ab initio, must be bound by the human way of doing things.
But it didn't work. The recipe for intelligence presented by CaTAI assumes an AI that possesses equivalents to the visual cortex, auditory cortex, and so on. Not necessarily these particular cortices; after all, Helen Keller (who was blind and deaf, and spoke in hand signs) learned to think intelligently. But even Helen Keller had proprioception, and thus a parietal lobe for spatial orientations; she had a sense of touch, which she could use to "listen" to sign language; she could use the sensory modalities she had to perceive signed symbols, and form symbols internally, and string those symbols together to form sentences, and think. (23) Some equivalent of some type of "cortex" is necessary to the CaTAI design.
"Cortex" is a specifically neurological term referring to the surface area of the brain, and therefore I will use the term "sensory modality", or "modality", instead of cortex. (The previous edition of "Coding a Transhuman AI" used the term "domdule" (from "domain module") for "modality", and the term still appears in my other web pages.)
| DEFN: | Modality: Modalities in an AI are analogous to human cortices - visual cortex, auditory cortex, et cetera - enabling the AI to visualize processes in the target domain. Modalities capture, not high-level knowledge, but low-level behaviors. A modality has data structures suited to representing the target domain, and °codelets or processing stages which extract higher-level features from raw data. |
Why does an AI need a visual modality? Because the human visual cortex and associated neuroanatomy - our visual modality - is what makes our thoughts of 2D and 3D objects real. Drew McDermott, in Artificial Intelligence Meets Natural Stupidity, pointed out that, just because a LISP token is labeled with the character string "hamburger", it does not mean that the program understands hamburgers. The program has not even noticed hamburgers. If the symbol were called G0025 instead of hamburger, nobody would ever be able to figure out that the token was supposed to represent a hamburger.
When two objects collide, we don't just have a bit of propositional logic that says collide(car, truck); we imagine two moving objects. We model 2D pixels and 3D features and visualize the objects crashing together. The edges touch, not as touch(edge-of(car), edge-of(truck)), but as two curves meeting and deforming at all the individual points along the edge. You could successfully look at a human brain and deduce that the neurons in question were modelling edges and colliding objects; this is, in fact, what visual neuroanatomists do. But if you did the same to a classical AI, if you stripped away the handy English variable names from the propositional logic, you'd be left with G0025(Q0423, U0111) and H0096(D0103(Q0423), D0103(U0111)). No amount of reasoning could bind those cryptic numbers to real-world cars or trucks.
Furthermore, our visual cortex is useful for more than vision. Philosophy in the Flesh (George Lakoff and Mark Johnson) talks about the Source-Path-Goal pattern (24) - a trajector that moves, a starting point, a goal, a route; the position of the trajector at a given time, the direction at that time, the actual final destination... Philosophy in the Flesh also talks about "internal spatial 'logic' and built-in inferences": If you traverse a route, you have been at all locations along the route; if you travel from A to B and B to C, you have traveled from A to C; if X and Y are traveling along a direct route from A to B and X passes Y, then X is further from A and closer to B than Y is.
These are all behaviors of spatial reality. Classical AI would attempt to capture descriptions of this behavior; i.e. "if travel(X, A, B) and travel(X, B, C) then travel(X, A, C)". The problem is that the low-level elements (pixels, trajectors, velocities) making up the model can yield a nearly infinite number of high-level behaviors, all of which - under the classical-AI method - must be described independently. If A is-contained-in B, it can't get out - unless B has-a-hole. Unless A is-larger-than the hole. Unless A can-turn-on-its-side or the hole is-flexible. Trying to describe all the possible behaviors exhibited by the high-level characteristics, without directly simulating the underlying reality, is like trying to design a CPU that multiplies two 32-bit numbers using a doubly-indexed lookup table with 2^64 (around eighteen billion billion) entries.
Real CPUs take advantage of the fact that 32-bit numbers are made of bits. This enables transistors to multiply using the wedding-cake method (or whatever it is modern CPU designs use). A 32-bit number is not a monolithic object. The numerical interpretation of 32 binary digits is not intrinsic, but rather a high-level characteristic, an observation, an abstraction. The individual bits interact, and yield a 32-bit (or 64-bit) result which can then be interpreted as the resulting number. The computer can multiply 9825 by 767 and get 7535775, not because someone told it that 9825 times 767 is 7535775, but because someone told it about how to multiply the individual bits.
A visual modality grants the power to observe, predict, decide, and manipulate objects moving in trajectories, not because the modality captures knowledge of high-level characteristics, but because the modality has elements which behave in the same way as the external reality. An AI with a visual modality has the potential to understand the concept of "closer", not because it has vast stores of propositional logic about closer(A, B), but because the model of A and B is composed of actual pixels which are actually getting closer. (25).
Source-Path-Goal pattern is not just a visual pattern. It is a metaphor that applies to almost any effort. Force and resistance aren't just people pushing carts, they're companies pushing products. Source-Path-Goal applies not just to walking to Manhattan, but a programmer struggling to write an application that conforms to the requirements spec. It applies to the progress of these very words, moving across the screen as I type them, decreasing the distance to the goal of a publishable Web page. Furthermore, the visual metaphor is in many cases a useful metaphor, one which binds predictively. (26). A metaphor is useful when it involves, not just a similarity of high-level characteristics, but a similarity of low-level elements, or a single underlying cause. (See previous footnote.) The visual metaphor that maps a programming task to Source-Path-Goal (a visual object moving along a visual line) is useful if some measure of "task completed" can be mapped to the quantitative position of the trajector, and the perceived velocity used to (correctly!) predict the amount of time remaining on the task.
Of course, one must realize that having a visual modality is Necessary, But Not Sufficient, to pulling that kind of stunt. In such cases, noticing the analogy is ninety percent of the creativity. The atomic case of such an intuition would consist of generating models at random, either by generating random data sets or by randomly mixing previously acquired models, until some covariance, some similarity, is noticed between the model and the reality. And then the AI says "Eureka!"
Of course, except for very simple metaphors, the search space is too large for blind constructs to ever match up with reality. It is more often necessary to deliberately construct a model - in this case, a visual model - whose behaviors correspond to reality. Discussion of such higher-level reasoning doesn't belong in the section on "sensory modalities", but being able to "deliberately construct" anything requires a way to manipulate the visual model. In addition to the hardware/code for taking the external action of "draw a square on the sheet of paper", a mind requires the hardware/code to take the internal action of "imagine a square". The consequence, in terms of how sensory modalities are programmed, is that feature extraction needs to be reversible. Not all of the features all of the time, of course, but for the cognitive act of visualization to be possible, there must be a mechanism whereby the perception for the "line" feature has an inverse function that constructs a line, or transforms something else into a line.
Feature reconstruction is much more difficult to program than feature extraction. More computationally intensive, too. It's the difference between multiplying the low-level elements of "7" and "17", and reconstructing two low-level elements which could have yielded the high-level feature of "119". This may be one of the reasons why thalamocortical sensory pathways are always reciprocated by corticothalamic projections of equal or greater size; for example, a cat has 10^6 neural fibers leading from the lateral geniculate nucleus to the visual cortex, but 10^7 fibers going in the reverse direction. (27).
Even a complete sensory modality, capable of perception and visualization, is useless without the rest of the AI. "Necessary, But Not Sufficient," the phrase goes. A modality provides some of the raw material that concepts are made of - the space in which visualizations exist, but nothing more. But, granting that the rest of the AI has been done properly, a visual modality will create the potential to understand the concept of "closer"; to use the concept of "closer", and heuristics derived from examining instances of the concept "closer", as a useful visual metaphor for other tasks; and to use deliberately constructed models, existing in the visual modality, to ground thinking about generic processes and interactions. (In other words, when considering a "fork" in chess or an "if" statement in code, it can be visualized as an object with a Y-shaped trajectory decision.)
Is a complete visual modality - pixels, edge detectors, surface-texture decoders, and all - really necessary to engage in spatial reasoning? Would a world of Newtonian billiard balls, with velocities and collision-detection, do as well? It would apparently suffice to represent concepts such as "fork", "if statement", "source-path-goal", "closer", and most generic processes composed of discrete objects. The billiard-ball world has significantly less representative power; it's harder to understand a "curved trajectory" in spacetime if you can't visualize a curve in space. (28) But, considering the sheer programmatic difficulty of coding a visual modality, are metaphors with billiard balls composed of pixels that superior to metaphors with billiard balls implemented directly as low-level elements?
Well, yes. In a visual modality, you can switch from round billiard balls to square billiard balls, visualize them deforming as they touch, and otherwise "think outside the box". The potential for thinking outside the box, in this case, exists because the process being modeled has elements that are represented by high-level visual objects; these high-level visual objects in turn are composed of mid-level visual features which are composed of low-level visual elements. This provides wiggle room for creativity.
Consider the famous puzzle with nine dots arranged in a square, where you're supposed to draw four straight lines, without lifting pen from paper, to connect the dots. (29). To solve the puzzle one must "think outside the box" - that is, draw lines which extend beyond the confines of the square. A conventional computer program written to solve this problem would probably contain the "box" as an assumption built into the code, which is why computers have a reputation for lack of creativity. (30). A billiard-ball metaphor, even assuming that it could represent lines, might run into the same problem.
I suspect that many solvers of the nine-dot problem do so because a particular configuration of tried-out lines suggests an incomplete triangle whose corners lie outside the box. "Seeing" an "incomplete triangle" is an optical illusion, which is to say that it's the result of high-level features being triggered and suggesting mid-level features - in this case, some extra lines that turn out to be the solution to the problem. Sure, you can make up ways that this could happen in a billiards modality, but then the billiards modality starts looking like a visual cortex. The point is that, for our particular human style of creativity, it is Necessary (But Not Sufficient) to have a modality with rich "extraneous" perceptions, and where high-level objects in the metaphor can be made to do unconventional things by mentally manipulating the low-level elements. (Even so, it would make development sense to start out with a billiards modality and work up to vision gradually.)
There are two final reasons for giving a seed AI sensory modalities: First, the possession of a codic modality may improve the AI's understanding of source code, at least until the AI is smart enough to make its own decisions about the balance between slow-conscious and fast-autonomic thought. Second, as will be discussed later, thoughts don't start out as abstract; they reach what we would consider the "abstract" level by laying down a layer cake of ideas. That layer cake starts with the non-abstract, autonomic intuitions and perceptions of the world described by modalities. The concrete world provided by modalities is what enables the AI to learn its way up to tackling abstract problems.
| NOTE: | One of the greatest advantages
of seed AI - second only to recursive self-improvement - is going beyond
the human sensory modalities. It's possible to create a sensory modality
for source code. Various processes that are autonomic in humans -
memory storage, symbol formation - can become sensory modalities subject
to deliberate manipulation.
In programmatic terms, any program module with a coherent set of data structures and an API, which could benefit from higher-level thinking, is a candidate for transformation into a modality with world-model-capable representations, feature extraction, reversible features to allow mental actions, and the other design characteristics required to support concept formation. |
Modalities in the human brain are more or less preprogrammed, as opposed to learned. (Human modalities require external stimuli to grow into their preprogrammed organization, but this is not the same as learning.) Individual neurons can have meanings that are visible and understandable to an eavesdropper. Programmers may legitimately take the risk of creating modalities through deliberate programming, with low-level elements that correspond to data structures, and human-written procedures for feature extraction.
Within CaTAI, the term concept is used to refer to the kind of mental stuff that exists as a pattern in the modality. A learned sequence of instructions that reconstructs the default prototype of "light bulb" in the visual modality is a concept. Symbols, categories, and some memories are concepts. (Despite common usage, "concept" might technically refer to non-declarative mental stuff such as a human cognitive reflex or a human motor skill. However, in a seed AI, where everything is open to introspection, it makes sense to call the equivalents of human reflexes or skills "concepts".) Concepts are patterns, learned or preprogrammed, that exist in long-term storage and can be retrieved.
A structure of concepts creates a thought. The archetypal example, in humans, is words coming together to form sentences. Thoughts are visualized; they operate within the RAM of the mind, the "workspace" represented by available content capacity in the sensory modalities, commonly called "short-term memory" or "working memory". (The capacity of working memory is not determined by available RAM, but by available CPU capacity to perform feature extraction on the contents of memory. If you have the data structures without the feature extraction, the AI won't notice the information.) Thoughts manipulate the world-model.
In humans, at least, it's hard to draw clean boundaries between thoughts and concepts. (31). The experience of hearing the word for a single concept, such as "triangle", is not necessarily a concept; it may be more valid to view it as a thought composed of the single concept "triangle". And, although some concepts are formed by categorizing directly from sense perception, more abstract concepts such as "three" probably occur first as deliberate thoughts. We'll be discussing both types in this section.
In chemistry, abstract means remove; to "abstract" an atom from a molecule means to take it away. Use of the term "abstract" to describe the process of forming concepts implies two assumptions: First, to create a concept is to generalize; second, to generalize is to lose information. That, to form the concept of "red", it is necessary to ignore other high-level features such as shape and size, and focus only on color.
This is the classical-AI view of abstraction, and we should therefore be suspicious of it. On the other hand, it is certainly true that our mechanisms for abstraction can learn the concept for "red". In a being with a visual modality, this concept would consist of a piece of mindstuff that had learned to distinguish between red objects and non-red objects. Since redness is probably detected directly as a low-level feature, it shouldn't be very hard to train a piece of mindstuff to do so - whether the mindstuff is made of trainable neurons, evolving code, or whatever. A neural net needs to learn to fire when the "red" feature is present, and not otherwise; a piece of code only needs to test for the presence of the redness feature. At most, it might also be necessary to test for solid-color or same-hue groupings. Given a visual modality, the concept of "red" lies very close to the surface.
Of course, to have a real concept for "red", it's not enough to distinguish between red and non-red. The concept has to be applicable; you have to be able to apply it to visualizations, as in "red dog". You also need a default prototype (32) for "red"; and an extreme prototype for "red"; and memories of experiences that are stereotypically red, such as stoplights and blood. (For all we know, leaving out any one of these would be enough to totally hose the flow of cognition.) Again, these features lie close to the surface of a visual modality. "Red" would be one of the easiest features to make reversible, with little additional computational cost involved; just set the hue of all colors to a red value. (Although hopefully in such a way as to preserve all detected edges, contrasts, and so on. Making everything exactly the same color would destroy non-color features.) The default prototype for red can be a red blob, or a red light; the extreme prototype for red may be the same as the default prototype, or it may be a more intensely red blob. And the stereotypically red objects, such as stoplights and blood, are the objects in which the redness is important, and much remarked upon.
(33).
For the moment, however, let's concentrate on the problem of forming categories. The conventional wisdom states that categorization consists of generalization, and that generalization consists of focusing on particular features at the expense of others.
We'll use the microdomain of letter-strings as an example. To generalize from the instances {"aaa", "bbb", "ccc"} to form the category "strings-of-three-equal-letters", the information about which letter must be abstracted, or lost, from the model. Actually, this misstates the problem. If you lose that information on a letter-by-letter basis, then "aaa" and "aab" both look like "***". What's needed is for the letter-string modality to first extract the features of "group-of-equal-letters", "number=3", and "letter=b", after which the concept can lose the last feature or focus on the first two. If the second feature, "number", is also lost, then the result is an even more general concept, "strings-of-equal-letters". Of course, this concept is precisely identical to the modality's built-in feature-detector for "group-of-equal-letters", which again points up that only very simple conceptual categories, lying very close to the surface of the modality's preprogrammed assumptions about which features are important, can be implemented by direct information-loss.
To examine a more complex concept, we'll look at the example of "three".
To a twenty-first-century human, trained in arithmetic and mathematics, the concept of "three" has enormous richness. It must be emphasized that we are dealing solely with the concept of "three", and that a mind can understand "three" without understanding "two" or "four" or "number" or "addition" or "multiplication". A mind may have the concept "three" and the concept "two" without noticing any similarity between them, much less having the aha! that these concepts should go together under the heading "number". If a mind somehow manages to pick up the categories of groups-of-three-dogs and groups-of-three-cats, it still doesn't follow that the mind will generalize to the category of "three". To think about infant-level or child-level AIs, or for that matter to teach human children, it's necessary to slow down and forget about what seems "natural". It's necessary to make a conscious separation between ideas; ideas that, to us, look so close together that it takes a deliberate effort to see the distance.
Likewise, just because the AI exists on a machine performing billions of arithmetical operations per second doesn't mean that the AI itself must understand arithmetic or "three". (John Searle, take note!) Even if the AI has a codic modality which grants it direct access to numerical operations, it doesn't necessarily understand "three". If every modality were programmed with feature-extractors that counted up the number of objects in every grouping, and output the result as (say) the tag "number: three", the AI might still fail to really understand "three", since such an AI would be unable to count objects that weren't represented directly in some modality. An AI that learns the concept of "three" is more likely to notice not just three apples but that it (the AI) is currently thinking three thoughts. A preprogrammed concept only notices what the programmer was thinking about when ve wrote the program.
What is "three", then? How would the concept of "three" be learned by an AI whose modalities made no direct reference to numbers - whose modalities, in fact, were designed by a programmer who wasn't thinking about numbers at the time? How can such a simple concept be decomposed into something even simpler?
There's an AI called "Copycat", written by Melanie Mitchell and conceived by Douglas R. Hofstadter, that tries to solve analogy problems in the microdomain of letter-strings. If you tell Copycat: "'abc' goes to 'abd'; what does 'bcd' go to?", it will answer "'bce'". And it can handle much harder problems, too. (34). Copycat is a really fascinating AI, and you can read about it in Metamagical Themas, or read the source code (it's a good read, and available as plain text online - no decompression required). If you do look at the source code, or even just browse the list of filenames, you'll see the names of some very fundamental cognitive entities. There are "bonds", "groups", and "correspondences". There are "descriptors" (and "distinguishing descriptors") and "mappings", and all sorts of interesting things.
Without going too far into the details of Copycat, I believe that some of the relations, or mental constructions, are primitive enough to lie very close to the foundations of cognition. Copycat measures numbers directly, although it can only count up to five, but that's not the feature we're interested in. Copycat was designed to understand relations and invent analogies. It can notice when two letters occupy "the same position" in a letter-string, and can also notice when two letters occupy "the same role" in a higher-order mental construct. It can notice that "c" in "abc" and "d" in "abd" and "d" in "bcd" all occupy the same position. It can understand the concept of "the same role", if faced by an analogy problem which forces it to do so - for example, if "abc" goes to "abd", what does "pqrs" go to? Copycat sees that "c" and "s" occupy the same role, even though they no longer occupy the same numerical position in the string, and so replies "pqrt".
Relations such as correspondence and mapping are probably basic cognitive functions (as well as concepts). Correspondences and roles and mappings are probably autonomically-detected features on the modality-level (as well as being very advanced concepts in cognitive science). Intuitive, directly perceived correspondences are what allow two images in the same modality to be compared, and that is a basic part of what makes a modality go.
These intuitions obey certain underlying cognitive pressures (also modeled by the Copycat project): If two high-level structures are equal, then the low-level structures should be mapped to each other. Symmetry, which - very loosely defined - is the idea that each of these low-level mappings should be the same. If one is reflected, they should all be reflected, and so on. Completeness: You shouldn't map five elements to each other but leave the sixth elements dangling.
Copycat shows an example of how to implement this class of cognitive intuitions using conflict-detectors, equality-detectors, and a feature called a "computational temperature". Roughly speaking, conflicts raise the temperature and good structures lower the temperature. The higher the temperature, the more easily cognitive perceptions break - the more easily groups and bonds and mappings dissolve. Lower temperatures indicate better answers, and thus answers are more persistent - perceived pieces of the answer in the cognitive workspace are harder to break. Copycat's intuitions may not have the same flexibility or insight as a human consciously trying to solve a "symmetry problem" or a "completeness problem", but they do arguably come up to approximately the level of a human's unconscious intuitions about analogy problems. Each low-level built-in cognitive ability has its analogue as a high-level thought-based skill, and it is dangerous to confuse the standards to which the two are held.
We now return to the concept of "three". We'll suppose for the moment that we're operating in a Newtonian billiard-ball modality, and that we want the AI to learn to recognize three billiard balls.
The basic concept for "three" looks like this:
The mental image on the left is a prototype, attached to the concept and stored in memory. The mental image on the right is the target, containing the objects actually being counted. The concept of "three" is satisfied when correspondences can be drawn between each object in the three-prototype and each object in the target image. If the target image contains two objects, a dangling object will be detected in the three-prototype image, and the concept will not be satisfied. If the target image contains four objects, then a dangling object will be detected in the target image. (35).
This isn't a full answer to the "problem of three", of course. A full answer would also consider the question of how to computationally implement a "unique correspondence" in a non-fragile way; how to distinguish each object from the background; how to apply the concept to a mental image with two or four objects to yield a mental image with three objects; how to retrieve the prototype from memory; how to extend the intuition of "correspondence" across modalities; the type of mindstuff needed to implement these instructions in a non-fragile way; and how the prototype and concept were created or learned in the first place.
In fact, the problem of three is so complicated that it would probably be first solved by conscious thought, and compiled into a concept afterwards. This adds the problem of figuring out how the thoughts got started; what types of task would force a mind to notice "three" and evolve a definition like that above; and how the skill gets compiled into a pattern. Also, an understanding of three that extends from "three billiard balls" to "three groups of three billiard balls" means asking what kind of problem would force the generalization, and how the generalization would take place inside the thought-based skill or mindstuff-based concept. And then there are questions about moving towards the adult-human understanding of "three", such as noticing that it doesn't matter which particular correspondences are chosen.
However, the diagram above does constitute a major leap forward in solving the problem. It is a functional decomposition of three, one that invokes more basic forces such as unique correspondence and prototype retrieval. It is a concept that could be learned even by an AI whose programmer had never heard of numbers, or wasn't thinking about numbers at the time. It is a concept that can mutate in useful ways. By relaxing the requirement of no dangling objects in the prototype, we get "less than or equal to three". By relaxing the requirement of no dangling objects in the target image, we get "greater than or equal to three". By requiring a dangling object in the target image, we get "more than three". By comparing two images, instead of a prototype and an image, we get "same number as" (36), and from there "less than" or "less than or equal to".
In fact, examining some of these mutations suggests a real-world path to threeness. Often, concepts don't get invented until they're useful. Many physical tasks in our world require equal numbers of something; four pegs for four holes, and so on. The problem of perceiving a particular number of "holes" and selecting, in advance, the correct number of pegs, might force the AI to develop the concept of corresponding sets, or sets that contain the same number of objects. The spatial fact that two pegs can't go in the same hole, and that one peg can't go in two holes, would be a force acting to create unique (one-to-one) correspondences. "Corresponding-sets" would probably be the first concept formed. After that, if it were useful to do so, would come a tendency to categorize sets into classes of corresponding sets, when it was useful to do so; after that would come the selection of a three-prototype and the concept of three.
The decomposition of three in the above graphic is not the most efficient concept for three. It is simply the most easily evolved. After the formation of the prototype-and-comparision concept for three would come a more efficient procedure: Counting. To evolve the counting concept requires that the counting skill be developed, which occurs on the thought-level, which thoughts in turn require a more sophisticated concept-level depiction of three. It requires that one and two have also been developed, and that one and two and three have been generalized into number. Once this occurs, and the AI has been playing around with numbers for a while, it may notice that any group of three objects contains a group of two objects. It may manage to form the concept of one-more-than, an insight that would probably be triggered by watching the number of a group change as additional objects are added. It might even notice that physical processes which add one object at a time always result in the same sequence of numerical descriptions: "One, two, three, four..."
If multiple experiences of such physical processes can be generalized, and a prototypical process selected and applied, the result might be a counting procedure like that taught to human children: Tag an object as counted and say the word 'one'; tag another object as counted and say 'two'; tag another object as counted and say the word that, in the learned auditory chanting sequence, comes after 'two'; and so on. Do not count any object that has already been tagged as counted. The last word said aloud is the number of the group. This method is more efficient than checking correspondences, and also reflects a deeper understanding of numbers.
Finally, once "three" has been used long enough, it's likely that a human brain evolves some type of neural substrate for seeing threeness directly. That is, some piece of the human visual modality - probably the object-recognition system in the temporal lobe, but that's just a guess - learns to respond to groups of three objects. (Larger numbers like "five" or "six" are harder to recognize directly - that is, without counting - unless they come in stereotypical five-patterns and six-patterns like those on the sides of dice.) The analogue for an AI might be a piece of code (or assembly language, or a neural net - you know, mindstuff) that counts items directly.
However, even if the mind eventually creates a highly-optimized counting method, implemented directly, the previous definitions of the concept will still exist. When new situations are encountered that force the extension of the concept, the mind can switch from the optimized method to the methods that reflect underlying causes and underlying substrate. If necessary, the problem can rise all the way to the level of conscious perception, so that the deliberate, thought-level methods - the ones that originally got turned into the concept - are used. The experiences that underlie the original definition, the experience of noticing the definition, the experience of using the definition - all can be reviewed. This is why a concept is so much richer, so much more powerful, if it's learned instead of preprogrammed. It's why learned, rich concepts are so much more flexible, so much likelier to mutate and evolve and spin off interesting specializations and generalizations and variations. It's why learned concepts are more useful in special cases and high-level reasoning. It's why high-level cognitive objects are vastly more powerful, more real, than the flat, naked predicates of classical AI.
This also casts the idea of "information-loss" or "focus" in a different light. Sure, calling something a three-group, or placing it into the three-category, can be said to "lose" a lot of information - in information-theoretical terms, you've moved from specifying the distinct and individual object to specifying a member of the class of things that can be described by "three". In classical-AI terms, you've decided to focus on the feature called "number" and not any of the other features of the object. But both perspectives are wrong. To label a rich, complex, multi-step act of perception "information loss" borders on perversion. Seeing the "threeness" of a group doesn't destroy information, it adds information. One perceives everything that was previously known about the object, and its threeness as well. Nor could that threeness be focused on until the methods for perceiving threeness were learned.
"When you hear the phrase "triangular light bulb", you visualize a triangular light bulb... How do these two symbols combine? You know that light bulbs are fragile; you have a built-in comprehension of real-world physics - sometimes called "naive" physics - that enables you to understand fragility. You understand that the bulb and the filament are made of different materials; you can somehow attribute non-visual properties to pieces of the three-dimensional shape hanging in your visual cortex. If you try to design a triangular light bulb, you'll design a flourescent triangular loop, or a pyramid-shaped incandescent bulb; in either case, unlike the default visualization of "triangle", the result will not have sharp edges. You know that sharp edges, on glass, will cut the hand that holds it."How do the concepts of "triangular" and "light-bulb" combine? My current hypothesis involves what might be called "reductionist energy minimization" or "holistic network relaxation", a conflict-resolution method that takes cues from both the "potential energy surface" of chemistry and the "computational temperature" of Copycat.
-- 1.2: Thinking About AI
Neural networks, when perturbed, are known to seek out what might be called "minimal-energy states". A network-relaxation model of concept combination could conceivably be computationally realistic as an operation that neurons can accomplish in the 200 ops/second timescale. My current hypothesis for the basic neural operation in concept-combination is the resonance. A neural resonance circuit - not a physical, synaptic circuit, but a virtual message-passing circuit, established by one of the higher-level neural communications methods (binding by neural synchrony, maybe) - can either resonate positively, reinforcing that part of the concept-combination, or resonate negatively, generating a conflict. My guess at the network-relaxation method resembles the "potential energy surface" of chemistry in that multiple, superposed alternatives are tried out simultaneously.
The high-level, salient facets of the concepts being combined are combined first. These high-level features then visualize the mid-level features; if no conflict is detected, the mid-level features visualize the low-level features. If a conflict is detected at any level, the conflict propagates back up to the conflicting high-level or mid-level features causing the problem. The more salient, more important, or more useful feature - remember, we're talking about two concepts, each with its own set of features along various dimensions - is selected as dominant, and the network relaxation algorithm proceeds. When one concept modifies another, the "more salient" feature is the one specified by the concept doing the modifying. (Note also that, in casual reading, not all the facets of a concept may be important. Only the facets that resonate with the paragraph, with the subject of discussion, will be visualized.)
In the case of "triangular light bulbs", "triangular" is an adjective. The concept for "triangle" or "triangular" is modifying the concept of "light bulb", rather than vice versa. The default prototype for "light bulb" - that is, an image of the generic light bulb - is loaded into the mental workspace, including the visual facet of the prototype being loaded into the visual cortex. Next, the concept for "triangular" is applied to this mental image.
The concept of "triangular", as it refers to physical objects, has a single facet: It alters the physical shape of the target image. Note that I say "physical shape", not "visual shape". The default prototype for "light bulb" is a mental image of a three-dimensional bulb-shaped object, made of glass, having a metal plug at the bottom, whose purpose is to emit light. It is this mental image that "triangular" modifies, not just the visual component of the image. In particular, the "shape" facet of the light-bulb concept, which is being modified, is a high-level feature describing the shape of the three-dimensional physical object, not the shape of the visual image. Thus, modifying light-bulb shape will modify the mental image of the physical shape, rather than manipulating the 2-D visual shape in the visual cortex.
The "triangular" concept, when applied along the dimension of "shape", manipulates the mental image of the light bulb, changing the 3D model to be triangle-shaped. However, since the image of a flat light bulb fails to resonate, "triangle" automatically slips to "pyramid". I'm not sure whether this conflict is detected at the mid-level feature of "flat light bulb", or whether a flat light bulb actually begins to visualize before the conflict is detected. The slippage happens too fast for me to be sure. I suspect that "triangular" has slipped to "pyramidal" before, when applied to three-dimensional mental images; for neural entities, anything that happens once is likely to happen again. Neurons learn, and neural thinking wears channels in the neurons. It could be that the non-flatness of light bulbs is salient because of their bulbous shape, and that this resonance with non-flatness causes "triangular" to slip to "pyramidal" before the concept is even applied.
Pyramids are sharp. I do know, from introspection, that the "sharp pyramidal light-bulb" got all the way down to the visual level before the conflict was noticed. (The conflict rose to the level of conscious perception, but was resolved more or less intuitively; I didn't have to "stop and think". So this is probably still a valid example of concept-level processes.) The particular conflict: Sharp glass cuts the person who holds it. We've all had visual experience of sharp glass, and the associated need for visual recognition and avoidance; thus, the mental image of sharp glass would trigger this recognition and create a conflict. This conflict, once detected, was also visualized all the way down to the visual cortex; I briefly saw the mental image of a thumb sliding along the edge of the pyramid.
The problem of sharp edges is one that is caused by sharpness and can be solved by rounding, and I've had visual experience of glass with rounded edges, so the sharp edges on the mental image slipped to rounded edges. The result was a complete mental image of a pyramidal light bulb, having four triangular sides, rounded edges and corners, and a square bottom with a plug in it. (37)
Every sentence in the last five paragraphs, of course, is just begging the question: "Why? Why? Why?" A full answer is really beyond the scope of the section on "Mind"; I just want to remind my readers that often the real answer is "Because it happened that way at least once before in your lifetime." A human mind is not necessarily capable of simultaneously inventing all the reflexes, salient pathways, and slippages necessary to visualize a triangular lightbulb. Neurons learn, and thoughts wear channels in the network. The first time I ever had to decide whether triangle-imposition applied to visual, spatial, or physical images, I may have made a comical mistake. A seed AI may be able to avoid or shorten this period of infancy by using deliberate, thought-level reasoning about how concepts should combine; if so, however, this is functionality over and above that exhibited by humans.
You'll note that, throughout the entire discussion of concept combination, I've been talking about humans and even making appeals to specific properties of neurally based mindstuff, without talking about the problem of implementation in AIs. Most of the time, the associational, similarity-based architecture of biological neural structures is a terrible inconvenience. Human evolution always works with neural structures - no other type of computational substrate is available to evolution - but some computational tasks are so ill-suited to the architecture that one must turn incredible hoops to encode them neurally. This is why I tend to be instinctively suspicious of someone who says, "Let's solve this problem with a neural net!" When the human mind comes up with a solution, it tends to phrase it as code, not a neural network. "If you really understood the problem," I think to myself, "you wouldn't be using neural nets."
Concept combination is one of the few places where neurons really shine. It's one of the very rare occasions when the associational, similarity-based, channel-wearing architecture of biological neural structures is so appropriate that a programmer might reinvent naked neurons, with no features added or removed, as the correct computational elements for solving the problem. Neural structures are just very well-suited to "reductionist energy minimization" or "holistic network relaxation" or whatever you want to call it.
Even so, neural networks are very hard to understand and debug, or alter on the design level. I believe in the ideal of mindstuff that both human programmers and the AI can understand and manipulate. To expect direct human readability may be a little too much; that goal, if taken literally, tends to promote fragile, crystalline, simplistic code, like a classical AI. Still, even if concept-level mindstuff doesn't have the direct semantics of code, we can expect better than the naked incomprehensibility of assembly language. We can expect the programmer to be able to see and manipulate what's going on, at least in general terms, perhaps with the aid of some type of "decompiler". I currently tend to lean towards code for the final mindstuff, while acknowledging that this code may tend to organize itself in neural-like patterns which will require additional tools to decode.
Thoughts are created by structures of concept-level patterns. The archetypal example is a grammatical sentence: a linear sequence of words parsed by the brain's linguistic centers into a more-or-less hierarchical structure, in which the referents of targetable words and phrases (adjectives, for example) have been found, either inside the sentence or in the most salient part of the current mental image. The inverse of this process is when a fact is noticed, turned into a concept structure, translated into a sentence, and articulated out loud within the mind. (A possible reason for the stream-of-consciousness phenomenon is discussed in 2.4.3: Thoughts about thoughts.)
The current section has discussed concepts as mindstuff-based patterns in sensory modalities - that is, the mindstuff is assumed to pay attention to, or issue instructions to, the sensory modalities and the features therein. That concepts interact with other concepts, and are influenced by the higher-level context in which they are invoked, has been largely ignored. This was deliberate. The farther you go from the mindstuff level, and the more "abstract" you get, the closer you are to the levels that are easily accessible to human introspection. These are the introspective perceptions that come out in words; the qualities that modern culture associates with above-average intelligence; the levels enormously overemphasized by classical AI.
Still, there are some thoughts that are so abstract as to appear distant from any sensory grounding. In that last sentence, for example, only the term "distant" has an obvious grounding, and since the sentence wasn't interpreted in a spatial context, it's unlikely that even that term had any direct visualizational effect. Metaphors do show up more often than you might think, even in abstract thought (see George Lakoff and Mark Johnson, Metaphors We Live By or Philosophy in the Flesh). Still, there are concepts whose definition and grounding lies primarily in their effect on other concepts. Why doesn't the classical-AI method work for them?
Even abstract concepts, mental images composed entirely of concepts referring to othe concepts, exist within a holistic system. If abstract concepts don't have reductionist definitions that ground directly in sensory experience, they have reductionist definitions that ground in other concepts. What are apparently high-level object-to-object interactions between two abstract concepts can, if conflicts appear, be modeled as mid-level structure-to-structure interactions between two definitions. Abstract concepts still have lower-level structure, mid-level interactions, and higher-level context.
Still, defining concepts in terms of other concepts is what classical AIs do. I can't actually recall, offhand, any (failed!) classical AIs with explicit holistic structure - or rather, any classical AIs that used explicitly multilevel model-construction methods to reason using semantic networks - but it seems likely that someone would have tried it at some point. (Eurisko and Copycat don't count for reasons that will be discussed in future sections. Besides, they didn't fail.) So, what else is missing?
Many classical AIs lack even basic quantitative interactions (such as fuzzy logic), rendering them incapable of using methods such as holistic network relaxation, and lending all interactions an even more crystalline feeling. Still, there are classical AIs that use fuzzy logic.
What's missing is flexibility, mutability, and above all richness; what's missing is the complexity that comes from learning a concept. Perhaps it would be theoretically possible to select a piece of abstract reasoning in an adult AI in which the complexity of sensory modalities played no part at all; perhaps it would even be possible to remove all the grounding concepts below a certain level and most of the modality-level complexity without destroying the causal process of the reasoning. Even so, even if the mind were deprived of its ultimate grounding and left floating, the result wouldn't be a classical AI. Abstract concepts are learned and grown in a world that's almost as rich as a sensory modality because the grounding definitions are composed of slightly less abstract concepts with rich interactions, and those are rich because they grew up in a rich world composed of interactions between less abstract concepts, and so on, until you reach the level of sensory modalities. Richness isn't automatic. Once a concept is created, you have to play around with it for a while before it's rich enough to support another layer. You certainly can't start from the top and build down.
Another factor that's missing from classical AIs is the ability to attach experience to concepts, to gain experience in thinking, to wear a channel in the mind. Even a concept-combination like "triangular light bulb" has a dynamic pattern, a flow of cause and effect on the cognition level, that relies on having done most of it before. That complexity is also absent from classical AIs. (And of course, most classical AIs just don't support the dimensions of cognition - attention, focus, causality, goals, subjunctivity, et cetera.)
I think this provides an adequate explanation of why classical AIs can't support thought-level reasoning or a stream of consciousness; why sensory modalities are necessary to learn abstract thought; and why concepts must be learned in order to be rich enough to support coherent thought.
Rational reasoning is very large, and very complicated. In trying to duplicate the functionality of a line of rational reasoning, it's very easy to bite off too much, and despair, or oversimplify. The remedy is an understanding of precedence, a sequence that tells you when you're getting ahead of yourself and building the roof before you've laid the foundations; °heuristics that tell you when to slow down and build the tools to build the tools. Before you can create a thing, there must be the potential for that thing to exist, and sometimes you have to recurse on creating the potential.
Drew McDermott, in the classic article "Artificial Intelligence Meets Natural Stupidity", pointed out that the first task, in AI, is to get the AI to notice its subject. Not "understand". Notice. If a classical AI has a LISP token named "hamburger", that doesn't mean the token is a symbol, or that there's any hamburgerness about it. For an AI to notice something, its internal behavior must change because of what is noticed. A LISP token named "hamburger" has no attached hamburgerness. A philosopher of classical AI would say that the LISP token has semantics because it refers to hamburgers in external reality, but the AI has no way of noticing this alleged reference. The "reference" does not influence the AI's behavior - neither external behavior, nor the internal flow of program causality.
I've extended McDermott's heuristic to describe a sequence called RNUI, which stands for Represent, Notice, Understand, and Invent. Represent comes before Notice; before you can write feature-detectors in a modality, you need data structures (or non-crystalline equivalents thereof) for the data being examined and the features being perceived. Understand comes before Invent; before an AI can design a good bicycle, it needs to be able to tell good bicycles from bad bicyles - perceive the structure of goals and subgoals, understand a human designer's explanation of why a bicycle was designed a particular way, be capable of Representing the explanation and Noticing the difference between explanations and random babbling - before it can independently invent a bicycle and explain it to someone else.
Represent is when the skeleton of a cognitive structure, or the input and output of a function, or a flat description of a real thought, can be represented within the AI. Represent is about static data, what remains after dynamic aspects and behaviors have been subtracted. Represent can't tell the difference between data constituting a thought, and data that was provided by a random-number generator.
Notice provides the behaviors that enforce internal relations and internal coherence. Notice adds the dynamic aspect to the data. Applied to the modality-level, Notice describes the feature-extractors that annotate the data with simple facts about relations, simple bits of causal links, obvious similarities, temporal progressions, small predictions, and other features created by the "laws of physics" of that domain. The converse of modality-level Notice perception is Notice manipulation, the availability of choices and actions that manipulate the cognitive representations in direct ways. This sequence also applies to higher levels, and to the AI as a whole; it's possible to be capable of Representing and Noticing threeness without Understanding it, or being able to do anything useful with it.
Understand is about intentionality and external relations. Understand is about coherence with respect to other cognitive structures, and coherence with respect to both context and substance (the upper and lower levels of the holistic representation). Understanding means knowledge and behaviors that reflect the goal-oriented aspects of a cognitive structure, and the purpose of a design feature. Understanding reflects the use of heuristics that can bind high-level characteristics to low-level characteristics, and distinguish a good design from a bad one. Understanding is the ability to fully represent the cognitive structures that would be created in the course of designing a bicycle or inventing an explanation, and to verify that these cognitive structures represent a good design or a good explanation.
Invent is the ability to design a bicycle, to invent a heuristic, to analyze a phenomenon, to create a plan for a chess game - in short, to think.
If you have trouble getting an AI to design a bicycle, ask yourself: "Could this AI understand a design for a bicycle if it had one? Could it tell a good design for a bad design?" If you have trouble getting an AI to understand the design for a bicycle, ask yourself: "Can this AI notice the pieces of a bicycle? Could it tell the difference between a bicycle and random static?" If you have trouble getting an AI to notice the pieces, ask yourself: "Can this AI represent the pieces of the bicycle? Can it represent what is being noticed about them?"
| NOTE: | This section is about what thoughts do. For an explanation of what thoughts are - how they work, where they come from, and so on - see the previous sections. |
Before the AI can act, it needs to learn. "Learning" can be divided into knowledge-formation and skill-formation. Skill formation happens when mindstuff, reflexes, or other unconscious processes are modified. In humans, the modification is autonomic; in seed AIs, it can be either autonomic or deliberate; but skills are always executed autonomically. (Note that "skill", as used here, includes not only motor reflexes but cognitive reflexes, and that "skill" does not include conscious skills like knowing (in theory!) how to disassemble a motorcycle.) The usual term for the dichotomy between skill and knowledge is "procedural vs. declarative", although this involves an assumption about the underlying representation that isn't necessarily true. In general, "knowledge" is the world-model, the contents of the mind, and "skill" is the stuff the mind is made of. Because skills tend to be located at the concept-level or modality-level, this section focuses on knowledge.
The world-model is holistic or reductionist, depending on whether you're looking up or looking down. We live in a Universe where complex objects are built from simpler structures, and stochastic regularities in the interactions between simple elements become complex elements that can develop their own interactions.
Thus, broadly speaking, there are at least three kinds of knowledge problems. You can look for a regularity in the way an object interacts with another object. You can take an object, an event, or an interaction, and try to analyze it; explain how the visible complexity is embodied in the constituent elements and their interactions. Or you can take elements and interactions that you already know something about, and try to understand the high-level behavior of the system. Starting from what you know, you can look sideways, down, or up.
Actually, this is speaking too broadly. Where, for example, do you fit "taking an object that you know something about, and suddenly understanding its purpose within a higher system"? I suppose you could explain this as a variant of analysis - when the "Aha!" is done, the result is a better understanding of a system in terms of its constituents. But then there are other knowledge problems, like guessing the properties of an element by taking the intentional stance towards the system and assuming the object is well-designed for its purpose. Where does that fit in? The moral, I suppose, is that "reductholism" has its uses as a paradigm, but there are limits.
Maybe we should generalize to generic causal models, regardless of level? Then you could divide activities into noticing a property or interaction, deducing the cause of a property or interaction, or projecting from known causes to the expected results. This model is a little more useful, since it sounds like the three problem types may correspond to three problem-solving methods: (A) Examine the model for unexpected regularities, correspondences, covariances, and so on. (B) Generate and test possible models to explain an effect. (C) Use existing knowledge to fill in the blanks (and, if you're a scientific mind, test the predictions thus created).
Still, even that view has its limitations. For example, asking Why? or looking for an explanation isn't strictly a matter of generate-and-test. In fact, generate-and-test is simply a genteel, thought-level version of that old bugaboo of AI, the search algorithm. It seems likely that some type of "genteel search algorithm" - not "blind", but not really deliberate either, and with a definite random component - is responsible for sudden insights and intuitive leaps and a lot of the go-juice of intelligence on the concept level. On the thought level, however, it's often more efficient to take a step back and think about the problem. One implementation for thinking about the problem is "abstraction is information-loss" classical-AI-type "abstract thought", running the problem through with Unknown Variables substituted in for everything you don't know, to see if there are places where the Unknowns cancel out to yield partial results that would hold true of every possible solution, thus constraining the search space. A more accurate implementation would be "applying heuristics that operate on the general information you have, to build up general information about the answer".
The thought-level is a genuine layer of the mind. There isn't any simple way to characterize it. There's a complex way to characterize it, which would consist of watching people solve problems while thinking out loud ("protocol analysis"), then figuring out a set of generalizations that corresponded to underlying neurology or underlying functional modules of the problem-solving method, and which categorized all the individual thoughts in the experimental observations. This problem is large, but finite; the set of underlying abilities and mental actions is limited. Still, such a project is beyond the scope of this particular section. (What I will attempt to do, in later topics, is describe enough of the underlying abilities - enough that implementing them would give rise to sustainable thought. Remember, seed AI isn't about perfectly describing the complete functionality of humans, it's about building minds with sufficient functionality to work.)
The thought-level is a genuine layer of the mind, and has around the same amount of internal complexity as might be associated with the modality-level or the concept-level. The difference is that thoughts are open to introspection, and thus, when I make sweeping generalizations, my readers can catch me at it. Nonetheless, I hope that the generalizations that have been offered here are sufficient to convey a vague general image of what goes on in a mind searching for knowledge. Noticing interesting coincidences and covariances and similarities (looking sideways), building and testing and thinking about the reason why something happens (analysis, looking down in the holistic model, looking backwards in the causal model), trying to fill in the blanks from the knowledge you already have (prediction, looking up in the holistic model, looking forwards in the causal model). The goal is a holistic model with good high-level/low-level bindings, or a causal model where the consequences and preconditions of a perturbation are well-understood, or a goal-and-subgoal model with plans and convergences and intentionality. The goal is a model that holds together, on all levels, when you think about changing it; a model rich enough to support what we think of as intelligent thought.
It is literally impossible to draw a sharp line between understanding and creativity. Sometimes the solution to a difficult knowledge question must be invented, almost ab initio. Sometimes the creation of a new entity is not a matter of searching through possibilities but of seeing the one possibility by looking deeper into the information that you already have. But, usually, when building the world-model, you're trying to find a single, unique solution; the answer to the question. When trying to design something new, you're looking for anyanswer to the question. Understanding is more strongly constrained, but this actually makes the problem easier, since a solution exists and the problem is finding it... the constraints might rather be called clues.
In invention, each constraint eliminates options and makes it less likely that a solution exists. The distinction between understanding and invention is something like the difference between P and NP, between verifying a solution and finding it. Returning to the quadrivium of Sensory, Predictive, Decisive, and Manipulative binding, and to Manipulation's sub-trinity of qualitative, quantitative, and structural bindings, then invention, or high-level manipulation, adds a fourth binding, the holic binding. It's the ability to take a desired high-level characteristic and specify the low-level structure that creates it. It's the ability to engage in hierarchical design, to start from the goal of rapid travel and move to a complete physical design for a bicycle.
The methods of invention are even less clear-cut than the methods of understanding. Unless the problem is one of qualitative manipulation (choice from among a limited number of alternatives), the design space is essentially infinite. An intelligent mind reduces the effective search space through possession of a holistic model that ultimately grounds in heuristics capable of direct backwards manipulation. In other words, if you can choose any real number to specify the width of the wheel, what's needed is a heuristic that binds it - reversibly - to a higher-level design feature, such as desired stability on turns. If desired stability on turns is itself a design variable, a heuristic is needed that binds it to a known quantity, such as the weight range of the rider. And so on.
Such reasoning acts to reduce the search space from the space of all possible low-level specifications of a design, to the space of cognitive objects constituting reasonable high-level designs. If there are enough heuristics left to constrain the design further, or to specify design features from high-level goals, then the task can be completed without special inspiration. If there's a gap, a high-level feature with no heuristics that directly determine how it might be implemented, then there sometimes comes that special event known as an "insight", an intuitive leap.
Sometimes you try to invent the bicycle without knowing about the wheel. The crucial insight may consist of remembering logs rolling down a hill. It may consist of just suddenly seeing the answer. Or it may lie in finding the right heuristic to attack the problem. The key point is that a wide search space is crossed to find the single right answer, apparently without any guide or heuristic that simplifies the problem. (If the aha! is finding the right heuristic, then the act of creativity lies in crossing the search space of possible heuristics.)
What is creativity? Creativity is the name we assign to the mental shock that occurs when a large and novel load of high-quality mental material is delivered to our perceptions. I would say that it's the perception of "unexpected" material, meaning "unexpected" not in the sense that the delivery comes as a surprise, but in the sense that our mental model can't predict the specific content of the material being delivered. We perceive a thought as "creative", in ourselves or others, on one of two occasions: First, seeing someone thinking outside the box; second, on perceiving a single good solution selected from a nearly infinite search space. In the first case, a concept is redefined, or what was thought to be a constraint is broken; the answer is unexpected, which creates - to the viewer - the mental shock that we name "creativity". The second case consists of seeing the very large gap between "high-speed travel" and "bicycle" crossed; the viewer - unless ve verself has designed a bicycle - has no single heuristic that can cross a gap of that size, that can anticipate the content of the material presented. There's a nearly infinite space of possible paintings, so when we see any single painting of reasonable quality, a large quantity of unexpected cognitive material is delivered to our eyes and we call it "creativity".
It seems likely to me that the experience of creative insight happens when the mind decides to brute-force, or rather intelligent-force, the search problem. The aha! of wheels comes because, somewhere in the back of your mind, possible memories were tested at random for applicability to the problem until the memory of logs rolling down a hill resonated with the problem and rose to conscious attention. This unconscious "blind" search may employ some of the tricks of deliberation, such as searching through memories of objects that were seen traveling very fast. (Or not. It seems likely to me that only deliberate thought produces that kind of constraint.) Even so, it remains in essence a try-at-random algorithm. If there's anything more to subconscious creative insights than that, I don't know what it is.
Since thoughts are reasonably accessible to the human mind, there's a good deal of existing research on how they work. The specific methods are important, but what's more important is getting a working system of thoughts, enough methods that work well enough that the AI can continue further.
Most important to the system of thoughts is introspection. Introspection is the glue that holds the thought-level together. Coherent thoughts don't happen at random. They happen because we know how to think, and because we have the right reflexes for thinking. The problem of what to think next is itself a problem domain. To prevent an infinite-recursion error, our solution to this problem on the moment-to-moment level is dictated entirely by reflex, the channels worn into our neural minds. Even when we deliberately stop and say to ourselves, "Now, what topic should I think about next?", the thinking about thinking proceeds by reflex. These reflexes are formed during infancy, and before they exist, coherent thought doesn't happen. To get past that barrier you'd have to be a seed AI, capable of watching a replay of your own source code in action, or halting and storing the current state of high-level thought to recurse on examining the stuff the thought is made of.
The self is a domain fully as complex as any in external reality. It consists not just of perceiving the self but of manipulating the self. The experience you remember of introspection consists of the occasions when the problems became large enough to require conscious thought. Beneath that remembered, introspection-accessible experience lies perceptions and reflexes that have become so invisible we don't even notice them. The intuitions of introspection are far more basic to thought than Hamlet's soliloquy. The problem of introspection should be approached with the same respect, and the same attention to the RNUI method, that would be given to the problem of designing a bicycle.
Introspection requires introspective senses, perhaps even an introspective modality. But the idea of an introspective modality is a subtle and perhaps useless one. The obvious implementation is to have an introspective modality that reports on all the cognitive elements inside the AI, but what does this add? The AI has already noticed that the cognitive elements are there. How does "the introspective modality" differ from "a useless and static additional copy of all the information inside the AI"? What can you do with the detected feature of "the feature of redness" that you can't do with the feature of redness itself?
To answer this question, it is necessary to step back and consider the problem in context. Sensory modalities don't exist in a vacuum. They are useful because concepts lie on top. The question, then, is not how to build an introspective sensory modality, but how to insure that concepts about introspection can form. This may involve creating a new introspective modality, or it may involve attaching a new dimension to the old modalities and to the other modules of cognition.
Concepts manipulate their referents, as well as extracting information from them. How would you go about tweaking the visual modality so that you could imagine "thinking about redness"? How do you get the AI to notice, declaratively, that a concept has been activated, and how is this perception reversed to give rise to visualizing the consequences of activating a concept?
This design problem may go a bit towards explaining that peculiar phenomenon called "stream of consciousness". You notice a fact, the fact gets turned into a conceptual structure, the conceptual structure gets turned into a sentence by your language centers, and then you speak the sentence "out loud" within your mind. The fascinating thing is this: If you try to skip the step of "speaking the sentence out loud" within your mind, even after you know exactly what the words will be, you can't go on thinking. Why? What new information is added by this act?
One possible explanation is that the human mind notices concepts by noticing the auditory cortex. Humans have no built-in introspective modality, so concepts become "visible" to our mental reflexes when they add recognizable content - words - to the auditory cortex. This closes the loop. Concept activation becomes detectable, and we can form concepts about concepts. I don't think this is the entire explanation, but it's a good start.
What about thoughts? On the thought-level, human introspection is fairly primitive. There's this tendency to lump everything together under the term "I". When we attribute causality, we say "I remembered" instead of "the long-term memory-retrieval subsystem reports..." Perhaps this is because, historically speaking, we didn't know anything about what was inside the mind until yesterday afternoon. Perhaps it's because fine-grained introspection doesn't contribute useful complexity to self-modeling unless you're, oh, writing a paper on AI or something. There's plenty of useful heuristics about the self that can be learned by looking at cause and effect, even when all the causal chains start at a monolithic self-object. A seed AI may have uses for more fine-grained self-models, but with both design and source code freely accessible, it shouldn't be too hard for such a self-model to develop.
When can an AI legitimately use the word "I"?
Understand that we are asking about a very limited and purely technical aspect of self-awareness. We are not talking about the kind of self-awareness that will cause an ethical system to treat you as a person. We are not talking about "qualia", the hard problem of conscious experience, what it means to be a bat, or anything of that sort. These are different puzzles.
The question being asked is: When can an AI legitimately use the word "I" in a sentence, such as "I want ice cream", without Drew McDermott popping up and accusing us of using a word that might as well be translated as "shmeerp" or G0025?
Consider the SPDM distinction: Sensory, Predictive, Decisive, Manipulative. A binding between a model and reality starts when the model "maps" in some way to reality (although this is ultimately arbitrary), becomes testable when the model can predict experiences, and becomes useful when the model can be used to decide between alternatives, with the acid test being manipulation of reality in quantitative or structural ways. Consider also the distinction between modality-level, concept-level, and thought-level.
Self-modeling begins when the AI - let's call it Aisa, for "AI, self-aware" - starts to notice information about itself. Introspective sensations of sensations are hard to distinguish from the sensations themselves, so this ball doesn't really get rolling until Aisa forms introspective concepts. The self-model doesn't begin to generate novel information, information that can impose a coherent view of internal events, until it can make predictions - for example: "Skipping from topic to topic, instead of spending a lot of time on one topic, will result in conceptual structures that are connected primarily through association." Likewise, this information doesn't become useful until it plays a part in goal-oriented decisions - a decisive binding.
When Aisa can create introspective concepts and formulate thought-level heuristics about the self, it will be able to reason about itself in the same fashion that it reasons about anything else. Aisa will be able to manipulate internal reality in the same way that it manipulates external reality. If Aisa is impressively good at understanding and manipulating motorcycles, it might be equally impressive when it comes to understanding and manipulating Aisa.
But to say that "Aisa understands Aisa" is not the same as saying "Aisa understands itself". Douglas Lenat once said of Cyc that it knows that there is such a thing as Cyc, and it knows that Cyc is a computer, but it doesn't know that it is Cyc. That is the key distinction. A thought-level SPDM binding for the self-model is more than enough to let Aisa legitimately say "Aisa wants ice cream" - to make use of the term "Aisa" materially different from use of the term "shmeerp" or "G0025". There's still one more step required before Aisa can say: "I want ice cream." But what?
Interestingly, assuming the problem is real is enough to solve the problem. If another step is required before Aisa can say "I want ice cream", then there must be a material difference between saying "Aisa wants ice cream" and "I want ice cream". So that's the answer: You can say "I" when the behavior generated by modeling yourself is materially different - because of the self-reference - from the behavior that would be generated by modeling another AI that happened to look like yourself.
This will never happen with any individual thought - not in humans, not in AIs - but iterated versions of Aisa-referential thoughts may begin to exhibit materially different behavior. Any individual thought will always be a case of A modifying B, but if B then goes on to modify A, the system-as-a-whole may exhibit behavior that is fundamentally characteristic of self-awareness. And then Aisa can legitimately say of verself: "I want an ice-cream cone."
Humans also throw a few extras into the pot. We have observer-biased social beliefs, a whole view of the world that's skewed toward the mind at the center, which tends to anchor the perception of the self. We attribute internal causality to a monolithic object called the "self", which generates a lot of perceived self-reference because you don't notice the difference between the thought doing the modifying and the cognitive object being modified - the source of the thought is the "self", and the item being modified is part of the "self".
A seed AI will probably be better off without these features. I mention them because they constitute much of what a human means by "self".
Since this curve folds in on itself, most "reasonable" images of the local curves for intelligence and efficiency, when combined, are likely to result in a breakthrough-and-bottleneck series at the global level. At least, this is what's likely to happen in the prehuman areas of the curve. Once a breakthrough carries the seed AI past the human level, I would expect the nanotechnology-to-Sysop "curve" to take over.
Introspection, like evolutionary reasoning, is an incredibly powerful tool. Like evolutionary reasoning, it takes practice, talent, and self-awareness to use it on a professional level - to reliably distinguish between post facto and "pre facto" reasoning, or between original thought and °cached thought.
Some people, maybe even a majority of readers, may not have needed to visualize the car smashing before deducing that it would break.
If proprioception does have a separate area of cortex (with distinct representations and extractable features), then it's a distinct sensory modality and should be known as such.
Otherwise, it's like suggesting that translating between Microsoft Word and HTML should be programmatically trivial because both files are really just magnetic patterns in the atoms of the hard disk. What matters is the level where they're different - that's where the Law of Pragmatism says the intelligence is. And if they aren't different anywhere - why, then, there's probably no intelligence.
It [the thalamus] has a simple position in the overall architecture; virtually all information arriving at the cerebral cortex comes from the thalamus, which receives it from subcortical structures... In particular, all visual, auditory, tactile, and proprioceptive information passes through the thalamus on its way to cortex...The most popular hypothesis is that these fibers play a gatekeeping role, assisting in focus of attention (why do you need more fibers to do that?); or, more plausibly, top-down constraints in feature extraction. And since this particular statistic is for cats, the latter hypothesis may be mostly correct. Visualization - imagination - is stereotypically associated with minds directed by general intelligence. While cats may need a memory, and thus the ability to reconstruct images from remembered high-level features, they probably don't need the detailed, fine-grained imagination of a human. So I wouldn't be surprised to find an even greater discrepancy in humans!These facts give rise to the classic view that the thalamus is a passive relay station which generates virtually all the information bearing input to the cortex...
BUT the above picture has omitted one fundamental fact: all projections from thalamus to cortex are reciprocated by feedback projections from cortex to thalamus of the same or even larger size. For instance, Sherman and Koch (1986) estimate that in cat there are roughly 10^6 fibers from the lateral geniculate nucleus in the thalamus to the visual cortex, but 10^7 fibers in the reverse direction! (Italics in original.)
Or perhaps, even for cats, more fibers go from cortex to thalamus than vice versa because even mnemonic sensory manipulation is just computationally harder than sensory perception.
Note that Copycat doesn't select the correct answer from a list of alternatives; it actually invents the answer, which is very impressive and astonishingly rare.
