Connectionism
Connectionism today generally refers to an approach in in the fields of cognitive psychology, cognitive science and philosophy of mind which models mental or behavioral phenomena with neural networks, and is associated with a certain set of arguments for why this is a good idea (among others, that connectionist models are more biologically plausible than other, for example purely symbolic representational models).
While symbolic computation (or cognitivism as it is sometimes known) is understood as the rule (or algorithm)-governed manipulation of the explicit symbols (which are usually also explicit representations of some aspects of the world), the connectionism is often considered an alternative approach since:
- whereas symbolic computationalism uses symbolic models that do not resemble underlying brain structure at all, connectionists engage in "low level" modelling, trying to ensure that their models resemble the neurological structures.
- whereas symbolic computationalism generally focuses on the structure of explicit symbols (mental tokens) and sintactical rules for its internal manipulation, the connectionism focuses on learning from the 'experiences' with environment and storing this information in a form of (variously weighted) connections (therefore connectionism) between neurons.
- whereas symbolic computationalism believes that internal mental activity consists (at least at the highest levels) of manipulation of explicite symbols (see Language of thought), connectionists (or anti-representationalists, such as Rodney Brooks's work in artificial life) believe that the mind does not utilize explicit symbols internaly at all simply because the implicitely stored and retreived information perfectly meets all the needs (except the need to economicaly transfere the complex implicit knowledge to another person when symbolic representation is a necessity).
The prevailing form of connectionist models today is known as Parallel Distributed Processing (PDP). PDP form became popular in the 1980s with the release of Parallel Distributed Processing: Explorations in the Microstructure of Cognition - Volume 1 (foundations) & Volume 2 (Psychological and Biological Models), by James L. McClelland, David E. Rumelhart, and the PDP Research Group. PDP's roots are the perceptron theories from the 1950s. In spite of the fact that as early as 1952 Friedrich Hayek posited the idea of spontaneous order in the brain arising out of decentralized networks of simple units (neurons), Hayek's work was never cited in the literature of connectionism.
Another form of connectionist models is the Relational Network framework developed by the linguist Sydney Lamb in the 1960s. Relational Networks have only ever been used by linguists.
All modern connectionist models adhere to two major principles regarding the mind.
- Any given mental state can be described as a n-dimensional vector of numeric activation values over neural units in a network.
- Memory is created by modifying the strength or the architecture of the connections between neural units. The connection strengths, or "weights", in PDP models are generally represented as a N x N dimensional matrix.
An earlier and rather different connectionistic view was held by Edward Thorndike, a turn of the century psychologist who studied learning, with his most famous contributions being work on how cats escaped from puzzle boxes, and his formulation of the Law of Effect. His analysis (and its descendants) are peppered with references to associations between stimuli and responses. Though the S-R aspect has today been abandoned by radical behaviorists and cognitive psychologists (including connectionists), it is easy to impose the notion of association and modification of association strength on connectionist models.
Many sophisticated learning procedures for neural networks have been developed by connectionists. Learning always involves modifying the connection weights. These generally involve mathematical formula to determine the change in weights when given sets of data consisting of activation vectors for some subset of the neural units.
By formalizing learning in such a way connectionists have many tools at their hands. A very common tactic in connectionist learning methods is to incorporate gradient descent over an error surface in a space defined by the weight matrix. All gradient descent learning in connectionist models involves changing each weight by the partial derivative of the error surface with respect to the weight. Backpropagation, first made popular in the 1980s, is probably the most commonly known connectionist gradient descent algorithm today.
Connectionists are generally in agreement that recurrent networks (networks wherein connections of the network can form a directed cycle) are more like the human brain than feedforward networks (networks with no directed cycles). A lot of recurrent connectionist models incorporate dynamical systems theory as well. Many researchers, such as the connectionist Paul Smolensky (one of the authors of the original PDP books), have argued that the direction connectionist models will take is towards fully continuous, high-dimensional, non-linear dynamic systems approaches.