Chapter 3

NEURAL NETWORKS

The brain is of course a neural network made out of biological material. We must examine briefly some of the well known properties of neurons and then we will look at artificial electronic "Neural networks". Neurobiologists will want to skip my simple description of the neuron but please do not skip what I have to say about current down a nerve. Please note - The brain is full of medullated nerves and this might not be simply for speed of conduction and economy of diameter. The distances inside the brain are quite short and conduction times are probably much shorter than synaptic action. While there is no net external current arising from the inward positive charges flowing into the axon and their rapid reversal, there is indeed a current flowing down medullated nerves as a result of the skipping of electrical charge from one node of Ranvier to the next and this is important in that it causes a magnetic field which is able to induce currents in other parts of a neural network.

The typical neuron is represented in Figure #12. It consists of a cell body and a long nervous process called the axon. This has an outer protective covering made of Schwann cells called the neurilemma and typically a myelin sheath made out of lipoid material, which is an insulator and which is supplied by the Schwann cells. The lipoid material is interrupted at intervals of about a millimeter by small gaps called the nodes of Ranvier. Arising from the cell body are a number of shorter processes called dendrites. These branch many times terminating in synapses which make contact with the processes or cell bodies of other cells. There may be as many as 5000 such synapses to each neuron. The dendrites tend to be on the input side and the axon is on the output side of a long cell. The contacts with other nerve cells, the synapses, are small swellings with a narrow space between one cell and another, the synaptic cleft, into which chemicals are secreted whenever the neuron fires. These chemicals are neurotransmitters such as acetyl choline, dopamine, GABA (Gamma-amino benzoic acid), adrenalin etc. Many of these are excitatory but some are inhibitory in their activity. In addition to these chemically active synapses there are some electrical synapses where there is continuity of the cytoplasm between cells. It will be seen later that this could be important for the transmission of currents in a loop, because the speed of transmission is so much greater, depending, as it does, on an electrical rather than a chemical process. The inside of the axon is polarized negatively to about 70 millivolts. Each time a neuron fires a small amount of neuro- transmitter is secreted into all of its synaptic clefts and when sufficient chemical accumulates there, the receiving cell will fire. This happens repeatedly and quite rapidly. Whether or not a cell will fire, depends on the balance of excitatory and inhibitory neurotransmitter substances which the synapse receives. When sufficient chemical accumulates to cause firing, the polarization of the cell wall at that point breaks down and a positive charge flows into the interior. (Figure #13.) This is a local breakdown of the polarization and it spreads to the adjacent part of the axon or dendrite causing it to break down also. A wave of depolarization sweeps all along the dendrite to the cell body with a high velocity and then all over the neuron and its other branches. After that, the electrical polarization is rapidly restored. If the nerve is medullated, the electrical impulse travels faster because the impulse leaps electrically to the next node of Ranvier.

Successive depolarisations can occur extremely rapidly. See Figure #5 (Oscilloscope tracing of nervous activity). The frequency of the discharges determines the strength of the signal. In other words transmission is frequency modulated and is not amplitude modulated.

The system as a whole is not purely digital but it is a mixture of digital and analog. We should not be looking for any complicated system depending upon the temporal interplay of individual digital signals. Firing of the next successive neuron depends on the accumulation of chemicals and the system is essentially analog in electrical terms. Synapses have quite complex properties but it is reasonable to assume that within a certain range the amounts of chemical add and subtract arithmetically and that the first order effect is linear. One could say that the middle part of the transmission function across a synapse is linear. All of this can be imitated quite well by electronics. An artificial model of a nervous network can be constructed using capacitors to store charge and transistors as detectors of voltage level. When a certain voltage level is reached, firing occurs in a circuit which serves as a pulse generator, resulting in the transmission of a pulse just like that in the CNS. This pulse is transmitted to many more pulse generator modules each of which represents a synapse. Artificial neural nets are based on an idealized version of neural function called the McCulloch and Pitts model. Usually the transfer function is taken to be that of a sigmoid shaped curve but it turns out that neural nets will work very well with any one of a variety of transfer functions. The neuron is a very complex structure but it turns out that one can build a neural network with tremendous power assuming a rather limited number of properties. It may well be that these properties are the most important ones for biological systems and that they are responsible for the effects of the first order of magnitude. There is an integrative process at the input side due to the "Cable" effects already described with regard to what happens at a junction in the dendrites. With artificial nets these properties are usually ignored.

Years ago Alan Turing showed that his very basic computer, called the Turing machine, was functionally as capable as any computer (Though painfully slow). So all computers are in a sense equivalent. It is therefore probably not important that neural networks usually predicate a bunch of identical units whereas the CNS uses a great variety. They will ultimately have the same power of solution to a problem. The CNS will win on the score of speed and efficiency because of the variety of the units. One can build an electrical analogue using real hardware but nowadays it is much more usual to construct a neural network as a software program modeling the electrical hardware. It is run on a conventional computer of the Von Neuman type.

THE PERCEPTRON

Modern neural nets derive originally from the "Perceptron" which was invented by Rosenblatt in 1957. It was built in actual hardware. It was improved upon by Kohonen, Minsky and Papert, Hopfield, Grossberg and many others. The original perceptron, Figure #14 (Diagram of the Perceptron) was intended to recognize visual patterns and it was organized in three layers. The first layer was a bank of 400 photo cells essentially representing a retina. These connected in originally random strengths to each one of a second or "Hidden" layer of very many associator units. Those connected in a similar initially random fashion to the output layer. The units functioned on the basis of a simplified McCulloch and Pitts model using electronic units communicating with other cells using a sort of synapse. When signals arriving at a synapse built up to a certain threshold level, the "Cell" would "Fire" sending out a signal to all of its connected cells.

The type of task which the perceptron was intended to accomplish was to recognize a letter of the alphabet. In other words it would be a reading machine. The output display was intended to portray a symbol which showed recognition some particular input pattern. This might, for example be a letter of the alphabet. Hopefully the machine would eventually be able to distinguish between all of the different letter shapes and display the correct symbol at the output. It was necessary to train the network before it could do anything useful. This was done first by presenting an image to the input retina. Then an evaluation of the output had to be made according to some rule. One needed a human to be the "Instructor", who had to make the evaluation. If the output had improved, then the changes were retained or augmented. If the output deteriorated, then the changes were made in the opposite direction. All of this was repeated a few thousand times. Any improvements in the output were retained each time. Eventually the network would achieve the capability to solve the problem regularly, so long as a definitive solution did exist.

Over the years neural networks were greatly improved by Hopfield, Grossberg, and many others. The rules for changing the strengths were improved so that they could be performed automatically by a mechanism known as back propagation (Rumelhart). It is sometimes necessary to have a human mind behind the system of training the neural network but if evaluation can be determined numerically, it is possible to set up automatic instructions. Kohonen did show that neural nets could have some self organizing properties. Gerald Edelman and his associates have had some notable success in their modeling of the visual system. There has been a great improvement in the power of neural networks using conventional computers simulating a neural net. Recently neural networks have been used for solving a great variety of problems, including architectural ones, some of which cannot be solved at all by normal algebraic methods. Ordinary algebra has great difficulties in solving general equations higher than the third degree but neural nets have no problem with this, provided that there is a definite solution. They are particularly successful with design problems where some particular parameter has to be optimized or when the normal algebraic rules of procedure are impossibly complicated or totally obscure. A classic problem is that of the "Traveling Salesman". Here we have a hundred cities all of which must be visited once. The problem is to find the shortest route by which this can be accomplished. Neural nets can solve this quite well but you have to accept an approximation. The method is essentially trial and error and it is entirely non-algebraic. It can be made quite automatic as it is easy to compute the total time of travel and to tell whether an improvement has been made. Another example is the development of a program for the extraction of future stock market prices from data derived from the past record. One program produced a 60 per cent correct prediction rate instead of the usual guessing rate of 50 per cent (Instruction book to "NEURALWORKS" - A commercial program). Most of my information about artificial neural networks is taken from the instruction manual which comes with a software package which is a practical neural network called "Neuralworks Professional II and Neuralworks Explorer". This is produced by "Neuralware Inc." of Pittsburgh PA. (I have their address as Building IV, Suite 277, Penn Center West, Pittsburgh PA 15276. Telephone (412) 787 8222. FAX (412) 787 8220 but this may be out of date)

The present position with artificial neural networks is that when using numbers as the input and output it is possible to solve any mathematical equation, or a group of simultaneous equations, at least to an appropriate approximation, provided that a definitive and unique solution exists. It will find a solution provided that the output is a function of the input. Whether or not there exists a defined algebraic method for finding the answer does not matter. The whole thing is very plastic like the CNS and when modeling the CNS it is probably not at all important what the precise rules are with regard to the transfer function or anything else. (A better transfer function might give more speed but would probably not alter the result.) The system is very powerful but there may be a problem of training requiring an intelligent human. With numerical problems it is usually possible to set up automatic rules of assessment.

In the biological equivalent the program has to be self trained and this is very difficult. However, we do have some answers. For example, take a primitive organism such as the water snail, Aplysia or the nematode Caenorhabtidis Elegans. The CNS is essentially a neural network. Here it is interesting that we find that the nervous systems of these creatures are almost completely invariant. All of the various nerve ganglia in one individual compared with another, are identical both with regard to the number of neurons and also with their connections. Each Caenorhabtidis Elegans is a carbon copy of another. They have a diffuse system of information transfer just like we do though on a much more limited scale. It is an interesting fact that the exact wiring diagram of the Caenorhabtidis CNS has now been worked out by Sydney Brenner and his associates. Caenorhabtidis has a rather limited set of inputs and outputs but the output has to be in topographic form for any useful action to be taken. For instance, with Aplysia, stimulation by touching causes retraction of the gill. There are also appropriate responses to the presence of food, light and shadow. How did all this come about? Where is the instructor who programmed the neural net? In this case it is "Evolution". Along with other gradual improvements in the organism over time, "Nature" caused random mutations to modify the organism and that included variations of neural connectivity. If the connections were not good, the Aplysia was simply thrown away; It died. Any improvements were retained by the surviving animals. In higher organisms some less drastic methods are required. The general connectivity of major neural pathways was obtained in this way by the evolutionary process of natural selection. More detailed connections have to be formed by a learning process leading to the "Topographic Form Analyzer" using the feedback mechanisms already described.

One certainly can expect that the CNS will have many of the properties which artificial nets have. One important property which modern neural nets do have, and which the visual system must have also, is the extraction of any function which is the unique solution of a problem. At a certain level we have a variety of complex cells with different abilities, initially self organized from a lot of random connections, notably the ability to respond to lines and ends of lines, and also to grids but you can only carry feature extraction so far. One might look for specific feature extractors like corners and try to build up a hierarchy. Indeed that is what has been attempted by many computer programs. Most of them do not work very well. What the visual system does, is first to code the attributes of vision in very many forms, targeting very many different types of neuron - "On" center and "Off" surround, "Line segment moving this way in this direction, another in that way and all the rest of hundreds of cells with different attributes. Initially this is a random process. When this collection is large enough and sufficiently varied, there is enough information there to imply or define a particular picture. Only the mixture of activity which you actually have, will define the image which you are looking at that particular time. We do know that sufficient information is encoded there because we do know that we are actually able to see. The biological neural network is doing what its artificial counterpart can always do. It extracts the unique answer which it is able to do whenever the output is a function of the input. There is no need to build up intentionally any recognizable hierarchy, though you actually do find that the program has itself assembled groups which show hierarchical features such as oriented line detectors. This is what one might expect from Kohonen's work. The neural network will find the answer if enough information is present for an answer to be defined. This is done by a totally non algebraic process. What it is doing is to extract whatever is hidden there and it translates it to topographic form. The system of reconstruction by the reciprocal network is in fact a "Topographic form Analyzer". Over millions of years evolution has sharpened up this ability and made it faster and more accurate by supplying more of this kind of neuron and more of that. It then selects the organisms which survive the best. The creatures with the poorer mixtures simply die out. There is one interesting point to be made here. That is that with monocular vision the activity of the cells does define the unique output which we actually see. With binocular vision that is not the case. With a Julesz dot diagram stereoscopic pair (See Julesz b. 1971 in "Seeing" by Frisby 1979 Oxford university press, pp 480,542) you cannot immediately see the 3D image (Or Marr's 2 1/2 D), with the dots nicely segregated, but if you focus your eyes back and forth looking at various different distances, the picture suddenly springs into view. When you randomly move your two images against each other and momentarily match up the correct ocular disparity neurons, the information suddenly becomes sufficient to define the 3D picture and the neural network quickly solves it. What the brain is doing here, as it does all the time, is acting as a topographic form analyzer. It is using its ability to extract a unique answer wherever the output is a function of the input. A very similar thing happens with the achievement of superacuity. The information coded in the different values of excitation of the cones do define a picture of composed of pixels which are considerably smaller than a single cone and the clever system finds it. The system takes the resolution to the highest level possible because the feedback is maximal there. Such is the power of neural nets whether they be biological or made of silicon. When one is thinking, information is being passed from one part of the brain to another, alternately in diffuse and in topographic form. At a lower level we are recognizing a picture or identifying a word in an unbroken stream of sound. At a higher level we have concepts.