Abstract
This article presents novel networks that demonstrate how neurons can be connected to process information. The networks perform the same functions as standard electronic logic circuits, but they have different architectures because a neuron’s logic capability is different from the logic gates commonly used in electronic computational systems. With only the capabilities of neuron excitation and inhibition, the networks generate detailed phenomena of short-term memory and electroencephalograms. A single neuron can operate as a functionally complete logic primitive. As few as two neurons can be connected to form a robust flip-flop. Neurons in a memory bank of flip-flops produce the seven characteristics of neuron activity that are associated with memory formation, retention, retrieval, termination, and errors. Neural flip-flops also predict seven more phenomena that are testable by the same methods that led to the discovery of the first seven. Three neurons can form an oscillator, and basic flip-flops can function as toggles. With input from an oscillator, a toggle oscillates at half the frequency of the input. A cascade consisting of an oscillator and four toggles connected in sequence can produce the synchronized firing found in electroencephalograms by enabling neural structures to change states simultaneously. The means and variances of five EEG frequency bands are given as explicit functions of the mean and variance of the neuron delay times in all such cascades’ initial oscillators. The delay times’ two parameters determine the specific frequencies that separate the EEG bands and the peak frequency within each band.
Cascaded oscillators determine the octave relationships between the bands’ boundaries and peaks, and they suggest selective advantages for synchronization and for synchronization in different frequency bands. A simple experiment is described to test for the predicted relationship between the distribution parameters of neuron delay times and EEG frequencies. Any of the networks can be constructed with neurons and tested for many predicted behaviors.
Introduction
DEEP neural networks
This is the fourth in a series of articles that show how neurons can be connected to process information. The first three articles [1-3] explored the analog properties of neuron signals in combinational processing of information, i.e., logic functions whose outputs depend only on the current state of the inputs. The present article considers the digital (Boolean logic) properties of neuron signals in sequential logic operations, whose outputs are functions of both the current inputs and the past sequence of inputs. Neural models for flip-flops and oscillators are shown to produce detailed characteristics of short-term memory and electroencephalograms (EEGs). These phenomena have been known but unexplained for decades.
The networks proposed in these four articles are dynamic, explicit, evolutionary, and predictive (DEEP). The networks’ dynamic operation means the only changes are the levels of neuron activity. No material change is required in the structure of the brain, nor is any change required in the way cells or synapses function. All connections between neurons are shown explicitly, and all assumptions of neuron capabilities are stated explicitly. Only minimal neuron capabilities are assumed, and no network capabilities are assumed. The networks are evolutionary in the sense that they suggest selective advantages for the phenomena they generate. This includes phenomena whose functions are uncertain, such as the matched periods of neural structure activity found in EEGs (a.k.a. brainwaves). (The advantage of simultaneous enabling of neural structures by cascaded oscillators is timing error avoidance.) Finally, the networks are predictive of nervous system phenomena. That is, based on the explicit connections and neuron capabilities, it can be demonstrated that the models generate known nervous system phenomena, and they may predict testable phenomena that are as yet unknown.
Explicit neuron connections and capabilities can lead to micro explanations of macro phenomena, and to numerous predictions of major phenomena that are quite specific. For example, the hypothesis that cascaded oscillators produce EEG frequencies implies that the boundary separating the EEG alpha and beta frequency bands is where μd and σd are the mean and standard deviation (in ms) of the normal distribution of delay times of neurons that make up the initial ring oscillators in the cascades. The precision and number of such predictions make hypotheses eminently falsifiable.
How neurons are connected remains one of the most important unanswered questions in biology [4]. The DEEP properties are necessary for a model to demonstrate how neurons might actually be connected to process information in the “real time” of most brain functions (a few milliseconds). The networks presented here and in the three previous papers [1-3] are apparently the only models of nervous system activity that have these properties. The purpose of this article is to call attention to the dearth of DEEP models in the literature and to demonstrate that designing such models is possible. DEEP models are needed to begin to make progress in discovering how synaptic connections are organized.
Unexplained phenomena and previous models
Single neuron logic capability
McCulloch and Pitts’ seminal paper [5] proposed that the brain is made up of logic gates. This idea has had an enormous influence on artificial intelligence, but the paper’s “… concept of simplified Boolean neurons had a limited impact on neuroscience.” [6]. More than 70 years later, the nervous system’s fundamental computational abilities are still unclear [7]. In that time span, many theoretical models have been proposed for neuron responses as mathematical or logic functions, but the modern view of these models follows “the adage that all models are wrong, but some are useful” [8].
The neuron response models that are useful in AI are assumed to have sophisticated mathematical capabilities. This makes it possible to simulate the models on a computer with impressive results. However, there is little or no evidence that actual neurons can carry out the particular mathematical functions assumed for the models.
A prominent example of a neuron response model assumes a neuron has modifiable weights that are applied to signals at its synapses. The neuron then adds these weighted inputs in a nonlinear way to produce an output. Apparently there is no evidence that neurons have these capabilities. In addition, because the weights are modified over time with repeated inputs, the method is not dynamic and therefore cannot model fast functions such as short-term memory.
Perhaps more importantly, sophisticated neuron capabilities are unnecessary for the requirements of information processing. The model used here shows that neurons with one inhibitory input that can suppress one excitatory input are sufficient for all information processing. Apparently there is no other claim in the literature that a single neuron can function as a specific logic primitive (even a functionally incomplete logic primitive) based on minimal neuron capabilities.
Short-term memory
Memory tests have shown that certain neurons fire continuously while information is held in short-term memory. This activity was found in neurons in the visual, auditory, and sensorimotor cortexes of monkeys while corresponding sensory information is held in memory [9, 10]. Similar activity has been found more recently in humans [11].
In the experiments, seven characteristics of neural activity were associated with memory formation, retention, retrieval, termination, and errors: 1) Before the stimulus was presented, the sampled neuron discharged at a low, baseline level. 2) When the stimulus was presented, or shortly after, the neuron began to fire at a high frequency. 3) The high frequency firing continued after the stimulus was removed. 4) The response was still high when the memory was demonstrated to be correct. 5) The response returned to the background level shortly after the test. 6) In the trials where the subject failed the memory test, the high level firing had stopped or 7) had never begun.
Several models have been proposed for memory mechanisms composed of neurons, but apparently none produces the seven phenomena described above. It is also apparent that no memory model has been proposed that has the DEEP properties. The neural flip- flops and memory banks presented here produce all of the phenomena.
Electroencephalograms
Electroencephalograms show widespread rhythms, or brainwaves, that consist of many neurons firing with matched periods. The spectrum of frequencies has been partitioned into bands according to the behavioral and mental state associated with the frequencies in each band. Within each band the distribution of frequencies is unimodal [12-15]. The ratios of consecutive EEG band boundaries [16] and the ratios of consecutive EEG band peak frequencies [12-15] are 2.
The EEG phenomena raise several questions. What causes a neuron to fire with a regular period for many seconds? What is the function of such long-term, periodic firing? What produces and what is the function of the widespread synchronization of periodic firing found in EEGs? What produces and what is the function of the wide distribution of EEG frequencies in bands? What produces the unimodal distribution in each band and the octave relationships between the bands’ peaks and boundaries?
Answers to a few of these questions have been proposed, but there has been no explicit model that can explain more than one or two phenomena. Most models are based on “black box” networks or broad assumptions of neuron capabilities. Apparently none has the DEEP properties. Below are two prominent examples of proposed oscillator models.
Pacemaker cells are natural oscillators that cause involuntary muscles and other tissues to contract or dilate. They are spontaneously active neurons with a specialized cell membrane that allows sodium and potassium to cross and generate regular, slow action potentials (around 100 spikes per minute) [17, 18]. Modulating input controls the spike frequency. Except for generating periodic signals, pacemaker cells do not offer answers to any of the questions above. It is not clear, for example, how pacemaker cells could generate the wide distribution of EEG frequencies, their unimodal distribution in bands, or the octave relationships of the bands.
The Kuramoto model [19, 20] provides a widely accepted explanation of synchronized firing found in EEGs. The model assumes, without supporting evidence and without an explanation of a mechanism or function for this behavior, that the neurons’ signals oscillate naturally, that these oscillations are nearly identical, and that each neuron is linked to all the others. Moreover, the model does not appear to answer any of the above questions besides how synchronization occurs.
The cascaded oscillators model proposed here can produce the synchronized firing found in EEGs by enabling neural structures’ state changes simultaneously. It will be shown that the model provides answers to all of the questions above. In addition, the distribution of EEG frequencies in each band is entirely determined by just two parameters: the mean and variance of the initial oscillators’ neuron delay times. This distribution includes the specific frequencies that separate the EEG bands and the peak frequency within each band.
The models in brief
The neural networks presented here do not build on previous neural models, including those in [1-3]. However, the networks are similar to standard logic circuits designed for electronic computational systems, and they perform the same functions. The new networks are also significantly different from the standard architectures (designs specifying components and connections) because a neuron’s logic function, described immediately below, is virtually never used in electronic systems.
Single neuron logic gate
Recognizing a neuron’s capability of functioning as a logic primitive makes a useful connection between neuroscience and the field of logic circuit design in electronic computational systems.
All neurons considered here have one excitatory and one inhibitory input. Previously it was shown than such a neuron can perform the AND-NOT logic operation, i.e., X AND NOT Y [2]. In simplest terms, this is because the neuron is active when it has excitatory input and does not have inhibitory input. This neuron logic capability is discussed more thoroughly in two sections on neuron response. The AND-NOT operation is not to be confused with the NAND operation (NOT(X AND Y)) that is commonly used in electronic systems.
If the truth value of X is TRUE, then the truth value of X AND NOT Y is NOT Y. This means a neuron with constant excitatory input is a logic NOT gate. A NOT gate is commonly called an inverter since it inverts the value of the single variable input.
The AND-NOT gate together with a NOT gate make up a functionally complete set. So with access to a high excitatory input, the AND-NOT operation is functionally complete, meaning any logic function can be performed by a network of such logic gates. In other words, sufficiently many neurons operating as AND-NOT gates can perform all of the brain’s processing of information.
Neural flip-flops
Flip-flops are the basic building blocks of sequential logic systems. Information is stored in an electronic flip-flop by a brief input signal that makes a different component produce a high output signal. A neural flip-flop (NFF) functions the same way, but the components are neurons. Here it is shown that a few AND-NOT gates can be connected to form two standard flip-flops, the Set-Reset (SR) and JK. Both are commonly used as memory elements in electronic computational systems, and the JK flip-flop can be configured to function as a toggle. A toggle is a flip-flop with one input that inverts the flip-flop’s memory state each time the input is high.
Cascaded oscillators
As mentioned above, an AND-NOT gate can be configured to invert an input signal (high input to low output and vice versa). An odd number of three or more inverters connected sequentially in a ring produces periodic bursts as each gate inverts the next one. The odd number of inverters makes the network state unstable, resulting in the oscillation of each inverter’s output. The signal produced by each gate has a regularly repeating pattern of a burst of high activity followed by an interval of low activity.
With input from an oscillator, a toggle functions as another oscillator. Because two high inputs are required for each cycle (one to set the memory state, another to reset it), a toggle-as-oscillator produces a signal whose frequency is exactly half that of the toggle’s input. One of the outputs of a toggle-as-oscillator can provide input to another toggle, making it an oscillator. A ring oscillator and several toggles can thus be connected as a cascade of oscillators, with each oscillating at half the frequency of its input.
Testable predictions
This article ends with several testable predictions made by the models, briefly outlined here.
Any of the proposed networks could be constructed with actual neurons and tested for specific predicted behavior.
As noted above, memory tests have shown that certain neurons fire continuously while information is held in short-term memory, and NFFs produce all seven characteristics of these neurons’ activity. NFFs predict seven additional phenomena regarding this type of cell, along with a neighboring cell. These predictions can be tested by the same methods that were used in discovering the first seven phenomena.
The pair of cells with outputs labeled M and in the figures are predicted to have 1) close proximity, 2) reciprocal 3) inhibitory inputs, 4) complementary outputs, and 5) noise-reducing responses to the inputs. When information is changed in memory, 6) the cell with high output changes first with 7) the other changing a few milliseconds later.
The hypothesis that cascaded oscillators produce EEG frequencies implies the oscillators and EEG bands have the same distributions of frequencies. The oscillators’ frequencies means and variances are shown here to be functions of the mean and variance of neuron delay times. The means and variances of neuron delay times and EEG band frequencies can be estimated from random samples. Using standard tests for equal means and variances, the EEG sample statistics can be compared to the EEG parameters predicted by the delay time statistics.
Analysis
Figure symbols
For several reasons, the neural networks in the figures are illustrated with standard (ANSI/IEEE) logic symbols rather than symbols commonly used in neuroscience schematic diagrams. A comparison is shown in Fig 1.
The symbols in Fig 1A can be interpreted in two ways. As logic symbols, the rectangle with one rounded side represents the AND logic function, and a circle represents negation. So the networks in the figures can be constructed with ordinary electronic components or simulated with electronic circuit software. Second, it will be shown that the logic gate represented by an AND symbol and a circle can be implemented by a single neuron, with a circle representing inhibitory input and no circle representing excitatory input. As illustrated in Fig 1B, inhibition is often represented by a small closed circle and excitation by a closed triangle, but there does not seem to be an accepted standard of symbols for networks of neurons.
The circles in Fig 1A are a small exception to standard symbols. A circle representing negation normally comes after a logic gate. Standard notation would include a triangle before the circles in the figures to represent a buffer gate, whose output equals the single input. The triangle is omitted here to emphasize that the AND gate with a circle is considered a single logic gate or a single neuron.
The standard logic symbols normally represent Boolean logic, which for most electronic computational systems means digital signal processing. Neurons can convey analog signals, either with signals of graded strength or with the strength of signals consisting of spikes measured by spike frequency. It will be shown that the neural networks in the figures can generate robust digital signals, i.e., signals with only high and low strengths (except during transition from one to the other).
The similarities and differences between the novel diagrams of networks designed for neurons and diagrams of standard logic functions implemented electronically are easier to see if they are both illustrated with the same symbols. As mentioned in the introduction, although the new networks produce the same results as standard architectures, there are significant design differences because neurons function as logic gates that are different from commonly used electronic gates.
The single, branching output channels in Fig 1A are more realistic depictions of axons than the multiple output channels of Fig 1B.
Finally, diagrams in standard engineering form clarify the connectivity, the logic function of each component, the distinction between feedback and feed-forward signals, and the overall direction of signal processing.
Binary neuron signals
Neuron signal strength, or intensity, is normalized here by dividing it by the maximum possible intensity for the given level of adaptation. This puts intensities in the interval from 0 to 1, with 0 meaning no signal and 1 meaning the maximum intensity. The number is called the response intensity or simply the response of the neuron. If 1 and 0 stand for the truth values TRUE and FALSE, respectively, neurons can process information contained in neural signals by functioning as logic operators. The responses 1 and 0 are also referred to collectively as binary signals and separately as high and low signals. Normalization is only for convenience. Numbers labeled Max & Min would do as well as 1 and 0.
As noted above, the strength of a signal consisting of action potentials, or spikes, can be measured by spike frequency. A brief high signal consists of a burst of spikes at the maximum spiking rate. For a signal that oscillates between high and low, the frequency of the oscillation is the frequency of bursts, not the frequency of spikes.
For binary signals, the response of a neuron with two inputs is assumed to be as shown in Table 1. Of the 16 possible binary functions of two variables, this table represents the only one that is consistent with the customary meanings of “excitation” and “inhibition.” Table 1 is also a logic truth table, with the last column representing the truth values of the statement X AND NOT Y.
The networks presented here require a continuously high excitatory input. In the figures, this input is represented by the logic value “TRUE.” If the figure represents an electronic logic circuit, the high input is normally provided by the power supply. If the figure represents a neural network, the high input can be accomplished by neurons in at least four ways. 1) A continuously high signal could be provided by a cell that has excitatory inputs from many cells that fire independently [21]. 2) Neurons that are active spontaneously and continuously without excitatory input are known to exist [22, 23]. A network neuron that requires a high excitatory input could receive it from a spontaneously active neuron, or 3) the neuron itself could be spontaneously active. 4) It will be seen that the high input could be provided by one of a flip-flop’s outputs that is continuously high.
Noise in neuron signals
All results for the networks presented here follow from the responses to binary signals in Table 1 and the algebra of Boolean logic applied to the networks’ connections. Although binary signals are common in modeling neuron response, it has apparently not been demonstrated that neurons are capable of maintaining such signals in the presence of noise. Analog signals (intermediate strengths between 0 and 1) and noise-reducing responses are considered here only to show how the networks in the figures can have this capability.
Evidence exists that some neurons have at least a minimal capability of reducing moderate levels of additive noise in binary inputs. First, some neurons are known to have sigmoid responses to single inputs, including inhibitory inputs [24-26]. A sigmoid response reduces moderate levels of additive noise in a binary signal by producing an output that decreases an input near 0 and increases an input near 1. It is not clear whether a single neuron has been tested for sigmoid responses to both excitatory and inhibitory inputs. That would be sufficient for the noise-reducing requirements of the NFFs here.
Second, without some noise-reducing capability, no network would be able to sustain the high rate of firing known to be associated with short-term memory [9-11]. The cumulative effect of additive noise would quickly attenuate the high output.
Noise reduction in a response to both excitatory and inhibitory inputs can be accomplished by a function of two variables that generalizes a sigmoid function’s features. The noise reduction need only be slight for the proposed NFFs because they have continuous feedback loops that repeatedly reduce the effect of noise with each iteration of feedback.
Let F(X, Y) represent a neuron’s response to an excitatory input X and an inhibitory input Y. The function must be bounded by 0 and 1, the minimum and maximum possible neuron responses, and satisfy the values in Table 1 for binary inputs. For other points (X,Y) in the unit square, suppose F satisfies The neuron responses of Table 1 are max{0, X-Y} (the greater of 0 and X-Y). For binary inputs with moderate levels of additive noise that makes them non-binary, conditions 1 and 2 make the output either equal to 0 or closer to the intended output of Table 1 than max{0, X-Y}.
Conditions 1 and 2 are sufficient to suppress noise in the proposed NFFs. Neurons that make up the NFFs are assumed to have these minimal noise-reducing properties. The level of noise that can be tolerated by the NFFs depends on the minimum distance from a vertex of the unit square to a point outside the region in the unit square where condition 1 or 2 holds.
An example function is given here. For any sigmoid function f from f(0) = 0 to f(1) = 1, the following function has the noise-reducing properties 1 and 2 and also satisfies Table 1: This function is plausible as an approximation of a neuron response because it is sigmoid in each variable and some neurons are known to have sigmoid responses to single inputs, as mentioned above. The same sigmoid function applied to X and Y is not necessary to satisfy conditions 1 and 2. The function F could be the difference of two different sigmoid functions.
The function is illustrated in Fig 2 for a specific sigmoid function. The sine function of Fig 2A was chosen rather than any of the more common examples of sigmoid functions to demonstrate that a highly nonlinear function is not necessary for robust maintenance of binary signals. On half of the unit square, where Y ≥ X, Fig 2B shows that F has the value 0. This reflects the property that a large inhibitory input generally suppresses a smaller excitatory input.
The response function F(X, Y) in Fig 2 is used for the network simulations as follows. The whole number i represents time, and the time unit is neuron delay time. That is, each number i represents i neuron delay times. The variables are initialized at time i = 0. At time i > 0, the output Zi of each cell that has excitatory and inhibitory inputs Xi-1 and Yi-1 at time i-1 is: Conditions 1 and 2 do not indicate capabilities of sophisticated mathematics. Fig 3 shows that a single transistor and three resistors can be configured to satisfy the conditions. The network output was simulated in engineering software. The inputs X and Y vary from 0V to 5V in steps of 0.05V. A 5V signal commonly stands for logic value 1, and ground stands for logic value 0.
Logic gates and flip-flops
Fig 4 shows two logic primitives and several flip-flops. All are composed of the first type of logic primitive in Fig 4A, which can be implemented by a single neuron.
If X and Y are statements with truth values TRUE or FALSE, the statement “X AND NOT Y” is TRUE if and only if X is TRUE and Y is FALSE. This logic primitive is illustrated in Fig 4A. The figure can also represent a neuron with one excitatory input and one inhibitory input, whose response to binary inputs is X AND NOT Y by Table 1. The logic primitive NOT X is TRUE if and only if X is FALSE. Fig 4B shows that an AND- NOT gate with a continuously high input functions as a NOT gate.
The AND-NOT logic primitive has efficiency and power that have been underappreciated. It is in the minority of logic primitives that are functionally complete. With analog signals, AND-NOT gates can make up a powerful fuzzy logic decoder whose architecture is radically different from, and more efficient than, standard electronic decoder architectures [2, 27, 28]. Implemented with neural AND-NOT gates, these fuzzy decoders generate detailed neural correlates of the major phenomena of color vision and olfaction [1, 2]. Analogously to the single-neuron AND-NOT gate, the gate can be implemented electronically with a single transistor and one resistor [27]. Any mechanism that can activate and inhibit like mechanisms and has access to a high activating input is a functionally complete AND-NOT gate. It may not be coincidence that the components of disparate natural signaling systems have these capabilities, e.g., immune system cells [29-32] and regulatory DNA [33, 34] in addition to transistors and neurons.
The most common type of memory element used to store one bit of information in electronic computational systems is a latch or flip-flop. The more formal name is bistable multivibrator, meaning it has two stable states that can alternate repeatedly. A distinction is sometimes made between a “flip-flop” and a “latch,” with the latter term reserved for asynchronous memory mechanisms that are not controlled by an oscillator. The more familiar “flip-flop” will be used here for all cases.
A flip-flop stores a discrete bit of information in an output encoded as 0 or 1. This output is labeled M in Fig 4. The value of M is the flip-flop state or memory bit. The information is stored by means of a brief input signal that activates or inactivates the memory bit. Input S sets the state to M = 1, and R resets it to M = 0. Continuous feedback maintains a stable state. A change in the state inverts the state.
Two basic types of flip-flops are the Set-Reset (SR) and JK. Fig 4C shows an active low SR flip-flop. The S and R inputs are normally high. A brief low input S sets the memory bit M to 1, and a brief low input R resets it to 0. Adding inverters to the inputs produces the active high SR flip-flop of Fig 4D. The S and R inputs are normally low. A brief high input S sets the memory bit M to 1, and a brief high input R resets it to 0.
Fig 4E shows a flip-flop with an enabling input. The S and R inputs in Fig 4D have been replaced by AND-NOT gates that allow the S or R input to be transmitted only when the enabling input is low. In synchronized signaling systems, various logic circuits are enabled simultaneously by an oscillator to avoid timing errors.
In Fig 4F, the enabling input in Fig 4E has been replaced by input from the flip-flop outputs. A disadvantage of the SR flip-flop is that if S and R are high simultaneously, the outputs are unpredictable. The advantage of the JK flip-flop in Fig 4F is that if S and R are both high simultaneously, the flip-flop state is inverted because the most recent inverting input is inhibited by one of the outputs. This means the JK flip-flop can be configured as a toggle by linking the Set and Reset inputs, as illustrated in the figure. A problem is that the toggle functions correctly only for a short duration of high input. If the duration is too long, the outputs will oscillate.
Neural flip-flop simulation
The simulation in Fig 5 shows that the operation of the NFF in Fig 4D is robust in the presence of additive noise in the inputs. The simulation was done in a spreadsheet with the outputs of each cell computed by equation 3. At time i = 0, the outputs are initialized at M0 = 0 and . (If both are initialized at 0, they will oscillate until either S or R is high.) Low level additive noise and baseline activity in the inputs are simulated by a computer-generated random number between 0.01 and 0.1. The noise is offset by 0.01 so it does not obscure the high and low outputs in the graphs. Each of the two larger noise bursts in Set and Reset is simulated by the sum of two sine functions and the computer- generated noise. The higher Set and Reset signals that successfully invert the memory state are simulated by a sine function plus noise. The slow rise time of these inputs, over several neuron delay times, is exaggerated to make the robust operation of the network clear. The high Enabling input TRUE is simulated by 1 minus noise.
Memory bank
If information stored in short-term memory is no longer needed, active neurons consume energy without serving any useful purpose. An energy-saving function can be accomplished with NFFs. Fig 6 shows a memory bank of three NFFs of Fig 4D, with a fourth serving as a switch to turn the memory bank on and off. The memory elements are enabled by excitatory input from the switch. A large memory bank could be organized as a tree, with switches at the branch points and memory elements at the leaves, so that at any time only the necessary memory elements are enabled.
Oscillators
An oscillator is the basic element of a timing mechanism. An odd number of three or more inverters connected sequentially in a ring produces periodic bursts as each gate inverts the next one. All inverters in the ring produce oscillations with the same frequency. The signals have approximately the same burst duration and their phases are approximately uniformly distributed over one cycle. Their common period is twice the sum of the inverters’ delay times. (The sum is doubled because each component inverts twice per cycle.) A ring oscillator is the simplest type of oscillator implemented with logic gates, and the simplest ring oscillator consists of three inverters.
As described in the introduction, an oscillator can be connected in sequence with toggles to form a cascade of oscillators, with each toggle oscillating at half the frequency of its input. The master-slave toggle is the customary choice for cascaded oscillators because of its ability to invert no more than once regardless of the high input duration. However, it is shown here that the problem of JK toggles inverting more than once with a long high input can be resolved by using an early output in the JK toggle’s signal pathway as the input to the next toggle. The two initial neurons have the same pulse duration as the toggle’s input. This means an entire cascade can be composed of a ring oscillator and JK toggles, which require half as many components as master-slave toggles. This configuration is illustrated in Fig 7. The use of JK toggles as cascaded oscillators may be new.
Cascaded oscillator simulation
The simulation in Fig 8 illustrates the main properties of the cascaded oscillators in Fig 7. Like the simulation in Fig 5, this was done in a spreadsheet with the outputs of each cell computed by equation 3. At time i = 0, the outputs are initialized in a stable state. Noise is simulated as a random number between 0 and 0.1. The Enabling input begins as baseline noise and transitions between 0 and 1 as a sine function plus noise. During the interval when the enabling input is high, it is 1 minus noise.
Frequency distributions of cascaded neural oscillators
A normal distribution with mean μ and standard deviation σ will be denoted by N(μ, σ). The normal probability density function (PDF), whose graph is commonly known as the bell curve, is: Since delay times of neurons are determined by several factors, delay times should be approximately normally distributed (by the central limit theorem). The normal distribution of neuron delay times with mean μd and standard deviation σd is N(μd, σd).
As noted earlier, a ring oscillator’s period is twice the sum of the inverters’ delay times. If X1, …, Xn are the delay times of n neural inverters in a ring oscillator, the oscillator’s period is Y = 2(X1 + …+ Xn). The neurons’ delay times are independent and identically distributed as N(μd, σd), so by the elementary properties of PDFs, the ring oscillator’s period Y is normally distributed as
Five EEG frequency bands are considered in this article: gamma, beta, alpha, theta, and delta. Some researchers have found more bands or divided the bands into sub-bands depending on the focus of their research, but these five are discussed most often in the literature. To obtain five oscillations with cascaded oscillators, four toggles are needed in addition to the initial ring oscillator. These toggles’ periods are 2Y, 4Y, 8Y, and 16Y. Again by elementary properties of PDFs, these random variables are also normally distributed with mean and standard deviation double that of the input. This is the octave relationship between the periods’ distributions.
With all of the cascaded oscillators’ PDF parameters given by equation 5 and the octave relationship, and the PDFs given by equation 4, the four intersections of each pair of consecutive PDFs can be found by elementary algebra. For i = 1, …, 4 (i = 1 representing the ring oscillator), the intersection of PDFi and PDFi + 1 is:
The factor 2i shows the intersections also have the octave relationship.
For small networks of neurons with chemical synapses, nearly all of the delay occurs at the synapses. Several studies have measured neuron delay times [e.g., 35, 36], but the literature apparently does not have empirical estimates of the parameters of the delay times’ distribution. However, a description of the range is “at least 0.3 ms, usually 1 to 5 ms or longer” [22]. Although the description is far from precise, delay time parameters can be estimated.
The description of the range has two parts. The first part “at least 0.3 ms” seems to refer to all observations. The second part “usually 1 to 5 ms or longer” seems to describe the ranges of typical samples, with “5 ms or longer” representing the ranges’ right endpoints. In that case, the interval [1 ms, 7 ms] is at least a reasonable, rough estimate of the range of a moderately sized sample.
If only the range of a sample (minimum value, m, and maximum, M) is known, the midpoint can be used as an estimate of the mean of a distribution. Simulations have shown that (M - m)/4 is the best estimator of the standard deviation for moderately sized samples [37]. Based on this and the estimated range [1 ms, 7 ms], neuron delay times are estimated to be normally distributed as
About 99.3% of this distribution is greater than 0.3 ms. This agrees well with the description “at least 0.3 ms.” About 73% lies between 1 and 5 ms, and 95% is between 1 and 7 ms. This agrees reasonably well with the description “usually 1 to 5 ms or longer.”
The hypothesis that cascaded oscillators produce EEG frequencies implies the initial ring oscillator must have the minimum of three inverters, because an oscillator with more inverters would be too slow to generate EEG oscillations in the gamma band. (This in turn implies, unsurprisingly, that the brain evolved to enable some information to be processed as fast as possible.) By substituting the parameters of equation 7 into equation 5, the period of a three-inverter ring oscillator is estimated to be normally distributed as
This period PDF (equation 4) and the period PDFs of four cascaded toggle/oscillators (from the octave relationship) are shown in Fig 9. The PDF intersections are found by substituting the parameters in equation 7 or 8 into equation 6.
Synchronous logic systems
Logic systems have a timing problem in ensuring the various subcircuits change states in the correct chronological sequence. Synchronous logic systems generally have simpler circuit architecture and fewer errors than asynchronous systems. This is the reason nearly all electronic logic systems are synchronized by an enabling pulse to each component circuit, so that whenever any components change states, they change simultaneously. The pulse in such systems is usually produced by an oscillator. The enabling input in Fig 4E and oscillators in Fig 7 illustrate how such synchronization is possible with neural networks.
Timing problems are greater in sequential logic than in combinational logic, and greater in parallel processing than in serial processing. Much of the information processing in the brain involves sequential logic and virtually all of it is parallel. The selective pressure for synchronization in the brain would have been high, and the neural implementation demonstrated here is quite simple.
The processing speed in a synchronous system depends largely on the enabling oscillator’s speed. Electronic processors generally operate at a single speed. A large system like the brain that performs many diverse functions may have several different processing speed requirements. The tradeoff for greater processing speed is a higher error rate. Functions that can tolerate a few errors, and that need fast results with many simultaneous small computations, require high processing speeds. Functions that are less dependent on speed or massive computation, or that require few errors, or whose component networks are large and complex and therefore slow to change state, call for slower processing.
Electroencephalogram frequencies
The EEG frequency bands and associated behavioral and mental states are consistent with the function of multiple frequencies that was suggested in the preceding paragraph. Gamma waves (high frequencies) are associated with vision and hearing, which make sense out of massive data input in a few milliseconds. Beta waves are associated with purposeful mental effort, which may involve less data input while requiring few errors and complex operations. Alpha waves are associated with relaxed wakefulness, theta waves with working memory and drowsiness, and delta waves with drowsiness and sleep. These categories require successively slower information processing, and they have corresponding EEG bands of lower frequencies.
Cascaded oscillators can produce this neural activity. A neural oscillator can synchronize state changes in neural structures by enabling them simultaneously. The enabling oscillator pulse by itself does not produce any state changes. It only forces states to change simultaneously when they do change. So the initial ring oscillator’s high frequency signal could simply be connected directly and permanently to the enabling gates (as illustrated in Fig 4E) of networks in the visual and auditory cortexes, the first toggle’s signal to the prefrontal cortex for purposeful mental effort, etc. A large number of neural structures synchronized in this way by cascaded oscillators would exhibit the bands of matched periods found in EEGs.
Cascaded oscillators hypothesis
The cascaded oscillators hypothesis is that the brain structures’ matched periods that are found in EEGs are the result of the structures’ synchronization by cascaded neural oscillators. The main implication of the hypothesis is that the distribution of EEG frequencies in each band is the same as the distribution of the corresponding oscillator’s frequencies (Fig 9).
Two implications of the cascaded oscillators hypothesis are considered here. 1) The EEG frequency band boundaries are the intersections of the oscillators’ PDFs. 2) The EEG band peaks occur at the means of the oscillators’ PDFs. These implications are independent; i.e., neither follows from the other. It follows from implications 1) and 2) that the ratios of consecutive EEG band boundaries and the ratios of consecutive EEG band peak frequencies are 2.
The graphs in Fig 10 compare estimates of EEG band peaks and boundaries with the means and intersections of estimated oscillator PDFs in Fig 9. The graphs strongly suggest that the oscillators hypothesis is consistent with the available data. Following the EEG convention, oscillations are measured in frequencies rather than periods. For convenient comparison, the data in Fig 10 are shown together in Table 2. The linear regression results are in Table 3.
The following analysis for Fig 10 uses three properties of exponential functions:
By the octave relationship of the means and intersections of the PDFs in Fig 9, they all lie on an exponential function with base two:
The value 2.6042 Hz (=(1000 ms/sec)/(384 ms/cycle)) was chosen as the constant because EEG oscillations are customarily measured in frequencies and 2.6042 Hz is the smallest frequency in Fig 9.
Fig 10A shows the oscillator PDF means and intersections of Fig 9 graphed as an exponential function y = 2.6042(2x) in log2 linear form. By equation 9, the function in log form is log2(y) = x + log2(2.6042). The x coordinates of the points are computed from equation 10. The whole number differences between the x coordinates for the means and intersections reflect the frequencies’ octave relationships, according to equation 11.
Fig 10B compares the graph of Fig 10A with a regression line for several frequencies that are commonly cited as partition points separating the EEG frequency bands and peak frequencies of three of the bands. The regression is based on 57 data points [12-15, 38-55]. (Estimates of peak frequencies apparently have not been found for the lower frequency delta and theta bands [49].) Since the oscillators hypothesis implies the EEG frequency distributions are the same as the cascaded oscillators, the same x coordinates are assumed for the EEG data.
Fig 10C shows how sensitive the distribution of oscillator frequencies is to small differences in neuron delay time parameters μd and σd. The distribution is especially sensitive to small differences in estimates of the mean delay time μd. Equation 5 shows the ring oscillator PDF mean, and therefore the y-intercepts of the lines in Fig 10C, are determined by μd (with n = 3). The graphs in Fig 10C show that if the average neuron delay time μd differs from 4 ms by as little as 1 ms, the cascaded oscillator means and intersections lie outside the entire range of cited estimates of EEG peak locations & band boundaries. Since the oscillators hypothesis implies the oscillator means and intersections equal the EEG peaks and boundaries, the mean neuron delay alone makes the hypothesis easily falsifiable.
Fig 10D illustrates the effect that errors in the estimates of EEG band boundaries can have on the estimates’ regression line. The diverse values cited as band boundaries show the difficulty in obtaining accurate estimates. There are several possible causes. EEG data are obtained in the noisy environment of the brain. The cascaded oscillators hypothesis implies that the probability of an EEG frequency occurring near an intersection of two oscillator PDFs is small compared to a frequency occurring near a PDF peak. Both of these factors imply that a large amount of data may be needed to obtain a significant estimate of a boundary point.
Possibly most importantly, rounding errors that result from the common practice of stating EEG band boundaries as whole number frequencies (Fig 10B) could have a large effect on the regression line. The errors described in the preceding paragraph are unbiased, so their effects could be minimized with large samples. This is not the case with rounding errors because the errors are biased in the direction of the integer nearest the value being estimated.
For example, the cascaded oscillators hypothesis’ estimate of the delta-theta boundary is near 3.7 Hz (Figs 9 and 10A), but nearly all of the cited boundary estimates in Fig 10B were 4 Hz. (It may be significant that one of the cited estimates for the boundary was 3.5 Hz, the only non-integer in the whole data set.) If the 12 cited values of 4 Hz in Fig 10B were obtained by rounding actual experimental findings near 3.7 Hz, correcting this error alone would increase the slope of the regression line from 0.944 to 0.973, substantially closer to the predicted value of 1. This regression line is illustrated in Fig 10D. In addition, the regression line’s new y-intercept is closer to the predicted y-intercept in Fig 10A. The new slope and intercept together move the regression’s estimated EEG band peaks and boundaries substantially closer to the predicted oscillator PDF means and intersections.
The two lines in Fig 10B show the EEG frequency values predicted by the oscillator hypothesis are close to the EEG linear regression estimates. The closeness can be measured by the square root of the sum of the squared differences of the nine pairs of values (in log2 linear form), i.e., the Euclidean distance between two points with nine coordinates. By this measure, the correction of a possible rounding error shown in Fig 10D nearly cuts the distance in half. The distance between the regression estimates in Fig 10B and the oscillator predictions of Fig 10A is 0.221. For Fig 10D the distance is 0.133. These values are shown in Table 2.
Carefully documented statistics for EEG band peaks and boundaries are not easily obtained. Various frequencies are routinely cited in the literature without a source. Some frequencies that were cited several times may have been based on the same source. But the cascaded oscillators’ predicted values of Fig 10A do fall within or near the range of values cited for each EEG band peak and boundary, as illustrated in Fig 10B. The regressions in Figs 11B and 11D and the small values of both distances in Table 3 show the oscillator hypothesis predictions are close to observed EEG values. The log2 scale and the two regression lines’ slopes near 1 suggest that peak locations and successive boundary points increase by a factor of two, as predicted by the oscillators hypothesis. Although the apparent octave relationship between EEG frequency bands “has been widely reported but rarely commented upon” [16], the regression in Fig 10B is apparently the first test of the octave relationship.
Results and explanations of known phenomena
Memory
NFFs are a plausible mechanism for the brain’s short-term memory. Flip-flops are the main elements for random access (working) memory in electronic systems. The simulation shown in Fig 5 demonstrated that with minimal neuron noise-reducing capabilities, NFFs can be robust in storing information. Because NFFs function dynamically, information can be stored quickly. The time required to set or reset an NFF is the time a signal takes to pass through two to four neurons, roughly 10-20 ms. NFFs are inexpensive in material requirements because they require only a few cells and because storing information requires no physical change except neurons’ activation level. NFFs consume energy continuously while they are holding information, but this is consistent with the brain’s high energy consumption and it may be one of the selective pressures that resulted in static mechanisms for long-term memory.
Direct electrophysiological evidence has shown that NFF memory banks (Fig 6) can generate neural correlates of neuron firing associated with memory. Memory tests in monkeys lasting up to 30 seconds found that certain neurons respond in the visual, auditory, or sensorimotor cortexes of monkeys while corresponding sensory information is held in memory [9, 10]. Similar neuron responses have recently been found in humans [11]. For all of the seven memory characteristics, one of the two outputs of an NFF in a memory bank is identical to the neuron’s response.
1) Before the stimulus was presented, the sampled neuron discharged at a low, baseline level. This is one of the two NFF output neurons before the NFF state is inverted. For convenience, label it M before the NFF is set. 2) When the stimulus was presented, or shortly after, the neuron began to fire at a high frequency. This is the output M after the NFF is set by the input S. 3) The high frequency firing continued after the stimulus was removed. This is the stored memory bit M after the NFF input S returns to its normal low value. 4) The response was still high when the memory was demonstrated to be correct. This is the high value of M holding information in memory. 5) The response returned to the background level shortly after the test. The memory bank is turned off when the stored information is no longer needed, disabling all of the outputs. 6) In the trials where the subject failed the memory test, the high level firing had stopped or 7) had never begun. In instances where the high level firing had stopped, the memory bank may have been turned off before the memory was tested, or a distraction may have caused the NFF to be overwritten with new information, or noise or other errors may have inverted the NFF. In instances where the high level firing had never begun, the NFF was not set to record the information or the NFF recorded it incorrectly (for one of many possible reasons, e.g., the subject was not paying attention). For each of these possibilities, the NFF would correctly predict both the failed memory test and the corresponding observed neuron behavior.
Electroencephalograms
The cascaded oscillators hypothesis answers the EEG questions raised in the introduction. The hypothesis is that cascaded neural oscillators synchronize neural structures’ state changes by enabling them simultaneously. Here are the answers that follow from the hypothesis.
What causes a neuron to fire with a regular period for many seconds? Neurons fire with a regular period because they are enabled by an oscillator. What is the function of such long-term, periodic firing? The periodic firing continues for a time because the neurons are processing information each time they are enabled.
What produces and what is the function of the widespread synchronization found in EEGs? Many neural structures enabled simultaneously by the same oscillator produce the synchronization. The function of synchronization is timing error avoidance in processing information.
What produces and what is the function of the wide distribution of EEG frequencies in bands? The frequencies in each EEG band are produced by a different oscillator. The function of multiple oscillators is meeting the needs of different brain functions in the tradeoff between speed and accuracy.
What produces the unimodal distribution in each band and the octave relationships between the peaks and boundaries? The unimodal distributions are due to the normal distribution of neuron delay times in the initial ring oscillators in cascades of oscillators. This makes the distribution of frequencies of each oscillator normal. The ratio of consecutive boundaries and peak locations is 2 because consecutive cascaded oscillators increase the oscillation period by a factor of 2.
Two implications of the cascaded oscillators hypothesis were considered here: The EEG frequency band boundaries are the intersections of the cascaded oscillators’ PDFs, and the peaks within the bands occur at the means of the PDFs. These results imply that EEG peaks and bands have an octave relationship. Fig 10 and the accompanying analysis showed these implications appear to be consistent with empirical measurements.
Testable predictions
Neural flip-flops
As stated earlier, one of the outputs of an NFF in a memory bank produces all seven characteristics of neural activity associated with short-term memory. An NFF’s two outputs M and together predict seven more properties that could further test whether NFFs are used for short-term memory. It should be possible to find these predicted phenomena by the same techniques used to find the original seven phenomena since either M or should be the cell already found in short-term memory. The other should be nearby.
1) The outputs M and are complements of each other; i.e., when one is high the other is low. So not only is one neuron firing continuously while information is held in memory, as already found experimentally, there should also be another neuron with complementary output. 2) The two cells have reciprocal input (every flip-flop in Fig 4). Because the two cells have reciprocal input and are part of a small network, they should be in close proximity. 4) The reciprocal inputs are inhibitory (Fig 4). 5) The two cells have responses that satisfy inequalities 1 and 2. 6) When the cells change states, the high state changes first. This is because the change in the cell with the high state causes the change in the cell with the low input. This can be seen in every flip-flop in Fig 4. The change order is difficult to see in the Fig 5 simulation because of the small time scale and the slow rise time of the Set and Reset inputs, but the simulation does have one neuron delay time between the completion of the two outputs’ state changes. 7) The other cell’s output then changes from low to high within a few milliseconds. This happens quickly because reciprocal input from the first change causes the second within approximately one neuron delay time, regardless of how long information is held in memory.
Constructed neural networks
Any of the networks in the figures could be constructed with actual neurons and tested for the predicted behavior. The neurons only need to have the property that high inhibitory input suppresses excitatory input. A neuron response that satisfies the inequality conditions 1 and 2 is sufficient for robust operation of the networks in the presence of noise, but the conditions may not be necessary.
The two-neuron NFF of Fig 4C and the three-neuron ring oscillator of Fig 7 may be the simplest to construct and test because they are composed of a small number of neurons. The NFFs are predicted to have outputs that are inverted by a brief input from S or R. (Recall Fig 4C is active low.) The outputs should also exhibit the seven known properties of neural firing associated with memory and the seven properties predicted for NFFs in the preceding section. The predicted behavior for the ring oscillator is oscillating outputs for all three neurons with a period that is twice the sum of the neurons’ delay times, and phases uniformly distributed over one cycle.
Cascaded oscillators hypothesis
Fig 10 and the accompanying analysis offered enough evidence for the oscillators hypothesis to warrant further research. More carefully documented examination of EEG frequencies and neuron delay times are needed for a simple, rigorous test of the oscillators hypothesis. The hypothesis implies an exact relationship between the distribution of neuron delay times and the distribution of EEG frequencies. With random samples of delay times and EEG frequencies, the EEG distributions can be compared with the predicted distributions by the standard tests for equal means and variances. Such samples may already exist in some database.
The oscillator hypothesis implies the EEG gamma band has the same distribution of frequencies as the initial ring oscillator in a cascade, N(μr, σr). Equation 5 gives the ring oscillator period parameters in terms of neuron delay time parameters: μr = 2nμd, and . The number of neurons n in the ring oscillator almost certainly has to be 3 to produce the fast frequencies in the gamma band. Subsequent cascaded toggles’ periods are also normally distributed with mean and standard deviation double that of the input. The oscillators hypothesis implies the other EEG bands have the same distributions as these toggles.
Neuron delay time mean and variance can be estimated from a random sample of neuron delay times. The frequency mean and variance for one or more EEG bands can be estimated from a random sample of EEG frequencies observed along with the behavioral and mental state associated with a band. The equality of the frequency distribution of an EEG band and the distribution predicted by equation 5 and the octave relationship can then be tested by the standard tests for equal means and variances.
Although it is possible that EEG frequencies are produced by cascades with initial ring oscillators made up of specialized neurons whose delay times are different from the general population of neurons, this appears to be unlikely. Fig 10B shows the EEG frequency bands are at least close to the values predicted by the description of the range of all neuron delay times that was used here to estimate oscillator neuron delay time parameters. In addition, neurons in general and oscillator neurons in particular would have both evolved under selective pressure to function as fast as possible.
Summary of results
The dynamic, explicit neural networks presented in Figs 4, 6, and 7 generate 13 EEG phenomena and the seven phenomena of neural firing associated with short-term memory. The networks also suggest selective advantages for both the synchronization of neural firing and synchronization in different frequency bands that are found in EEGs (Results and explanations of known phenomena section). Apparently no explanation of any of these phenomena has been offered before. The networks predict seven testable phenomena of neural firing associated with short-term memory, an exact relation between the distribution of neuron delay times and the distributions of the five main bands of EEG frequencies, and numerous behavioral phenomena of simple neural networks that can be constructed with neurons and tested (Testable predictions section).
Acknowledgements
Simulations were done with Excel and CircuitLab. Network diagrams were created with CircuitLab and MS Paint. Regressions were created with Converge 10.0. Graphs were created with Excel, Converge 10.0, and MS Paint. The author would like to thank Ernest Greene, Arturo Tozzi, David Garmire, Paul Higashi, Anna Yoder Higashi, Sheila Yoder, and especially David Burress for their support and many helpful comments.