1999 Volume 26 Issue 1 Pages 51-74
Researchers in analog computation theory have shown that a recurrent neural network (RNN) can be built to simulate a Turing machine (Pollack, 1987b; Siegelmann & Sontag, 1995). Recently, we showed that it is possible to train RNNs which implement some aspects of analog computation theory-namely a network can develop trajectories that count symbols (Wiles & Elman, 1995). But what are the implications for psychological models of sequence processing based on RNNs ? As a first step toward answering this question, we investigate an RNN in a psycholinguistically motivated task : predict the next letter in a simple Deterministic Context Free Language that has one level of center-embedding. We demonstrate how the network develops simple coordination between trajectories that enable it to perform limited counting, and in some cases generalize to longer strings. We geometrically identify and analyze several properties relevant for this task, including information loss that results from approaching attractors, divergence in phase space that is used to split states, and difficulty in learning temporal dependencies when the input-output probabilities overlap for different input symbols.
