DOC PREVIEW
UT PSY 394U - Finding Structure in Time

This preview shows page 1-2-3-26-27-28 out of 28 pages.

Save
View full document
View full document
Premium Document
Do you want full access? Go Premium and unlock all 28 pages.
Access to all documents
Download any document
Ad free experience
View full document
Premium Document
Do you want full access? Go Premium and unlock all 28 pages.
Access to all documents
Download any document
Ad free experience
View full document
Premium Document
Do you want full access? Go Premium and unlock all 28 pages.
Access to all documents
Download any document
Ad free experience
View full document
Premium Document
Do you want full access? Go Premium and unlock all 28 pages.
Access to all documents
Download any document
Ad free experience
View full document
Premium Document
Do you want full access? Go Premium and unlock all 28 pages.
Access to all documents
Download any document
Ad free experience
View full document
Premium Document
Do you want full access? Go Premium and unlock all 28 pages.
Access to all documents
Download any document
Ad free experience
Premium Document
Do you want full access? Go Premium and unlock all 28 pages.
Access to all documents
Download any document
Ad free experience

Unformatted text preview:

Page 1COGNITIVE SCIENCE, 14, 179-211 (1990). Finding Structure in TimeJEFFREY L. ELMANUniversity of California, San DiegoTime underlies many interesting human behaviors. Thus, the question ofhow to represent time in connectionist models is very important. Oneapproach is to represent time implicitly by its effects on processing ratherthan explicitly (as in a spatial representation). The current report developsa proposal along these lines first described by Jordan (1986) whichinvolves the use of recurrent links in order to provide networks with adynamic memory. In this approach, hidden unit patterns are fed back tothemselves; the internal representations which develop thus reflect taskdemands in the context of prior internal states. A set of simulations isreported which range from relatively simple problems (temporal versionof XOR) to discovering syntactic/semantic features for words. Thenetworks are able to learn interesting internal representations whichincorporate task demands with memory demands; indeed, in this approachthe notion of memory is inextricably bound up with task processing. Theserepresentations reveal a rich structure, which allows them to be highlycontext-dependent while also expressing generalizations across classes ofitems. These representations suggest a method for representing lexicalcategories and the type/token distinction.___________________________I would like to thank Jay McClelland, Mike Jordan, Mary Hare, Dave Rumelhart, MikeMozer, Steve Poteet, David Zipser, and Mark Dolson for many stimulating discussions. I thankMcClelland, Jordan, and two anonymous reviewers for helpful critical comments on an earlierdraft of this paper. This work was supported by contract N00014-85-K-0076 from the Office of NavalResearch and contract DAAB-07-87-C-H027 from Army Avionics, Ft. Monmouth. Requests forreprints should be sent to the Center for Research in Language, C-008; University of California,San Diego, CA 92093-0108. The author can be reached via electronic mail [email protected] 2IntroductionTime is clearly important in cognition. It is inextricably bound up with many behaviors (such aslanguage) which express themselves as temporal sequences. Indeed, it is difficult to know howone might deal with such basic problems as goal-directed behavior, planning, or causationwithout some way of representing time.The question of how to represent time might seem to arise as a special problem unique toparallel processing models, if only because the parallel nature of computation appears to be atodds with the serial nature of temporal events. However, even within traditional (serial)frameworks, the representation of serial order and the interaction of a serial input or output withhigher levels of representation presents challenges. For example, in models of motor activity animportant issue is whether the action plan is a literal specification of the output sequence, orwhether the plan represents serial order in a more abstract manner (e.g., Lashley, 1951;MacNeilage, 1970; Fowler, 1977, 1980; Kelso, Saltzman, & Tuller, 1986; Saltzman & Kelso,1987; Jordan & Rosenbaum, 1988). Linguistic theoreticians have perhaps tended to be lessconcerned with the representation and processing of the temporal aspects to utterances(assuming, for instance, that all the information in an utterance is somehow made availablesimultaneously in a syntactic tree); but the research in natural language parsing suggests that theproblem is not trivially solved (e.g., Frazier & Fodor; 1978; Marcus, 1980). Thus, what is one ofthe most elementary facts about much of human activity -that it has temporal extent -issometimes ignored and is often problematic.In parallel distributed processing models, the processing of sequential inputs has beenaccomplished in several ways. The most common solution is to attempt to “parallels time” bygiving it a spatial representation. However, there are problems with this approach, and it isultimately not a good solution. A better approach would be to represent time implicitly ratherthan explicitly. That is, we represent time by the effect it has on processing and not as anadditional dimension of the input.This paper describes the results of pursuing this approach, with particular emphasis onproblems that are relevant to natural language processing. The approach taken is rather simple,but the results are sometimes complex and unexpected. Indeed, it seems that the solution to theproblem of time may interact with other problems for connectionist architectures, including theproblem of symbolic representation and how connectionist representations encode structure. Thecurrent approach supports the notion outlined by Van Gelder (1989; see also Smolensky, 1987,1988; Elman, 1989), that connectionist representations may have a functional compositionalitywithout being syntactically compositional.The first section briefly describes some of the problems that arise when time is representedexternally as a spatial dimension. The second section describes the approach used in this work.The major portion of this report presents the results of applying this new architecture to a diverseset of problems. These problems range in complexity from a temporal version of the Exclusive-OR function to the discovery of syntactic/semantic categories in natural language data.The Problem with TimeOne obvious way of dealing with patterns that have a temporal extent is to represent timeexplicitly by associating the serial order of the pattern with the dimensionality of the patternPage 3vector. The first temporal event is represented by the first element in the pattern vector, thesecond temporal event is represented by the second position in the pattern vector; and so on. Theentire pattern vector is processed in parallel by the model. This approach has been used in avariety of models (e.g., Cottrell, Munro, & Zipser, 1987; Elman & Zipser, 1988; Hanson & Kegl,1987).There are several drawbacks to this approach, which basically uses a spatial metaphor fortime. First, it requires that there be some interface with the world which buffers the input so thatit can be presented all at once. It is not clear that biological systems make use of such shiftregisters. There are also logical problems; how should a system know when a buffer’s contentsshould be examined?Second, the shift-register imposes a rigid limit on the duration of patterns (since the inputlayer must provide for the longest possible pattern), and


View Full Document

UT PSY 394U - Finding Structure in Time

Documents in this Course
Roadmap

Roadmap

6 pages

Load more
Download Finding Structure in Time
Our administrator received your request to download this document. We will send you the file to your email shortly.
Loading Unlocking...
Login

Join to view Finding Structure in Time and access 3M+ class-specific study document.

or
We will never post anything without your permission.
Don't have an account?
Sign Up

Join to view Finding Structure in Time 2 2 and access 3M+ class-specific study document.

or

By creating an account you agree to our Privacy Policy and Terms Of Use

Already a member?