61A Lecture 35Monday, 28th November, 2011Monday, November 28, 2011Last time: sequential data and iteratorsSequencesThe sequence abstraction so farLengthElement selection•Lists and tuplesStore all elements up-frontcan’t deal with huge datacan’t deal with infinite sequencesIteratorsStore how to compute elementsCompute one element at a timeDelay evaluation2Monday, November 28, 2011Last time: sequential data and iteratorsStreams -- a unit of delayed evaluation. 2 elements, first and rest.“first” is stored“compute_rest” is storedcalculate “rest” on demandNative python iterator interface__iter__()__next__()for-loops rely on these methodsGenerator functionsFunctions that use yield to output valuesCreates a generator object__iter__() and __next__() automatically defined3Monday, November 28, 2011Today: modularity, processing pipelines, and coroutinesModularity in programs so farHelper functions a.k.a “subroutines”Coroutines: what are they?Coroutines in pythonTypes of coroutinesMultitasking4Monday, November 28, 2011Modularity so far: helper functions5Main functionsubroutinesubroutinesubroutinesubroutinesubroutineModularity in programming?Helper functions!•a.k.a. “subroutines”A sub-program responsible for a small piece of computationA main function is responsible for calling all the subroutinesMonday, November 28, 2011Modularity with CoroutinesCoroutines are also sub-computationsThe difference: no main functionSeparate coroutines link together to form a complete pipeline6coroutinecoroutinecoroutinecoroutinecoroutineMonday, November 28, 2011Coroutines vs. subroutines: a conceptual difference7coroutinecoroutinecoroutinecoroutinecoroutineMain functionsubroutinesubroutinesubroutinesubroutinesubroutinesubordinate to a main functioncolleagues that cooperateMonday, November 28, 2011Coroutines in python, or, the many faces of “yield”Previously: generator functionsProduce data with yield8def letters_generator(): current = 'a' while current <= 'd': yield current current = chr(ord(current)+1)pauses executionlocal variables preservedresumes when .__next__ is calledreturns the yielded valueNow: coroutinesConsume data with yieldvalue = (yield)pauses executionlocal variables preservedresumes when .send(data) is calledassigns value to yielded datasend(data)value = (yield)(yield) returns the sent data. Execution resumesMonday, November 28, 2011Coroutines in PythonConsuming data with yield:value = (yield)Execution pauses waiting for data to be sentSend a coroutine data using send(...)Start a coroutine using ___next__() Signal the end of a computation using close()•Raises GeneratorExit exception inside coroutine9Monday, November 28, 2011Example: print out strings that match a pattern10def match(pattern): print('Looking for ' + pattern) try: while True: s = (yield) if pattern in s: print(s) except GeneratorExit: print("=== Done ===")Step 2: Start with __next__()>>> m.__next__()Step 3: Send data>>> m.send(“the Jabberwock with eyes of flame”)Step 1: Initialize>>> m = match(“Jabberwock”)does nothingcreates a new objectstops here, waitingfor dataexecution starts‘Looking for Jabberwock’resumes heres = “the Jabberwock ...”match found‘the Jabberwock with eyes of flame’Step 4: close the coroutine>>> m.close()catch exception‘=== Done ===’Monday, November 28, 2011Pipelines: the power of coroutines11coroutinecoroutinecoroutinecoroutinecoroutineWe can chain coroutines together to achieve complex behaviorsCreate a pipelineCoroutines send data to others downstreamMonday, November 28, 2011A simple pipeline12match wordsread wordsMonday, November 28, 2011A simple pipeline: reading words13match wordsread wordsmatch wordsread words def read(text, next_coroutine): for word in text.split(): next_coroutine.send(word) next_coroutine.close()needs to know where to send()loop(yield) -- wait for next sendfor loopfor word in text.split(): next_coroutine.send(word)readvalue = (yield)next_coroutinesend -- activate (yield)Monday, November 28, 2011while True:line = (yield)if pattern in line:print(line)match14while loop(yield) -- wait for next sendfor loopfor word in text.split(): next_coroutine.send(word)readsend -- activate (yield)match wordsread wordsA simple pipelineMonday, November 28, 2011A simple pipeline15matcher‘ending’readtext = ‘Comm>>> matcher = match('ending')>>> matcher.__next__()‘Looking for ending’line = (yield)paused>>> text = 'Commending spending is offending to people pending lending!'>>> read(text, matcher)for word in text.split(): next_coroutine.send(word)CommendingCommending‘Commending’spendingspendingisisoffendingoffending‘spending’‘offending’‘pending’‘lending!’next_coroutine.close()pausedpausedpaused‘=== Done ===’closedGeneratorExitlast word!Monday, November 28, 2011Produce, Filter, Consume16producersendfilter(yield)send...filter(yield)sendconsumer(yield)Coroutines can have different roles in a pipelineBased on how they use send() and yieldThe producer only sends dataThe filter consumes with (yield)and sends results downstreamThe consumer only consumes dataThere can be many layers of filtersMonday, November 28, 201117readtext = ‘CommExample: simple pipeline def read(text, next_coroutine): for word in text.split(): next_coroutine.send(word) next_coroutine.close()matcher‘ending’def match(pattern): print('Looking for ' + pattern) try: while True: s = (yield) if pattern in s: print(s) except GeneratorExit: print("=== Done ===")Producer ConsumerMonday, November 28, 2011Breaking down match18match wordsread wordsProducer Consumerprintfind matchesfilter consumerMonday, November 28, 2011Breaking down match19printfind matchesfilterconsumer def match_filter(pattern, next_coroutine): print('Looking for ' + pattern) try: while True: s = (yield) if pattern in s: next_coroutine.send(s) except GeneratorExit: next_coroutine.close() def print_consumer(): print('Preparing to print') try: while True: line = (yield) print(line) except GeneratorExit: print("=== Done ===")>>> printer = print_consumer()>>> printer.__next__()‘Preparing to print’>>> matcher = match_filter('pend', printer)>>> matcher.__next__()‘Looking for pend’>>> text = 'Commending spending is offending'>>> read(text, matcher)‘spending’‘=== Done
View Full Document
Unlocking...