61A Lecture 35Monday, 28th November, 2011Last time: sequential data and iteratorsSequences!The sequence abstraction so far!Length!Element selection•Lists and tuples!Store all elements up-front!can’t deal with huge data!can’t deal with infinite sequencesIterators!Store how to compute elements!Compute one element at a time!Delay evaluation2Last time: sequential data and iteratorsStreams -- a unit of delayed evaluation. !2 elements, first and rest.!“first” is stored!“compute_rest” is stored!calculate “rest” on demandNative python iterator interface!__iter__()!__next__()!for-loops rely on these methodsGenerator functions!Functions that use yield to output values!Creates a generator object!__iter__() and __next__() automatically defined3Today: modularity, processing pipelines, and coroutinesModularity in programs so far!Helper functions a.k.a “subroutines”Coroutines: what are they?Coroutines in pythonTypes of coroutinesMultitasking4Modularity so far: helper functions5Main functionsubroutinesubroutinesubroutinesubroutinesubroutineModularity in programming?!Helper functions!•a.k.a. “subroutines”!A sub-program responsible for a small piece of computationA main function is responsible for calling all the subroutinesModularity with CoroutinesCoroutines are also sub-computationsThe difference: no main functionSeparate coroutines link together to form a complete pipeline6coroutinecoroutinecoroutinecoroutinecoroutineCoroutines vs. subroutines: a conceptual difference7coroutinecoroutinecoroutinecoroutinecoroutineMain functionsubroutinesubroutinesubroutinesubroutinesubroutinesubordinate to a main functioncolleagues that cooperateCoroutines in python, or, the many faces of “yield”Previously: generator functions!Produce data with yield8def letters_generator(): current = 'a' while current <= 'd': yield current current = chr(ord(current)+1)pauses executionlocal variables preservedresumes when .__next__ is calledreturns the yielded valueNow: coroutines!Consume data with yieldvalue = (yield)pauses executionlocal variables preservedresumes when .send(data) is calledassigns value to yielded datasend(data)value = (yield)(yield) returns the sent data. Execution resumesCoroutines in PythonConsuming data with yield:!value = (yield)!Execution pauses waiting for data to be sentSend a coroutine data using send(...)Start a coroutine using ___next__() Signal the end of a computation using close()•Raises GeneratorExit exception inside coroutine9Example: print out strings that match a pattern10def match(pattern): print('Looking for ' + pattern) try: while True: s = (yield) if pattern in s: print(s) except GeneratorExit: print("=== Done ===")Step 2: Start with __next__()>>> m.__next__()Step 3: Send data>>> m.send(“the Jabberwock with eyes of flame”)Step 1: Initialize>>> m = match(“Jabberwock”)does nothingcreates a new objectstops here, waitingfor dataexecution starts‘Looking for Jabberwock’resumes heres = “the Jabberwock ...”match found‘the Jabberwock with eyes of flame’Step 4: close the coroutine>>> m.close()catch exception‘=== Done ===’Pipelines: the power of coroutines11coroutinecoroutinecoroutinecoroutinecoroutineWe can chain coroutines together to achieve complex behaviorsCreate a pipelineCoroutines send data to others downstreamA simple pipeline12match wordsread wordsA simple pipeline: reading words13match wordsread wordsmatch wordsread words def read(text, next_coroutine): for word in text.split(): next_coroutine.send(word) next_coroutine.close()needs to know where to send()loop(yield) -- wait for next sendfor loopfor word in text.split(): next_coroutine.send(word)readvalue = (yield)next_coroutinesend -- activate (yield)while True:line = (yield)if pattern in line:print(line)match14while loop(yield) -- wait for next sendfor loopfor word in text.split(): next_coroutine.send(word)readsend -- activate (yield)match wordsread wordsA simple pipelineA simple pipeline15matcher‘ending’readtext = ‘Comm>>> matcher = match('ending')>>> matcher.__next__()‘Looking for ending’line = (yield)paused>>> text = 'Commending spending is offending to people pending lending!'>>> read(text, matcher)for word in text.split(): next_coroutine.send(word)CommendingCommending‘Commending’spendingspendingisisoffendingoffending‘spending’‘offending’‘pending’‘lending!’next_coroutine.close()pausedpausedpaused‘=== Done ===’closedGeneratorExitlast word!Produce, Filter, Consume16producersendfilter(yield)send...filter(yield)sendconsumer(yield)Coroutines can have different roles in a pipelineBased on how they use send() and yieldThe producer only sends dataThe filter consumes with (yield)and sends results downstreamThe consumer only consumes dataThere can be many layers of filters17readtext = ‘CommExample: simple pipeline def read(text, next_coroutine): for word in text.split(): next_coroutine.send(word) next_coroutine.close()matcher‘ending’def match(pattern): print('Looking for ' + pattern) try: while True: s = (yield) if pattern in s: print(s) except GeneratorExit: print("=== Done ===")Producer ConsumerBreaking down match18match wordsread wordsProducer Consumerprintfind matchesfilter consumerBreaking down match19printfind matchesfilterconsumer def match_filter(pattern, next_coroutine): print('Looking for ' + pattern) try: while True: s = (yield) if pattern in s: next_coroutine.send(s) except GeneratorExit: next_coroutine.close() def print_consumer(): print('Preparing to print') try: while True: line = (yield) print(line) except GeneratorExit: print("=== Done ===")>>> printer = print_consumer()>>> printer.__next__()‘Preparing to print’>>> matcher = match_filter('pend', printer)>>> matcher.__next__()‘Looking for pend’>>> text = 'Commending spending is offending'>>> read(text, matcher)‘spending’‘=== Done ===’Multitasking20coroutinecoroutinecoroutinecoroutinecoroutineWe do not need to be restricted to just one next stepRead-to-many21 def read_to_many(text, coroutines): for word in text.split(): for coroutine in coroutines: coroutine.send(word) for coroutine in coroutines: coroutine.close() def read(text, next_coroutine): for word in text.split(): next_coroutine.send(word) next_coroutine.close()coroutinereadcoroutineread_to_manycoroutineMatching
View Full Document
Unlocking...