DOC PREVIEW
UT Dallas CS 6350 - 06.DatabaseMapReduceBigData#4

This preview shows page 1-2-17-18-19-35-36 out of 36 pages.

Save
View full document
View full document
Premium Document
Do you want full access? Go Premium and unlock all 36 pages.
Access to all documents
Download any document
Ad free experience
View full document
Premium Document
Do you want full access? Go Premium and unlock all 36 pages.
Access to all documents
Download any document
Ad free experience
View full document
Premium Document
Do you want full access? Go Premium and unlock all 36 pages.
Access to all documents
Download any document
Ad free experience
View full document
Premium Document
Do you want full access? Go Premium and unlock all 36 pages.
Access to all documents
Download any document
Ad free experience
View full document
Premium Document
Do you want full access? Go Premium and unlock all 36 pages.
Access to all documents
Download any document
Ad free experience
View full document
Premium Document
Do you want full access? Go Premium and unlock all 36 pages.
Access to all documents
Download any document
Ad free experience
View full document
Premium Document
Do you want full access? Go Premium and unlock all 36 pages.
Access to all documents
Download any document
Ad free experience
Premium Document
Do you want full access? Go Premium and unlock all 36 pages.
Access to all documents
Download any document
Ad free experience

Unformatted text preview:

MapReduce algorithms for processing relational dataDesign Pattern: Secondary SortingSecondary Sorting: SolutionsValue-to-Key ConversionWorking ScenarioRelational AlgebraProjectionProjection in MapReduceSelectionSelection in MapReduceGroup by… AggregationSlide 12Relational JoinsNatural Join Operation – ExampleNatural Join ExampleTypes of RelationshipsJoin Algorithms in MapReduceReduce-side JoinReduce-side Join: 1-to-1Reduce-side Join: 1-to-manyReduce-side Join: V-to-K ConversionReduce-side Join: many-to-manyReduce-side Join: many-to-manyReduce-side Join: many-to-manyMap-side Join: Basic IdeaMap-side Join: Parallel ScansMap-side Join: Parallel ScansMap-side Join: Parallel ScansParallel Scan & JoinIn-Memory JoinIn-Memory JoinIn-Memory Join: VariantsMemcachedMemcached JoinWhich join to use?Processing Relational Data: SummaryMapReduce algorithms for processing relational dataDesign Pattern: Secondary Sorting•MapReduce sorts input to reducers by key–Values are arbitrarily ordered•What if want to sort value also?–E.g., k → (v1, r), (v3, r), (v4, r), (v8, r)…Secondary Sorting: Solutions•Solution 1:–Buffer values in memory, then sort–Why is this a bad idea?•Solution 2:–“Value-to-key conversion” design pattern: form composite intermediate key, (k, v1)–Let execution framework do the sorting–Preserve state across multiple key-value pairs to handle processing–Anything else we need to do?Value-to-Key Conversionk → (v1, r), (v4, r), (v8, r), (v3, r)…(k, v1) → (v1, r)BeforeAfter(k, v3) → (v3, r)(k, v4) → (v4, r)(k, v8) → (v8, r)Values arrive in arbitrary order……Values arrive in sorted order…Process by preserving state across multiple keysRemember to partition correctly!Working Scenario•Two tables:–User demographics (gender, age, income, etc.)–User page visits (URL, time spent, etc.)•Analyses we might want to perform:–Statistics on demographic characteristics–Statistics on page visits–Statistics on page visits by URL–Statistics on page visits by demographic characteristic–…Relational Algebra•Primitives–Projection ()–Selection ()–Cartesian product ()–Set union ()–Set difference ()–Rename ()•Other operations–Join ( )⋈–Group by… aggregation–…Projection R1R2R3R4R5R1R2R3R4R5Projection in MapReduce•Easy!–Map over tuples, emit new tuples with appropriate attributes–No reducers, unless for regrouping or resorting tuples–Alternatively: perform in reducer, after some other processing•Basically limited by HDFS streaming speeds–Speed of encoding/decoding tuples becomes important–Relational databases take advantage of compression–Semistructured data? No problem!SelectionR1R2R3R4R5R1R3Selection in MapReduce•Easy!–Map over tuples, emit only tuples that meet criteria–No reducers, unless for regrouping or resorting tuples–Alternatively: perform in reducer, after some other processing•Basically limited by HDFS streaming speeds–Speed of encoding/decoding tuples becomes important–Relational databases take advantage of compression–Semistructured data? No problem!Group by… Aggregation•Example: What is the average time spent per URL?•In SQL:–SELECT url, AVG(time) FROM visits GROUP BY url•In MapReduce:–Map over tuples, emit time, keyed by url–Framework automatically groups values by keys–Compute average in reducer–Optimize with combinersRelational JoinsSource: Microsoft Office Clip ArtRelational JoinsR1R2R3R4S1S2S3S4R1S2R2S4R3S1R4S3Natural Join Operation – Example•Relations r, s:A B12412C DaababB13123DaaabbErA B11112C DaaaabEsr sNatural Join ExampleR1S1R1 S1 =sidsnameratingagebidday22dustin745.010110/10/9658rusty1035.010311/12/96sid sname rating age 22 dustin 7 45.0 31 lubber 8 55.5 58 rusty 10 35.0 sid bid day 22 101 10/10/96 58 103 11/12/96Types of RelationshipsOne-to-OneOne-to-ManyMany-to-ManyJoin Algorithms in MapReduce•Reduce-side join•Map-side join•In-memory join–Striped variant–Memcached variantReduce-side Join•Basic idea: group by join key–Map over both sets of tuples–Emit tuple as value with join key as the intermediate key–Execution framework brings together tuples sharing the same key–Perform actual join in reducer–Similar to a “sort-merge join” in database terminology•Two variants–1-to-1 joins–1-to-many and many-to-many joinsReduce-side Join: 1-to-1R1R4S2S3R1R4S2S3keys valuesMapR1R4S2S3keys valuesReduceNote: no guarantee if R is going to come first or SReduce-side Join: 1-to-manyR1S2S3R1S2S3S9keys valuesMapR1S2keys valuesReduceS9S3…What’s the problem?Reduce-side Join: V-to-K ConversionR1keys valuesIn reducer…S2S3S9R4S3S7New key encountered: hold in memoryCross with records from other setNew key encountered: hold in memoryCross with records from other setReduce-side Join: many-to-manyR1keys valuesIn reducer…S2S3S9Hold in memoryCross with records from other setR5R8What’s the problem?Reduce-side Join: many-to-manyProduce mapper output with composite key that includes foreign key and table nameReduce-side Join: many-to-manyUse custom partitioning and grouping to send data with same key to a single reducerMap-side Join: Basic IdeaAssume two datasets are sorted by the join key:R1R2R3R4S1S2S3S4A sequential scan through both datasets to join(called a “merge join” in database terminology)Map-side Join: Parallel Scans•If datasets are sorted by join key, join can be accomplished by a scan over both datasets•How can we accomplish this in parallel?–Partition and sort both datasets in the same manner•In MapReduce:–Map over one dataset, read from other corresponding partition–No reducers necessary (unless to repartition or resort)•Consistently partitioned datasets: realistic to expect?Map-side Join: Parallel ScansSort and split both A and B before sending to mapper. Mapper will produce output, no reducer needed.Map-side Join: Parallel ScansParallel Scan & JoinIn-Memory Join•Basic idea: load one dataset into memory, stream over other dataset–Works if R << S and R fits into memory–Called a “hash join” in database terminology•MapReduce implementation–Distribute R to all nodes–Map over S, each mapper loads R in memory, hashed by join key–For every tuple in S, look up join key in R–No reducers, unless for regrouping or resorting tuplesIn-Memory JoinSplit A and distribute


View Full Document

UT Dallas CS 6350 - 06.DatabaseMapReduceBigData#4

Documents in this Course
HW3

HW3

5 pages

NOSQL-CAP

NOSQL-CAP

23 pages

BigTable

BigTable

39 pages

HW3

HW3

5 pages

Load more
Download 06.DatabaseMapReduceBigData#4
Our administrator received your request to download this document. We will send you the file to your email shortly.
Loading Unlocking...
Login

Join to view 06.DatabaseMapReduceBigData#4 and access 3M+ class-specific study document.

or
We will never post anything without your permission.
Don't have an account?
Sign Up

Join to view 06.DatabaseMapReduceBigData#4 2 2 and access 3M+ class-specific study document.

or

By creating an account you agree to our Privacy Policy and Terms Of Use

Already a member?