Slide 1Introduction to Big DataBig Data DefinitionCharacteristics of Big Data: 1-Scale (Volume)Characteristics of Big Data: 2-Complexity (Variety)Characteristics of Big Data: 3-Speed (Velocity)Big Data: 3V’sSome Make it 4V’sHarnessing Big DataWho’s Generating Big DataThe Model Has Changed…What’s driving Big DataValue of Big Data AnalyticsChallenges in Handling Big DataSlide 15Slide 16Big Data TechnologyWhat You Will Learn…Big Data Management and AnalyticsIntroduction Spring 2015Dr. Latifur Khan1Introduction to Big DataWhat is Big Data?What makes data, “Big” Data?2Big Data Definition•No single standard definition…“Big Data” is data whose scale, diversity, and complexity require new architecture, techniques, algorithms, and analytics to manage it and extract value and hidden knowledge from it…3Characteristics of Big Data: 1-Scale (Volume)•Data Volume•44x increase from 2009 2020•From 0.8 zettabytes to 35zb•Data volume is increasing exponentially 4Exponential increase in collected/generated dataCharacteristics of Big Data: 2-Complexity (Variety)•Various formats, types, and structures•Text, numerical, images, audio, video, sequences, time series, social media data, multi-dim arrays, etc…•Static data vs. streaming data •A single application can be generating/collecting many types of data 5To extract knowledge all these types of data need to linked togetherTo extract knowledge all these types of data need to linked togetherCharacteristics of Big Data: 3-Speed (Velocity)•Data is begin generated fast and need to be processed fast•Online Data Analytics•Late decisions missing opportunities•Examples•E-Promotions: Based on your current location, your purchase history, what you like send promotions right now for store next to you•Healthcare monitoring: sensors monitoring your activities and body any abnormal measurements require immediate reaction6Big Data: 3V’s7Some Make it 4V’s8Harnessing Big Data•OLTP: Online Transaction Processing (DBMSs)•OLAP: Online Analytical Processing (Data Warehousing)•RTAP: Real-Time Analytics Processing (Big Data Architecture & technology)9Who’s Generating Big DataSocial media and networks(all of us are generating data)Scientific instruments(collecting all sorts of data) Mobile devices (tracking all objects all the time)Sensor technology and networks(measuring all kinds of data) •The progress and innovation is no longer hindered by the ability to collect data•But, by the ability to manage, analyze, summarize, visualize, and discover knowledge from the collected data in a timely manner and in a scalable fashion10The Model Has Changed…•The Model of Generating/Consuming Data has ChangedOld Model: Few companies are generating data, all others are consuming data New Model: all of us are generating data, and all of us are consuming data 11What’s driving Big Data- Ad-hoc querying and reporting- Data mining techniques- Structured data, typical sources- Small to mid-size datasets- Optimizations and predictive analytics- Complex statistical analysis- All types of data, and many sources- Very large datasets- More of a real-time 12Value of Big Data Analytics•Big data is more real-time in nature than traditional DW applications•Traditional DW architectures (e.g. Exadata, Teradata) are not well-suited for big data apps•Shared nothing, massively parallel processing, scale out architectures are well-suited for big data apps13Challenges in Handling Big Data•The Bottleneck is in technology•New architecture, algorithms, techniques are needed•Also in technical skills•Experts in using the new technology and dealing with big data 14What Technology Do We HaveFor Big Data ??1516Big Data Technology17What You Will Learn…•We focus on Hadoop/MapReduce technology, NoSQL (key-Value), BigTable, Cassandra, SPARK, Stream…•Learn the platform (how it is designed and works)•How big data are managed in a scalable, efficient way•Learn writing Hadoop jobs/SPARK in different languages •Programming Languages: Java, C, Python, Scala •High-Level Languages: Apache Pig, Hive, CQL•Learn advanced analytics tools on top of Hadoop•Mahout: Data mining and machine learning tools over big data•Applications: Recommendations•Graph Processing: Pregel
View Full Document
Unlocking...