Introduction to Amazon Web ServicesIntroductionCloud ComputingAmazon Web ServicesAmazon Web ServicesDemo ApplicationArchitectureSequence Assembly in the CloudsCost to assemble to process 4096 FASTA files*Security CredentialsAWS Toolkit for EclipseInstalling AWS Toolkit in EclipseSimple Storage Service (S3)Simple Storage Service (S3)Using S3 as the Data StorageAWS Import/ExportSimple Queue ServiceSimple Queue ServiceUsing the Queue to Schedule JobsSimple Notification Service (SNS)SimpleDBSimpleDBSimpleDBUsing the SimpleDB for monitoring & metadata storageRelational Database Service (RDS)SimpleDB vs RDSElastic Compute ServiceElastic Compute ServiceElastic Compute ServiceSequence Assembly Performance with different EC2 Instance TypesGTM Interpolation performance with different EC2 Instance TypesHPC in AWSCloudWatchAuto ScalingDeploying the Application in EC2AMIRun the ProgramElastic MapReduceEMR bucket namesRunning WordCount using EMRElastic Block Store (EBS)Elastic Load BalancingVirtual Private Cloud (VPC)CloudFrontMechanical TurkThank You!AcknowledgmentsIntroduction toAmazon Web ServicesThilina GunarathneSalsa Group, Indiana University.With contributions from Saliya Ekanayake.Introduction• Fourth Paradigm – Data intensive scientific discovery– DNA Sequencing machines, LHC• Commercial Cloud Platforms– Amazon Web Services– Microsoft Azure Platform–Google AppEngineCloud Computing• On demand computational services over web– Backed by massive commercial infrastructures giving economies of scale– Spiky compute needs of the scientists• Horizontal scaling with no additional cost– Increased throughput• Cloud infrastructure services– Storage, messaging, tabular storage– Cloud oriented services guarantees– Virtually unlimited scalability• Future seems to be CLOUDY!!!Amazon Web Services• Compute– Elastic Compute Service (EC2)– Elastic MapReduce– Auto Scaling• Storage– Simple Storage Service (S3)– Elastic Block Store (EBS)– AWS Import/Export• Messaging– Simple Queue Service (SQS)– Simple Notification Service (SNS)• Database– SimpleDB– Relational Database Service (RDS)• Content Delivery– CloudFront• Networking– Elastic Load Balancing– Virtual Private Cloud• Monitoring– CloudWatch• Workforce– Mechanical TurkAmazon Web Services• Compute– Elastic Compute Service (EC2)– Elastic MapReduce– Auto Scaling• Storage– Simple Storage Service (S3)– Elastic Block Store (EBS)– AWS Import/Export• Messaging– Simple Queue Service (SQS)– Simple Notification Service (SNS)• Database– SimpleDB– Relational Database Service (RDS)• Content Delivery– CloudFront• Networking– Elastic Load Balancing– Virtual Private Cloud• Monitoring– CloudWatch• Workforce– Mechanical TurkDemo Application• Job queue based embarrassingly parallel application execution– BLAST, Monte Carlo simulations, many image processing applications, parametric studies• Cap3 – Sequence Assembly*– Assembles DNA sequences by aligning and merging sequence fragments to construct whole genome sequences• Executable available at http://seq.cs.iastate.edu/cap3.html• Demo programs– http://salsahpc.indiana.edu/tutorial/apps/aws/* Huang, X. and Madan, A. (1999) CAP3: A DNA sequence assembly program. Genome Res., 9, 868-877.ArchitectureSequence Assembly in the CloudsCap3 parallel efficiencyCap3 – Per core per file (458 reads in each file) time to process sequencesCost to assemble to process 4096 FASTA files*• Amazon AWS total :11.19 $Compute 1 hour X 16 HCXL (0.68$ * 16) = 10.88 $10000 SQS messages = 0.01 $Storage per 1GB per month = 0.15 $Data transfer out per 1 GB = 0.15 $• Azure total : 15.77 $Compute 1 hour X 128 small (0.12 $ * 128) = 15.36 $10000 Queue messages = 0.01 $Storage per 1GB per month = 0.15 $Data transfer in/out per 1 GB = 0.10 $ + 0.15 $• Tempest (amortized) : 9.43 $– 24 core X 32 nodes, 48 GB per node– Assumptions : 70% utilization, write off over 3 years, including support*~ 1 GB / 1875968 reads (458 reads X 4096)Security Credentials• Access Keys– Making a REST or Query API request– JAVA SDK for S3, SQS, SimpleDB• EC2 Key Pairs– Launching/connecting to EC2 instances• X.509 Certificate– SOAP API– Command line toolsAWS Toolkit for Eclipse• Open source plug-in for Eclipse• AWS Java SDK– Java API for AWS services• Amazon SimpleDB management– Configure, edit, query• Amazon EC2 management– Deploy, debug, manageInstalling AWS Toolkit in Eclipse• Installing– Java 1.5 or higher– Eclipse 3.5 or higher (Java EE distribution recommended)– http://aws.amazon.com/eclipse–http://media.amazonwebservices.com/videos/eclipse-java-sdk-video.htmlSimple Storage Service (S3)• Internet Data Storage– Reliable, Simple, Scalable, and Inexpensive• Three Concepts– Buckets• Analogous to a folder with no nesting• URL accessible• Option to enforce geographical constraints– Objects• Actual data stored in buckets, e.g. PDF, Video, etc.• Up to 5 gigabytes• Unlimited number of objects• Retrievable via HTTP, HTTPS, or BitTorrent• Private, public or selectively for users– Keys• Unique key to identify each object in a bucketSimple Storage Service (S3)• Access Logs– Option to enable to logs for buckets• Pricing– Data storage• 0.15$ per GB for first 50TB to 0.055$ per GB for over 5000TB– Data transfer in• 0.1$ per GB (free till Nov,2010)– Data Transfer out• 0.15$ per GB up to 10TB to 0.08$ per GB for over 150TB– Requests• PUT, COPY, POST, LIST -> 0.01 $ per 1000 requests• Others -> 0.01$ for 10,000 requests• Reduced Redundant Storage– 2/3 of the storage costUsing S3 as the Data Storage• S3 management console• Uploading the input data to S3• Downloading/uploading files (s3 objects) programmatically• Run Sample– AWSStepOne eclipse projectAWS Import/Export• Accelerates Moving Large Scale Data– In to and out of AWS using portable storage– Utilized Amazon’s high-speed internal network– Often faster than Internet upload/download for large data• Simple Steps– Prepare a portable storage device– Request AWS with S3 bucket, key, and shipping address– Receive an ID, digital signature, an AWS shipping address– Identify and authenticate storage device with digital signature– Ship it and wait for Amazon to ship it back • Data migration, content distribution, offsite backup, disaster recovery, direct data