YOLTOverview of YOLTA Useful YOLT ProgramSemanticsSemantics Lessons LearnedCode GenerationImplementationImplementation (tricks)Documentation and TestingIntegration TestingThe ResultLessons LearnedOne More ExampleYOLTYOLTYYuan Zhenguan ZhengOOmar Ahmedmar AhmedLLukas Dudkowskiukas DudkowskiTT. Mark Kuba. Mark KubaOverview of YOLTOverview of YOLTSimple scripting languageSimple scripting languageEasy for coding and maintenance.Easy for coding and maintenance.Regular expression supportRegular expression support:= and @:= and @““Web-scraping” usesWeb-scraping” usesNatural Language ProcessingNatural Language ProcessingGenerating RSS FeedsGenerating RSS FeedsReformatting HTML for other uses (XML,etc)Reformatting HTML for other uses (XML,etc)A Useful YOLT ProgramA Useful YOLT ProgramSemanticsSemanticsYOLT Semantic checker is extremely simple. It YOLT Semantic checker is extremely simple. It serves a few main tasks:serves a few main tasks:Make sure that functions are declared properly, i.e. Make sure that functions are declared properly, i.e. function declarations match functions, and function function declarations match functions, and function calls match the declarationscalls match the declarationsMake sure that variables are initialized before they are Make sure that variables are initialized before they are used (or, in some cases, un-initialized)used (or, in some cases, un-initialized)(redundant) Make sure that the tree is properly (redundant) Make sure that the tree is properly formed (i.e. make sure that an if-then-else node has formed (i.e. make sure that an if-then-else node has exactly three children, etc)exactly three children, etc)*note*: there was once basic type-checking, but no *note*: there was once basic type-checking, but no longer.longer.Semantics Lessons LearnedSemantics Lessons LearnedIt is very easy to do too much in semantic It is very easy to do too much in semantic checkingcheckingEither there are types, or no types (NO MIDDLE Either there are types, or no types (NO MIDDLE GROUND)GROUND)Scripting languages are an enormous relief to a Scripting languages are an enormous relief to a semantic checker--they take away the biggest semantic checker--they take away the biggest hassleshasslesThe tree walker should know EXACTLY what the The tree walker should know EXACTLY what the structure of the AST will look like and cannot structure of the AST will look like and cannot make ANY assumptions--things, as evident, can make ANY assumptions--things, as evident, can break down when you least expect them to.break down when you least expect them to.Code GenerationCode GenerationWritten in JavaWritten in JavaInput: correct AST Input: correct AST Output: Perl programOutput: Perl programAST Code generator Perl ProgramJavaImplementationImplementationWalk ASTWalk ASTAccording to the information of the According to the information of the node, generate code or go down to the node, generate code or go down to the child node child node e.g.:e.g.::=:=$a $a http://www.columbia.eduhttp://www.columbia.eduGo down to the tree at node “:=“Go down to the tree at node “:=“Generate code at node “$a” and Generate code at node “$a” and “http://www.columbia.edu”“http://www.columbia.edu”Implementation Implementation (tricks)(tricks)The httpget :=The httpget :=invoke UNIX system call “wget” to download the web page into invoke UNIX system call “wget” to download the web page into a temp filea temp fileRead the file line by line and store them into an perl array Read the file line by line and store them into an perl array Invoke another UNIX system call “rm” to remove the temp fileInvoke another UNIX system call “rm” to remove the temp fileKeep the web address in an perl scalarKeep the web address in an perl scalarScalar and arrays use same syntaxScalar and arrays use same syntaxCompiler (code generator) “guesses” whether the variable is a Compiler (code generator) “guesses” whether the variable is a scalar or an arrayscalar or an arrayArrays can only appears in certain places (e.g.. Foreach)Arrays can only appears in certain places (e.g.. Foreach)Documentation and TestingDocumentation and TestingLexer/Parser - Semantic CheckerLexer/ParserSemantic CheckerReference File:What I think itshould produceDifLog result:Good should be good.Bad should be bad.Test Cases●Good●BadIntegration TestingIntegration TestingGoal: display any comics that have the Goal: display any comics that have the word word hamster in the URL of hamster in the URL of www.toothpastefordinner.com, Summer www.toothpastefordinner.com, Summer 2002 archive.2002 archive.Trying little YOLT programs to see functionality, code generation, etc. Working out bugs in implementation & design.Example: <img src="http://www.toothpastefordinner.com/072802/hamster-table-tennis.gif"><br><img src="http://www.toothpastefordinner.com/072502/even-hamsters.gif"><br><img src="http://www.toothpastefordinner.com/060602/hamsters-are-the-best.gif"><br>Yolt ProgramGenerated PerlResultant HTML$toothpaste_home ="http://www.toothpastefordinner.com/";system('wget -q -O - http://www.toothpastefordinner.com/archives-sum02.php > toothpaste.txt');open INFILE, "toothpaste.txt";@toothpaste=<INFILE>;close INFILE;system ('rm toothpaste.txt');$toothpaste = "http://www.toothpastefordinner.com/archives-sum02.php";$tags ="<a href=\"(.*)\">.*hamster.*</a>";@tmp1=();foreach ( @toothpaste) {if ($_=~m/($tags)/i){push @tmp1, $2}}@elements = @tmp1;foreach $x ( @elements ) {print "<img src=\"".$toothpaste_home.$x."\""."><br>";print "\n";}begin$toothpaste_home="http://www.toothpastefordinner.com/";$toothpaste:="http://www.toothpastefordinner.com/archives-sum02.php";$tags="<a href=\"(.*)\">.*hamster.*</a>";$elements = $tags @ $toothpaste;foreach $x in $elements { echo "<img src=\"".$toothpaste_home.$x."\""."><br>"; echo "\n";}endThe ResultThe ResultThe source siteThe end resultLessons LearnedLessons LearnedDevelop and test incrementallyDevelop and test incrementallyThere are ALWAYS bugs, you just haven’t There are ALWAYS bugs, you just haven’t found them yetfound them yetCLIC is not designed to be lived inCLIC is not designed to be lived inOne More ExampleOne More
View Full Document
Unlocking...