Massachusetts Institute of Technology Department of Electrical Engineering and Computer Science 6.035, Spring 2010 Handout — Scanner-Parser Project Wednesday, Feb 3 DUE: Tuesday, Feb 16 This project consists of two segments: lexical analysis (scanni ng) and syntactic analysis (parsing). Preliminaries In this section we will describe the infrastructure that is provided for the pr oject. You are encour-aged to use it, but you may also choose to ignore it and design your infrast r uc tu re from scratch. The skeleton is located in /mit/6.035/provided/skeleton. Provided Infrastructure A skeleton compiler infrastructure has been provided that includes a typical compiler directory organization, the ANTLR parser and sc anne r generator, and an Apache Ant build system. The Java package name for the sources provided is decaf. The directory structure and the provided files are as follows: . |-- bin |-- build.xml |-- lib | | | ‘-- antlr.jar ‘-- src | |-- decaf | | | |-- Lexer.g | |-- Parser.g | ‘-- Main.java ‘-- java6035 | ‘-- tools | ‘-- CLI | ‘-- CLI.java The lib directory in cl ud es libraries that are needed during a run of your compiler; ri ght now it contains the ANTLR tool. The bin directory contains the tools (executables) needed to com-pile your compiler. However, since ANTLR uses the same binary as a tool and as a library, and we already included it in the lib directory, bin director is empty. build.xml is the ant build file for the infrastructur e . The src directory contains all your source code for the compiler. src/decaf/Parser.g is a skeleton par se r grammar and src/decaf/Lexer.g is a skeleton scanner 1grammar. src/decaf/Main.java is the skeleton driver and src/java6035/tools/CLI/CLI.java is the command-line interface library. To access and build the provided infrastruct ur e , you must add the following lockers to your file system. add 6.035 add java_v1.6.0_18 add gnu add sipb add git add eclipse-sdk The eclipse-sdk locker is optional and it adds the eclipse IDE. No changes to your $CLASSPATH vari-able are required, but due to an interaction betwee n sipb and java v1.6.0 18, we recommend that you set an envi r onme nt variable JAVA HOME to /afs/athena.mit.edu/software/java v1.6.0 18/; all the classes and jar files required for compilation are included by the Ant build file (se e next Section). Ant Build System Please re v i e w the Ant build file, build.xml, that is provided. For more information on Ant, please visit: http://ant.apache.org/ To build the system, execute ant from the root directory of the system. The build file include s tasks for running ANTLR on your scanner and parser grammars, compiling the sources, packaging the compiler into a jar file, and cl e aning the infrastructure. The default rule is for complete compilation and jar creation. The build file creates three directories during compilation: classes, java, and dist. The classes directory contains all .class files created during compilation. The java directory contains all the generated sources for the compiler (from scanner and parser grammars). The dist directory contains t he jar archive fil e for the compiler and any other runtime libraries that are necessary for runni ng the compiler. A clean will delete these three director i e s. If you are using revisi on cont r ol , you should not add these directories to your repository. Getting Started A good place t o start would be to create a copy of the provided skel e t on and create a git repository to track your changes. This can be accomplished with the following commands on athena: # initial setup: add 6.035 java_v1.6.0_18 gnu sipb git cp -r /mit/6.035/provided/skeleton ~/Private/6.035 cd ~/Private/6.035 git init 2 *Athena is MIT's UNIX-based computing environment. OCW does not provide access to it.# save all changes into git repository: git add . git commit -m "initial commit" # build: ant # run scanner: java -jar dist/Compiler.jar -target scan -debug /mit/6.035/provided/scanner/char1 # run parser: java -jar dist/Compiler.jar -target parse -debug /mit/6.035/provided/parser/legal-01 Scanner Your scanner must be able to identify tokens of the Decaf language, the simple imperative language we will be compiling in 6.035. The language is d es cr i bed in the Decaf Language Handout. Your scanner should note illegal characters, missing quotation marks, and other lexical errors with rea-sonable error messages. The scanner should find as many le x i cal errors as possible, and should be able to continue scanning after errors are found. The scanner should also filter out comments and whitespace not in string and character literals. You will not be writin g the scanner from scratch. Instead, you will generate the scanner using ANTLR. This program reads in an input specification of regular expression-like syntax (gramm ar) and creates a Java program to scan the specified language. More information on ANTLR (including the manual and examples) can be on the web at: http://antlr2.org/doc/index.html http://www.antlr.org/wiki/display/CS652/CS652+Home To get you started, we have provided a template in /mit/6.035/provided/skeleton/src/decaf/Lexer.g If you chose to work from this skeleton, you must complete the existing ANTLR rules and add new ones for your scanner. ANTLR generated scanners throw e x c ept i ons when they encounter an error. Each exception in-cludes a text of a potential error message, and the provided skeleton driver prints them out. You are free to use use the messages provided by ANTLR, if you have verified that they make sense and are specific enough. ANTLR is invoked by t he antlr task of the Ant build file. The generated files, DecafLexer.java, etc, are placed in the java/decaf director y by the task. Note that ANTLR merely generates a Java source file for a scanner class; it does not compile or even syntactically check