140CS 536 Spring 2007©The JLex specification file is:import java_cup.runtime.*;/* Expand this into your solution forproject 2 */class CSXToken {int linenum;int colnum;CSXToken(int line,int col){linenum=line;colnum=col;};}class CSXIntLitToken extends CSXToken {int intValue;CSXIntLitToken(int val,int line,int col){super(line,col);intValue=val;};}class CSXIdentifierToken extendsCSXToken {String identifierText;CSXIdentifierToken(String text,int line,int col){super(line,col);identifierText=text;};}141CS 536 Spring 2007©class CSXCharLitToken extends CSXToken {char charValue;CSXCharLitToken(char val,int line,int col){super(line,col);charValue=val;};}class CSXStringLitToken extends CSXToken{String stringText;CSXStringLitToken(String text,int line,int col){super(line,col);stringText=text; };}// This class is used to track line andcolumn numbers// Feel free to change to extend itclass Pos {static int linenum = 1;/* maintain this as line number currenttoken was scanned on */static int colnum = 1;/* maintain this as column numbercurrent token began at */static int line = 1;/* maintain this as line number afterscanning current token */142CS 536 Spring 2007©static int col = 1;/* maintain this as column numberafter scanning current token */static void setpos() {//set starting pos for current tokenlinenum = line;colnum = col;}}%%Digit=[0-9]// Tell JLex to have yylex() return aSymbol, as JavaCUP will require%type Symbol// Tell JLex what to return when eof offile is hit%eofval{return new Symbol(sym.EOF,new CSXToken(0,0));%eofval}143CS 536 Spring 2007©%%"+" {Pos.setpos(); Pos.col +=1; return new Symbol(sym.PLUS,new CSXToken(Pos.linenum,Pos.colnum));}"!=" {Pos.setpos(); Pos.col +=2;return new Symbol(sym.NOTEQ,new CSXToken(Pos.linenum,Pos.colnum));}";" {Pos.setpos(); Pos.col +=1;return new Symbol(sym.SEMI,new CSXToken(Pos.linenum,Pos.colnum));}{Digit}+ {// This def doesn’t check// for overflowPos.setpos();Pos.col += yytext().length();return new Symbol(sym.INTLIT,new CSXIntLitToken(new Integer(yytext()).intValue(),Pos.linenum,Pos.colnum));}\n {Pos.line +=1; Pos.col = 1;}" " {Pos.col +=1;}144CS 536 Spring 2007©The Java program that uses thisscanner (P2) is:class P2 {public static void main(String args[])throws java.io.IOException {if (args.length != 1) {System.out.println("Error: Input file must be named oncommand line." );System.exit(-1);}java.io.FileInputStream yyin = null;try {yyin =new java.io.FileInputStream(args[0]);} catch (FileNotFoundExceptionnotFound){System.out.println("Error: unable to open input file.”);System.exit(-1);}// lex is a JLex-generated scanner that// will read from yyinYylex lex = new Yylex(yyin);System.out.println("Begin test of CSX scanner.");145CS 536 Spring 2007©/**********************************You should enter code here thatthoroughly test your scanner.Be sure to test extreme cases,like very long symbols or lines,illegal tokens, unrepresentableintegers, illegals strings, etc.The following is only a starting point.***********************************/Symbol token = lex.yylex();while ( token.sym != sym.EOF ) {System.out.print(((CSXToken) token.value).linenum+ ":"+ ((CSXToken) token.value).colnum+ " ");switch (token.sym) {case sym.INTLIT:System.out.println("\tinteger literal(" +((CSXIntLitToken)token.value).intValue + ")");break; case sym.PLUS:System.out.println("\t+");break;146CS 536 Spring 2007© case sym.NOTEQ:System.out.println("\t!=");break; default:throw new RuntimeException();}token = lex.yylex(); // get next token}System.out.println("End test of CSX scanner.");}}}147CS 536 Spring 2007©Other Scanner IssuesWe will consider other practicalissues in building real scannersfor real programming languages.Our finite automaton modelsometimes needs to beaugmented. Moreover, errorhandling must be incorporatedinto any practical scanner.148CS 536 Spring 2007©Identifiers vs. Reserved WordsMost programming languagescontain reserved words like if,while, switch, etc. These tokenslook like ordinary identifiers, butaren’t.It is up to the scanner to decide ifwhat looks like an identifier isreally a reserved word. Thisdistinction is vital as reservedwords have different token codesthan identifiers and are parseddifferently.How can a scanner decide whichtokens are identifiers and whichare reserved words?• We can scan identifiers and reservedwords using the same pattern, andthen look up the token in a special“reserved word” table.149CS 536 Spring 2007©• It is known that any regular expressionmay be complemented to obtain all stringsnot in the original regular expression.ThusA, the complement of A, is regular ifA is. Using complementation we can writea regular expression for nonreservedidentifiers:Since scanner generators don’t usuallysupport complementation of regularexpressions, this approach is more oftheoretical than practical interest.• We can give distinct regular expressiondefinitions for each reserved word, and foridentifiers. Since the definitions overlap(if will match a reserved word and thegeneral identifier pattern), we give priorityto reserved words. Thus a token is scannedas an identifier if it matches the identifierpattern and does not match any reservedword pattern. This approach is commonlyused in scanner generators like Lex andJLex.ident if while …()150CS 536 Spring 2007©Converting Token ValuesFor some tokens, we may need toconvert from string form intonumeric or binary form.For example, for integers, weneed to transform a string a digitsinto the internal (binary) form ofintegers.We know the format of the tokenis valid (the scanner checkedthis), but:• The string may represent an integertoo large to represent in 32 or 64 bitform.• Languages like CSX and ML use anon-standard representation fornegative values (~123 instead of-123)151CS 536 Spring 2007©We can safely convert from stringto integer form by first convertingthe string to double form,checking against max and min int,and then converting to int form ifthe value is representable.Thus d = new Double(str) willcreate an object d containing thevalue of str in double form. Ifstr is too large or too small to berepresented as a double, plus orminus infinity is automaticallysubstituted.d.doubleValue() will give d’svalue as a Java double, which canbe compared againstInteger.MAX_VALUE orInteger.MIN_VALUE.152CS 536 Spring 2007©If d.doubleValue() represents avalid integer,(int) d.doubleValue()will create the appropriate integervalue.If a string representation of