Scripting languages • originally tools for quick hacks, rapid prototyping, gluing together other programs, ... • evolved into mainstream programming tools • characteristics – text strings as basic (or only) data type – regular expressions (maybe) built in – associative arrays as a basic aggregate type – minimal use of types, declarations, etc. – usually interpreted instead of compiled • examples – shell – Awk – Perl, PHP, Ruby, Python – Tcl, Lua, ... – Javascript, Actionscript – Visual Basic, (VB|W|C)Script, PowerShell – …Shells and shell programming • shell: a program that helps run other programs – intermediary between user and operating system – basic scripting language – programming with programs as building blocks • an ordinary program, not part of the system – it can be replaced by one you like better – therefore there are lots of shells, reflecting history and preferences • popular shells: – sh Bourne shell (Steve Bourne, Bell Labs -> ... -> El Dorado Ventures) emphasizes running programs and programmability syntax derived from Algol 68 – csh C shell (Bill Joy, UC Berkeley -> Sun -> Kleiner Perkins) interaction: history, job control, command & filename completion, aliases more C-like syntax, but not as good for programming (at least historically) – ksh Korn shell (Dave Korn, Bell Labs -> AT&T Labs) combines programmability and interaction syntactically, superset of Bourne sh provides all csh interactive features + lots more – bash GNU shell mostly ksh + much of csh – tcsh evolution of cshFeatures common to Unix shells • command execution + built-in commands, e.g., cd • filename expansion * ? [...] • quoting rm '*' Careful !!! echo "It's now `date`" • variables, environment PATH=/bin:/usr/bin in ksh & bash setenv PATH /bin:/usr/bin in (t)csh • input/output redirection, pipes prog <in >out, prog >>out who | wc slow.1 | slow.2 & asynchronous operation • executing commands from a file arguments can be passed to a shell file ($0, $1, etc.) if made executable, indistinguishable from compiled programs provided by the shell, not each programShell programming • the shell is a programming language – the earliest scripting language • string-valued variables • limited regexprs mostly for filename expansion • control flow – if-else if cmd; then cmds; elif cmds; else cmds; fi (sh…) if (expr) cmds; else if (expr) cmds; else cmds; endif (csh) – while, for for var in list; do commands; done (sh, ksh, bash) foreach var (list) commands; end (csh, tcsh) – switch, case, break, continue, ... • operators are programs – programs return status: 0 == success, non-0 == various failures • shell programming out of favor – graphical interfaces – scripting languages e.g., system administration setting paths, filenames, parameters, etc now often in Perl, Python, PHP, ...Shell programming • shell programs are good for personal tools – tailoring environment – abbreviating common operations (aliases do the same) • gluing together existing programs into new ones • prototyping • sometimes for production use – e.g., configuration scripts • But: – shell is poor at arithmetic, editing – macro processing is a mess – quoting is a mess – sometimes too slow – can't get at some things that are really necessary • this leads to scripting languagesOver-simplified history of programming languages • 1940's machine language • 1950's assembly language • 1960's high-level languages: Algol, Fortran, Cobol, Basic • 1970's systems programming: C • 1980's object-oriented: C++ • 1990's strongly-hyped: Java • 2000's copycat languages: C# • 2010's ???AWK • a language for pattern scanning and processing – Al Aho, Brian Kernighan, Peter Weinberger, at Bell Labs, ~1977 • intended for simple data processing: • selection, validation: "Print all lines longer than 80 characters" length > 80 • transforming, rearranging: ”Print first two fields in the opposite order" { print $2, $1 } • report generation: "Add up the numbers in the first field, then print the sum and average" { sum += $1 } END { print sum, sum/NR }Structure of an AWK program: • a sequence of pattern-action statements pattern { action } pattern { action } … • "pattern" is a regular expression, numeric expression, string expression or combination of these • "action" is executable code, similar to C • usage: awk 'program' [ file1 file2 ... ] awk -f progfile [ file1 file2 ... ] • operation: for each file for each input line for each pattern if pattern matches input line do the actionAWK features: • input is read automatically across multiple files – lines are split into fields ($1, ..., $NF; $0 for whole line) • variables contain string or numeric values (or both) – no declarations: type determined by context and use – initialized to 0 and empty string – built-in variables for frequently-used values • operators work on strings or numbers – coerce type / value according to context • associative arrays (arbitrary subscripts) • regular expressions (like egrep) • control flow statements similar to C: if-else, while, for, do • built-in and user-defined functions – arithmetic, string, regular expression, text edit, ... • printf for formatted output • getline for input from files or processesBasic AWK programs, part 1 { print NR, $0 } precede each line by line number { $1 = NR; print } replace first field by line number { print $2, $1 } print field 2, then field 1 { temp = $1; $1 = $2; $2 = temp; print } flip $1, $2 { $2 = ""; print } zap field 2 { print $NF } print last field NF > 0 print non-empty lines NF > 4 print if more than 4 fields $NF > 4 print if last field greater than 4 /regexpr/ print matching lines (egrep) $1 ~ /regexpr/ print lines where first field matchesBasic AWK programs, part 2 NF > 0 {print $1, $2} print two fields of non-empty lines END { print NR } line count { nc += length($0) + 1; nw += NF } wc command END {