Filtering ToolsInput Redirection OperatorOther Redirection OperatorsAn Example Using >>Another Example of >>And Another ExampleThe Pipe OperatorPipes Connecting FiltersFiltersFiltering Standard InputSome Useful FiltersMore Filters (2)Selected FiltersSelected Filters: sortSelected Filters: sort continuedMore on SortingSorting By FieldsSelected Filters: uniqSelected Filters: foldSelected Filters: wcUses of wcThings To TryCopyright 2006 Stewart WeissCopyright 2009 Stewart WeissFiltering ToolsFiltering ToolsFiltering Tools and a Bit More About I/O Redirection2 CSci 132 Practical UNIX with PerlInput redirection operator <The input redirection operator '<', replaces the standard input of a command by the file that follows it. It is not very useful for many commands because many commands let you put the names of files on the command line as their arguments, so there is no need to use this operator for most commands. cat < myfilefor example, is the same as cat myfile3 CSci 132 Practical UNIX with PerlOther redirection operatorsThere are two other redirection operators, '<< ' and '>>'. We will start with '>>':The output append operator, >>, which you saw in an earlier lesson, forces standard output to be appended to a file instead of replacing the file's contents. This is very useful; it gives you the means to add new lines to files.4 CSci 132 Practical UNIX with PerlAn example using >>To illustrate, suppose that story is a file with a single line of text as shown below. We'll use >> to lengthen the story.$ cat storyOnce upon a time$ echo "there was no UNIX." >> story$ cat storyOnce upon a timethere was no UNIX.$ You can see that the output of echo was added as a new line in story, instead of replacing its contents.5 CSci 132 Practical UNIX with PerlAnother example of >>There are many system programs that maintain log files, i.e., files that record the program's activities. Many of these programs are shell scripts. When they need to append information to their log files, they use the append operator. I could create a log file to record whenever I started a bash shell by adding this to my .bashrc file: echo "bash started on `tty` at `date`" >> ~/.mylog This would add a line to the file .mylog in my home directory each time I ran a bash shell. The expression `...` is explained next.6 CSci 132 Practical UNIX with PerlAnd another exampleI keep track of my own logins by adding the following line to my .bash_profile:echo "Login into `hostname` on `date`" >> .loghistThe .bash_profile file is executed only on logins, not when a non-login shell runs, so this does not add a line each time I start a shell.This line appends the name of the host and the time and date of login to my .loghist file. The backquoted expressions `hostname` and `date` cause the hostname and date commands to run and their output to be substituted for their backquoted names.7 CSci 132 Practical UNIX with PerlThe pipe operator |We have been using the pipe operator, '|', already so this is a review. The pipe operator connects the output of the command on its left to the input of the command on its right. It is a very powerful tool. For example ypcat passwd | awk -F: '{print $5}'displays the real names of everyone with an account on the computer. man -k | grep ' file ' | grep ' write ' looks for commands with the words file and write in the description.8 CSci 132 Practical UNIX with PerlPipes connecting filtersCommands can be composed upon each other using the pipe operator. The power of pipes was demonstrated in the first lesson with the following example:who | awk '{print $1}' | sort | uniqIn this example, the output of who is the input of awk, whose output is the input of sort, whose output is the input of uniq. The output is a list of usernames of people currently logged in, one per line. (There is a system command, users, that creates this list also, so we really didn't need to do all that work!)9 CSci 132 Practical UNIX with PerlFiltersThe awk, sort, and uniq commands are examples of a class of UNIX programs called filters.A filter is a UNIX command whose input and output are ordinary text, and that expects its input from standard input and puts its output on standard output.Filters transform their input in some way, such as by sorting it, removing words or lines based on a pattern or on their position in the line or file (e.g., remove every 3rd word in a line, or remove every 4th line, or remove any line that has a curse word.)10 CSci 132 Practical UNIX with PerlFiltering standard inputFilters may have file name arguments on the command line, but when they have no arguments, they read from standard input (the keyboard) instead:grep 'a clue' thehousesearches for 'a clue' in thehouse, whereas incat thehouse | grep 'a clue'grep searches through its standard input stream for lines with 'a clue'.11 CSci 132 Practical UNIX with PerlSome useful filters (1)grep family:grep, egrep, fgrepglobal regular expression parserssortsorts based on several criteriauniqremoves adjacent identical linesawkfull-fledged programming language for field-oriented pattern matchingcutremoves pieces of each line based on positionsSome of the most useful filters are:12 CSci 132 Practical UNIX with PerlMore filters These are filters too:head, taildisplay just top or bottom lines of filescatnull filter -- shows everything in ordertacshows lines in reverse orderfold -w<N>display output in width of N columnssedstream editor -- very powerful filter wcnot exactly a filter, display count of chars, words, and lines13 CSci 132 Practical UNIX with PerlSelected filtersIn the next chapter you will learn about patterns (called regular expressions). Then you will learn how to use the family of filters that use these patterns.First we will look a few of the filters that do not require patterns as arguments.14 CSci 132 Practical UNIX with PerlSelected filters: sortThe sort program can be used for sorting one or more text files in sequence. It can also merge files.In its simplest form,sort filenamewill sort filename using the first field of the line (first chars up to the first white space) in ASCII collating order, displaying the sorted file on the screen (standard output).sort will ignore case by default in some versions of UNIX, whereas in others, uppercase and lowercase letters will be treated as different.15 CSci 132 Practical UNIX with PerlSelected filters: sortThe sort program will not sort numbers properly unless you tell it to