Programming with Python
- Use the values of command-line arguments in a program.
- Handle flags and files separately in a command-line program.
- Read data from standard input in a program so that it can be used in a pipeline.
The Jupyter Notebook and other interactive tools are great for prototyping code and exploring data, but sooner or later we will want to use our program in a pipeline or run it in a shell script to process thousands of data files. In order to do that, we need to make our programs work like other Unix command-line tools. For example, we may want a program that reads a dataset and prints the average inflammation per patient.
This program does exactly what we want - it prints the average inflammation per patient for a given file.
$ python code/readings_04.py --mean data/inflammation-01.csv 5.45 5.425 6.1 ... 6.4 7.05 5.9
We might also want to look at the minimum of the first four lines
$ head -4 data/inflammation-01.csv | python code/readings_04.py --min
or the maximum inflammations in several files one after another:
$ python code/readings_04.py --max data/inflammation-*.csv
Our scripts should do the following:
- If no filename is given on the command line, read data from standard input.
- If one or more filenames are given, read data from them and report statistics for each file separately.
- Use the
--maxflag to determine what statistic to print.
To make this work, we need to know how to handle command-line arguments in a program, and how to get at standard input. We’ll tackle these questions in turn below.
Using the text editor of your choice, save the following in a text file called
import sys print('version is', sys.version)
The first line imports a library called
sys, which is short for “system”. It defines values such as
sys.version, which describes which version of Python we are running. We can run this script from the command line like this:
$ python sys_version.py
version is 3.4.3+ (default, Jul 28 2015, 13:17:50) [GCC 4.9.3]
Create another file called
argv_list.py and save the following text to it.
import sys print('sys.argv is', sys.argv)
The strange name
argv stands for “argument values”. Whenever Python runs a program, it takes all of the values given on the command line and puts them in the list
sys.argv so that the program can determine what they were. If we run this program with no arguments:
$ python argv_list.py
sys.argv is ['argv_list.py']
the only thing in the list is the full path to our script, which is always
sys.argv. If we run it with a few arguments, however:
$ python argv_list.py first second third
sys.argv is ['argv_list.py', 'first', 'second', 'third']
then Python adds each of those arguments to that magic list.
With this in hand, let’s build a version of
readings.py that always prints the per-patient mean of a single data file. The first step is to write a function that outlines our implementation, and a placeholder for the function that does the actual work. By convention this function is usually called
main, though we can call it whatever we want:
$ cat readings_01.py
import sys import numpy def main(): script = sys.argv filename = sys.argv data = numpy.loadtxt(filename, delimiter=',') for m in numpy.mean(data, axis=1): print(m)
This function gets the name of the script from
sys.argv, because that’s where it’s always put, and the name of the file to process from
sys.argv. Here’s a simple test:
$ python readings_01.py inflammation-01.csv
There is no output because we have defined a function, but haven’t actually called it. Let’s add a call to
$ cat readings_02.py
import sys import numpy def main(): script = sys.argv filename = sys.argv data = numpy.loadtxt(filename, delimiter=',') for m in numpy.mean(data, axis=1): print(m) if __name__ == '__main__': main()
and run that:
$ python readings_02.py inflammation-01.csv
5.45 5.425 6.1 5.9 5.55 6.225 5.975 6.65 6.625 6.525 6.775 5.8 6.225 5.75 5.225 6.3 6.55 5.7 5.85 6.55 5.775 5.825 6.175 6.1 5.8 6.425 6.05 6.025 6.175 6.55 6.175 6.35 6.725 6.125 7.075 5.725 5.925 6.15 6.075 5.75 5.975 5.725 6.3 5.9 6.75 5.925 7.225 6.15 5.95 6.275 5.7 6.1 6.825 5.975 6.725 5.7 6.25 6.4 7.05 5.9
Handling Multiple Files
The next step is to teach our program how to handle multiple files. Since 60 lines of output per file is a lot to page through, we’ll start by using three smaller files, each of which has three days of data for two patients:
$ ls small-*.csv
small-01.csv small-02.csv small-03.csv
$ cat small-01.csv
$ python readings_02.py small-01.csv
Using small data files as input also allows us to check our results more easily: here, for example, we can see that our program is calculating the mean correctly for each line, whereas we were really taking it on faith before. This is yet another rule of programming: test the simple things first.
We want our program to process each file separately, so we need a loop that executes once for each filename. If we specify the files on the command line, the filenames will be in
sys.argv, but we need to be careful:
sys.argv will always be the name of our script, rather than the name of a file. We also need to handle an unknown number of filenames, since our program could be run for any number of files.
The solution to both problems is to loop over the contents of
sys.argv[1:]. The ‘1’ tells Python to start the slice at location 1, so the program’s name isn’t included; since we’ve left off the upper bound, the slice runs to the end of the list, and includes all the filenames. Here’s our changed program
$ cat readings_03.py
import sys import numpy def main(): script = sys.argv for filename in sys.argv[1:]: data = numpy.loadtxt(filename, delimiter=',') for m in numpy.mean(data, axis=1): print(m) if __name__ == '__main__': main()
and here it is in action:
$ python readings_03.py small-01.csv small-02.csv
0.333333333333 1.0 13.6666666667 11.0
Handling Command-Line Flags
The next step is to teach our program to pay attention to the
--max flags. These always appear before the names of the files, so we could just do this:
$ cat readings_04.py
import sys import numpy def main(): script = sys.argv action = sys.argv filenames = sys.argv[2:] for f in filenames: data = numpy.loadtxt(f, delimiter=',') if action == '--min': values = numpy.min(data, axis=1) elif action == '--mean': values = numpy.mean(data, axis=1) elif action == '--max': values = numpy.max(data, axis=1) for m in values: print(m) if __name__ == '__main__': main()
$ python readings_04.py --max small-01.csv
but there are several things wrong with it:
mainis too large to read comfortably.
If we do not specify at least two additional arguments on the command-line, one for the flag and one for the filename, but only one, the program will not throw an exception but will run. It assumes that the file list is empty, as
sys.argvwill be considered the
action, even if it is a filename. Silent failures like this are always hard to debug.
The program should check if the submitted
actionis one of the three recognized flags.
This version pulls the processing of each file out of the loop into a function of its own. It also checks that
action is one of the allowed flags before doing any processing, so that the program fails fast:
$ cat readings_05.py
import sys import numpy def main(): script = sys.argv action = sys.argv filenames = sys.argv[2:] assert action in ['--min', '--mean', '--max'], \ 'Action is not one of --min, --mean, or --max: ' + action for f in filenames: process(f, action) def process(filename, action): data = numpy.loadtxt(filename, delimiter=',') if action == '--min': values = numpy.min(data, axis=1) elif action == '--mean': values = numpy.mean(data, axis=1) elif action == '--max': values = numpy.max(data, axis=1) for m in values: print(m) if __name__ == '__main__': main()
This is four lines longer than its predecessor, but broken into more digestible chunks of 8 and 12 lines.
Handling Standard Input
The next thing our program has to do is read data from standard input if no filenames are given so that we can put it in a pipeline, redirect input to it, and so on. Let’s experiment in another script called
$ cat count_stdin.py
import sys count = 0 for line in sys.stdin: count += 1 print(count, 'lines in standard input')
This little program reads lines from a special “file” called
sys.stdin, which is automatically connected to the program’s standard input. We don’t have to open it — Python and the operating system take care of that when the program starts up — but we can do almost anything with it that we could do to a regular file. Let’s try running it as if it were a regular command-line program:
$ python count_stdin.py < small-01.csv
2 lines in standard input
A common mistake is to try to run something that reads from standard input like this:
$ python count_stdin.py small-01.csv
i.e., to forget the
< character that redirect the file to standard input. In this case, there’s nothing in standard input, so the program waits at the start of the loop for someone to type something on the keyboard. Since there’s no way for us to do this, our program is stuck, and we have to halt it using the
Interrupt option from the
Kernel menu in the Notebook.
We now need to rewrite the program so that it loads data from
sys.stdin if no filenames are provided. Luckily,
numpy.loadtxt can handle either a filename or an open file as its first parameter, so we don’t actually need to change
$ cat readings_06.py
import sys import numpy def main(): script = sys.argv action = sys.argv filenames = sys.argv[2:] assert action in ['--min', '--mean', '--max'], \ 'Action is not one of --min, --mean, or --max: ' + action if len(filenames) == 0: process(sys.stdin, action) else: for f in filenames: process(f, action) def process(filename, action): data = numpy.loadtxt(filename, delimiter=',') if action == '--min': values = numpy.min(data, axis=1) elif action == '--mean': values = numpy.mean(data, axis=1) elif action == '--max': values = numpy.max(data, axis=1) for m in values: print(m) if __name__ == '__main__': main()
Let’s try it out:
$ python readings_06.py --mean < small-01.csv
That’s better. In fact, that’s done: the program now does everything we set out to do.
Arithmetic on the command line
Write a command-line program that does addition and subtraction:
$ python arith.py add 1 2
$ python arith.py subtract 3 4
Finding particular files
glob module introduced earlier, write a simple version of
ls that shows files in the current directory with a particular suffix. A call to this script should look like this:
$ python my_ls.py py
left.py right.py zero.py
readings.py so that it uses
-x instead of
--max respectively. Is the code easier to read? Is the program easier to understand?
Adding a help message
readings.py so that if no parameters are given (i.e., no action is specified and no filenames are given), it prints a message explaining how it should be used.
Adding a default action
readings.py so that if no action is given it displays the means of the data.
Write a program called
check.py that takes the names of one or more inflammation data files as arguments and checks that all the files have the same number of rows and columns. What is the best way to test your program?
Write a program called
line_count.py that works like the Unix
- If no filenames are given, it reports the number of lines in standard input.
- If one or more filenames are given, it reports the number of lines in each, followed by the total number of lines.
Generate an Error message
Write a program called
check_arguments.py that prints usage then exits the program if no arguments are provided. (Hint) You can use
sys.exit() to exit the program.
$ python check_arguments.py
usage: python check_argument.py filename.txt
$ python check_arguments.py filename.txt
Thanks for specifying arguments!