Short List of Text Editors

Table 2-3 presents a short list of some popular text editors. Use this table as a starting point in selecting an editor. This is by all means not a comprehensive list of available editors, so shop around and use the Internet to find more. Table 2-3. Short List of Open Source Editors http www.gnu.org software emacs Cygwin, Mac OS Has all the features described previously in the chapter and more. Downside available only in Windows, sorry Linuxfolks. A very good text editor, especially if you're...

Additional ospath Module Functionality

The module os.path provides functions that help manage file names and file paths. Table 10-3 lists some useful os.path functions. In the table, assume the current working directory is home user and the file in the directory is file.ext. Table 10-3. Useful os.path Functions Returns the file name, excluding path Returns the directory name of a path Returns True if the path or file specified by s exists Returns the last access time of a file os.path.getctime(s) Returns creation time of a file...

Analyzing GPS Data

Let's take a closer look at the GPS data. Each row seems to start with a text header stamp, beginning with the characters GP. There are several header stamps, for example, GPGSA and GPRMC. Following the header are additional values, most of which are numeric. Not being GPS savvy, I looked up the GPS format on the Internet. It turns out the format is known as NMEA 0183. NMEA stands for the National Marine Electronics Association see http www.nmea.org for more information. The NMEA 0183 data...

Beginning Python Visualization

Crafting Visual Transformation Scripts Beginning Python Visualization Crafting Visual Transformation Scripts Copyright 2009 by Shai Vaingast All rights reserved. No part of this work may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopying, recording, or by any information storage or retrieval system, without the prior written permission of the copyright owner and the publisher. ISBN-13 (electronic) 978-1-4302-1844-9 Printed and bound in the...

Case Study GPS Data

You just got a USB GPS receiver for your birthday You'd like to analyze GPS data and find out how often you exceed the speed limit and how much time you spend in traffic. You'd like to track data over a year, or even longer. Some hardware background most USB GPS receivers behave as serial ports (this is also true for Bluetooth GPS devices). What this means is that once a GPS is connected, and assuming it's installed properly, reading GPS data is as simple as opening the COM port associated with...

Converting File Formats

One of the common operations we perform on images is converting the image file format. It's important for a couple of reasons because we want to store images in a more efficient format using compression, or because the application that accepts the image requires a different format. The Image method save() enables saving an image to file in a specified image format. There are two methods to specify a format with a file name extension and by explicitly specifying the format argument. Assuming...

Date and Time

While text files are important, text isn't really the object of this book it has to do more with numbers. The most universal form of data you're bound to see when processing data files is date and time. Text files that record information based on date and time, especially recording the events that transpired, are commonly referred to as log files. I recently performed a search for file names containing the word log in my C Windows directory. I came up with dozens of files that are indeed log...

Date and Time in a File Name

Remember that time when we ran that test That was when you joined the group, about a year and a half ago. One of the best ways to capture date and time information is to use it to name a file. Following this guideline allows easy file searches. Instead of going through the files one at a time, opening them, and looking at the contents, you can browse the directory contents and find data based on date. The following are benefits of using date and time in a file...

Example Some Other Patches

The code in Listing 6-15 generates a list of patch objects and attaches them to a figure. The figure is originally empty. Listing 6-15. Some Patches from pylab import * Import Ellipse and Wedge to current namespace from matplotlib.patches import Ellipse, Wedge a list of some patches my_patches Arrow(0, 4, 0, -4, facecolor 'g'), Circle( -2, 2 , 1.5, linewidth 4, fc 'orange'), Ellipse( 2, 3 , 4, 1, 45.0, edgecolor 'r'), Polygon( 4, 2 , 3, 3 , 1, -1 , 3, -1 , ls 'dashed', fill False), Wedge( -1, 0...

Example Splitting and Combining Files

Back in the day, computer users used to transfer files on 360K diskettes. Since this was rather painful, they opted to use a file compression utility that did both splitting and compressing. Alas, one of the disks always seemed to be misplaced, rendering the data totally useless. Which brings up a common task when dealing with large files you may need to split them into smaller, more manageable files. By that, I don't mean that each chunk contains legible information, merely that now you can...

Exceptions try else and finally

Exceptions are Python's mechanism of dealing with runtime issues. You've already seen exceptions reported and also how to catch them, that is, prevent them from halting program execution, in Chapter 1. You can catch, or intercept, exceptions before they stop program execution with the following syntax ExceptBlockl except ExceptionType2 If an exception happens someplace inside the TryBlock, ExceptBlockl is executed. In case ExceptionTypel is specified, only exceptions that are of type...

Final Notes and References

The range of applications for which NumPy is of value is large. And as evidence, you'll find that a considerable number of packages rely on NumPy, and for a good reason NumPy provides a solid base for mathematical arrays. An interesting module that comes with the Python Standard Library is the decimal module. This module provides support for decimal floating-point values and allows, for example, arbitrary percision. The decimal module is a bit less intuitive than regular numbers in Python, but...

Functions where select and find

The first set of functions we'll cover is where(), select(), and find(). The function find(cond) finds the indices to an array for which a condition is met > > > from pylab import * > > > squares arange(l0)**2 > > > squares array( 0, 1, 4, 9, 16, 25, 36, 49, 64, 81 ) > > > I find(squares < 50) array( 0, 1, 2, 3, 4, 5, 6, 7 ) > > > squares I array( 0, 1, 4, 9, 16, 25, 36, 49 ) We created a vector holding the squares of the numbers 0-9 and found all the indices to...

Int and Long

The data type int is equivalent to C's long data type, and its precision is system dependent. I run a 32-bit machine, so on my system, int defaults to a 4-byte integer. This means the maximum int I can represent on my system is 231-1, and the minimum is -231. If you're uncertain of the bit count on your system or if your code might be running on different platforms (e.g., both 32-bit and 64-bit platforms), you can use the following to determine the maximum int value > > > import sys >...

Interactive Graphs vs Image Files

There are several ways you can use matplotlib Create dynamic content to be served on a web server for example, generating stock price images on the fly or displaying traffic information on top of a map. Embed it in a graphical user interface, allowing users to interact with an application to visualize data. Automatically process data and generate output in a variety of file formats, including JPG, PNG (Portable Network Graphics, see http www.libpng.org pub png ), PDF, and PostScript (PS). This...

Object Oriented Programming

Per the description I've given of the Python language in the beginning of the chapter, you can deduce that Python is an object-oriented programming language. You've already seen this. For example, the data structure list, whose methods are in essence member functions, is an object. The purpose of this section is to quickly (very quickly ) go over the syntax of object-oriented programming and to show how to implement a basic object. The reason I won't be covering OOP in detail is that this book...

Opening a File

To open a file, use the open(filename , mode , buffering ) built-in function. The function open() returns a file object that is used for subsequent file operations. The first argument, filename, is required. The second argument, mode, is optional and can take the values listed in Table 5-1. Table 5-1. File Open Modes Mode Description r Opens a file for reading. This is the default value if mode isn't specified. w Opens a file for writing, overwriting an existing file. a Opens a file for writing...

Parsing and Formatting Date and Time

The functions strftime() and strptime() are the two functions you're most likely to use when dealing with log files. The function strptime(), which was introduced in Chapter 4, accepts a template parse string and the string to parse, and returns a struct time representation of the time. The function strftime() does the opposite transforms a tuple into a string based on a supplied pattern. Both functions use a similar notation to indicate the values in the struct_ time tuple, as listed in Table...

Saving Graphs to Files

The function savefig() enables writing images of varying formats to a file. Out of the box, matplotlib supports several file formats including PDF, PNG, and PS. The simplest way to generate a file containing a graph is to issue savefig(filename), where filename has the extension associated with your selected image format > > > figure() > > > plot(arange(3)) < matplotlib.lines.Line2D object at 0x0l86A2D0> > > > savefig('line.png') > > > import os > > > file for...

Testing Your Implementation exec and assert

This is a bit of an off-track discussion and somewhat advanced, but I thought it appropriate in the context of the preceding discussion. As you implement the base-conversion helper functions, you'll find that it's quite possible that you've made a mistake. Those are implementations of nested function calls and are prone to human error. Python provides several testing modules doctest and unittest (see the Python Standard Library, However, I chose a different approach, one that does not make use...

The for Statement

The for statement is one of the most versatile statements in Python. The statement follows the following syntax In case of a one-line block, the ForBlock can appear on the same line as the for statement. Indentation rules for blocks are the same as those described in the if statement (and for any block for that matter they must be indented to the same level). The for statement assigns element to be a value from sequence and executes the ForBlock. This happens for all the values in sequence >...

Unicode Strings

The original ASCII table is based on the English language and does not account for a lot of other languages and symbols. Several designs were introduced to try to resolve this, and lately it appears that Unicode (http www.unicode.org) is the industry standard. The Unicode standard addresses such topics as character encoding, character properties, visual representation, and more. From a very simplistic approach, Unicode tries to support characters and symbols from other languages by assigning...

Version Control Systems

Version control systems (VCSs) enable management of several revisions of a document (or documents) with full history, tagging, and date capabilities. Most packages also support several developers working together simultaneously on the same file. AVCS allows going back to a previous working version, or checking the difference between the current version and an older one, or even viewing a version of the document based on date. It might hold such information as who edited the file or the tag...

Reading CSV Files

Again, we turn to Python's built-in modules, this time the csv module. Although the CSV file format is quite popular, there's no clear definition, and each spreadsheet and database employs its own dialect. The files we'll be processing adhere to the most basic CSV file dialect, so we'll use the default behavior of Python's csv module. Since we'll be reading several CSV files, it stands to reason to define a function to perform this task. Listing 1-4 shows...

Extracting GPS Data

In the case of a GPGSV header, the number of satellites is the fourth entry. In case of a GPRMC header, we have a bit more interesting information. The second field is the timestamp, the fourth field is the latitude, the sixth field is the longitude, and the eighth field is the velocity. Again, turn to the NMEA 0183 format for more details. Table 1-1 summarizes the fields and their values in a GPRMC line. Table 1-1. GPRMC Information Excerpt Table 1-1. GPRMC Information Excerpt Some caveats...

Example Stock Price Charts

Python Stock Chart Lib

Following a convention that stores a short description of the data in the beginning lines of the CSV files can be very useful for annotating a graph or a report associated with the data in the file. To follow along with the example, ensure your directory structure is similar to that presented in Chapter 2 in the section Example Directory Structure for the Book. Your base directory should be Ch4 within Ch4 there should be three subdirectories named src, data, and images. If you wish to use a...

Example Logging Information with a Date and Timestamp

The purpose of this example is to create a log file in accordance with the tip presented previously in the Date and Time section and the ISO time format recommendation see Chapter 4 . We'll use the function localtime , which returns a structtime tuple containing the current time. We then format the current time using the strftime function gt gt gt from time import localtime, strftime gt strftime Y- m- dT H- M- S, localtimeQ '2008-10-23T12-16-41' From now on, I'll assume you either imported the...

Array Methods and Properties

Loubere Method

Arrays are objects and as such have functions called methods and variables called properties. Using IPython see Chapter 2 , you can list an object's methods and properties by using character completion, accessible via the Tab key. Alternatively, you can issue the following gt gt gt m for m in dir numpy.ndarray if not m.startswith '_' 'T', 'all', 'any', 'argmax', 'argmin', 'argsort', 'astype', 'base', 'byteswap', 'choose', 'clip', 'compress', 'conj', 'conjugate', 'copy', 'ctypes', 'cumprod',...

Piecewise Linear Interpolation

Linear Interpolation

Let's turn back to our half-a-circle example. This time, we'll limit ourselves to a quarter of a circle, that is, positive values of x and y. We start by calculating the y values for x equal to 0, 0.2, . . . , 1. We'll store the results in vectors xp and yp array 0. , 0.2, 0.4, 0.6, 0.8, 1. gt gt gt yp sqrt l-xp 2 We'd like to calculate the values of y for x values equal to 0.1, 0.3, . . . , 0.9 given xp and yp. We'll use the function interp x, xp, yp for this. The function returns the value of...

Example Moving Average

Numpy Smooth List

On many occasions, filtering is used to smooth a signal. A simple algorithm is that of a moving average. For every two consecutive points, we calculate the average and use that value instead. The points are overlapping, so a result of using the algorithm on the vector l, 2, 0, 2 would be 1.5, 1, l . But why stop at two samples Moving average can be performed on several points, returning the average of those points. In Python, you could write gt gt gt from pylab import gt N 512 gt gt gt x l-exp...

Example Reading and Writing an Array of Structs

In this example, we'll create a structure containing two data types long and float , write it to file, and then read it using two different methods a structure at a time and the entire file at once. You can follow along by entering the commands interactively at the Python shell. First, we have to import the struct module To illustrate the concept of an array of structs, we'll create a list of rows. Each row is a list of three values a long and two floats, which represent a structure. We'll...

Example Automatically Reading Yahoo Financial Data

Matplotlib Yahoo

The following discussion is a bit off-topic, but as it is a direct continuation of the previous example, this is probably a logical spot for it. There's an alternative method to manually saving the charts.xls file from NASDAQ. One such option is using the matplotlib.finance module. The two core functions that fetch the data and parse it are fetch_historical_yahoo and parse_yahoo_historical although you could easily parse the data yourself . Another function of interest is the candlestick...

Example Base Conversion Binary Octal Decimal and Hexadecimal

At times, it's useful to convert a number from one base representation to another. In case you're dealing with octal, decimal, and hexadecimal bases, this is easily achieved using the functions int , hex , and oct . Since we're dealing with representations of numbers, it stands to reason we use strings. I've therefore chosen to define several functions, all of which accept a string as an argument. As shown in Listing 5-2, these functions are hex2dec , hex2oct , dec2hex , dec2oct , oct2dec , and...

Fractal Function Source Code

In Chapter 9 I made use of a variation of the fractal script in Chapter 7 to create a collage by wrapping it within a function. Listing A-5 shows the function used in creating the fractal collage in Chapter 9. Listing A-5. Fractal Collage Function from PIL import Image from cmath import def fractal delta 0.000001, res 800, iters 30 Creates a z 4 1 0 fractal using the Newton-Raphson method create an image to draw on, paint it black img Image.new RGB, res, res , 0, 0, 0 these are the solutions to...

Example Creating a Fixed Size File Stand Alone Script

We turn to modify the code in this chapter from the section Example Creating a Fixed-Size File to a stand-alone script callable from a CLI shell or command window . The script, shown in Listing 10-8 emptyfile.py , accepts the number of bytes and a file name and creates a file of specified name and size. Listing 10-8. Creating a Fixed-Size File Stand-Alone Script , empty_file.py from sys import argv, exit usage Usage python empty_file.py nbytes filename we expect three arguments script name,...

Example Thumbnail Index Image

Tkinter Examples Python

In a previous example we've created a catalog of images. While that catalog is quite useful, it doesn't show the contents of those images. A more useful catalog perhaps would be a collage of the images annotated with text showing each image's file name see Listing 9-7 . from PIL import Image, ImageDraw def thumbnail_index dirpath Create a thumbnail index from images in dirpath num_images 5 thumb_size 128, 96 cat_size num_images thumb_size o , num_images thumb_size l go through all the pictures...

Additional Graphs

Matplotlib also supports a great number of graphs used to depict more complex data. Here's a short list of some of the graphs available Functions contour and contourf are used for contour plots. Contour plots draw a line connecting equal x, y value pairs. They're used in weather maps, detailing lines of equal pressure or temperature in topographical maps, detailing the terrain in physics graphs, to describe fields and more. Function specgram displays the frequency contents of data over time....

GPS Location Plot

Matplotlib Gps

Now the moment we've been waiting for, plotting GPS data. To be able to follow along and plot data, be sure to define the functions read_csv_file and process_gps_data as previously detailed and set the file name variable to point to your GPS data file. I've suppressed matplotlib responses so that the code is cleaner to follow. gt gt gt filename 'GPS-2008-05-30-09-00-50.csv' gt gt gt y read_csv_file ' data ' filename gt gt gt lat, long, v, t, sats process_gps_data y gt px long-min long NMI 60.0...

Dict Reader and Dict Writer Objects

The csv module provides us with additional useful objects the DictReader and DictWriter objects, which are similar to the csv.reader and csv.writer objects. If you follow the convention that places a header at the beginning of a CSV file, that is, that each column in the CSV file starts with a field name see Chapter 4 for a discussion of this , accessing values can be done by accessing the dictionary with the field name as key. Let's turn to an example. To follow along, create the file data...

NDimensional Arrays

NumPy arrays are N-dimensional arrays and can be created in the same manner as 1-D and 2-D arrays A useful operator for N-dimensional arrays is , which means, all the remaining dimensions. One of the common questions is how useful N-dimensional arrays are. Some people feel that they do pretty well with one or two dimensions and have little use for N-dimensions. My experience with N-dimensional arrays is that they provide an excellent data storage when dealing with a combination of several...

Setting Figure and Axis Parameters

In the preceding examples we stored the return value from the call to the function plot , which is a matplotlib object of a line, specifically the line we drew actually, a list containing one line . But how do we modify the behavior of the figure or the axis The function gcf returns a handle to the current figure. The function gca returns a handle to the current axis. Armed with these, we can now modify the axis and figure parameters. To set the y label, instead of calling ylabel 'Y value' , we...