Acknowledgments Noahs Acknowledgments

As I sit writing an acknowledgment for this book, I have to first mention Dr. Joseph E. Bogen, because he made the single largest impact on me, at a time that it mattered the most. I met Dr. Bogen while I was working at Caltech, and he opened my eyes to another world giving me advice on life, psychology, neuroscience, math, the scientific study of consciousness, and much more. He was the smartest person I ever met, and was someone I loved. I am going to write a book about this experience...

Apache Config File Hacking

Now that you've been introduced to Python regular expressions, let's work through an Apache config file NameVirtualHost 127.0.0.1 80 < VirtualHost localhost 80> DocumentRoot var www < Directory > Options FollowSymLinks AllowOverride None < Directory> ErrorLog var log apache2 error.log LogLevel warn CustomLog var log apache2 access.log combined ServerSignature On < VirtualHost> < VirtualHost local2 80> DocumentRoot var www2 < Directory > Options FollowSymLinks...

Comparing Data

Comparing data is quite important to a sysadmin. Questions you might often ask yourself are, What files are different between these two directories How many copies of this same file exist on my system In this section, you will find the ways to answer those questions and more. When dealing with massive quantities of important data, it often is necessary to compare directory trees and files to see what changes have been made. This becomes critical if you start writing large data mover scripts....

Install Source Distribution from URL

Now, we'll automatically install a source distribution from a URL easy_install sqlalchemy SQLAlchemy-0.4.3.tar.gz Downloading Running SQLAlchemy-0.4.3 setup.py -q bdist_egg --dist-dir zip_safe flag not set analyzing archive contents sqlalchemy.util module MAY be using inspect.stack sqlalchemy.databases.mysql module MAY be using inspect.stack Adding SQLAlchemy 0.4.3 to easy-install.pth file Installed Processing dependencies for SQLAlchemy 0.4.3 Finished processing dependencies for SQLAlchemy...

Log Parsing

No discussion of text processing from a sysadmin's point of view would be complete without addressing parsing a logfile, so here it is. We have laid the foundation for you to be able to open a logfile, read in each line, and read the data in the way that works best for you. Before we begin coding this example, we have to ask ourselves, What do we want this logfile reader to do Our answer is pretty simple read in an Apache access log and determine the number of bytes each unique client...

Manual Information Gathering

Let's also look at the more complicated path of manually gathering information. By this, we mean information that you gather with your own eyes and key in with your own hands. Examples include a list of servers with corresponding IP addresses and functions, a list of contacts with email addresses, phone numbers, and IM screen names, or the dates that members of your team are planning to be on vacation. There are certainly tools available that can manage most, if not, all of these types of...

Pattern Matching Files and Directories

So far you have seen how to process directories and files, and perform actions such as finding duplicates, deleting directories, moving directories, and so on. The next step in mastering the directory tree is to use pattern matching, either alone or in combination with these previous techniques. As just about everything else in Python, performing a pattern match for a file extension or filename is simple. In this section, we will demonstrate a few common pattern matching problems and apply the...

Python Builtins and Modules str

A string is simply a sequence of characters. If you ever need to deal with textual data, you'll almost certainly need to work with it as a string object or a series of string objects. The string type, str, is a powerful, flexible means for manipulating string data. This section shows you how to create strings and what you can do with them once they've been created. The most common way to create a string is to surround the text with quotation marks In 1 stringl 'This is a string' In 2 string2...

Re

Since Python comes with batteries included, you might expect that it would include a regular expression library. You won't be disappointed. The emphasis in this section will be on using Python to work with regular expressions rather than on the ins and outs of regular expression syntax. So if you aren't familiar with regular expressions, we recommend that you pick up a copy of Mastering Regular Expressions (O'Reilly) by Jeffrey E. F. Friedl (also available on Safari at http safari.oreilly.com...

Registering a Package with the Python Package Index

If you write a cool tool or useful module, naturally, you want to share it with other people. This is one of the most enjoyable parts of open source software development. Thankfully, it is a relatively simple process to upload a package to the Python Package Index. The process is only slightly different from creating an egg. Two things to pay attention to are to remember to include a ReST, reStructuredText, formatted description in the long_description, and to provide a download_url value. We...

Search for Packages on a Web Page

As we saw earlier, easy_install can automatically search the central repository for packages and automatically install them. It can also install packages in just about any way you can think of. Following is an example of how to search a web page and install or upgrade package by name and version easy_install -f http code.google.com p liten liten Searching for liten Reading http code.google.com p liten Best match liten 0.1.3 In this situation, there is a Python2.4 and a Python2.5 egg at http...

Setuptools and Python Eggs

According the official documentation, setuptools is a collection of enhancements to the Python distutils (for Python 2.3.5 on most platforms, although 64-bit platforms require a minimum of Python 2.4) that allow you to more easily build and distribute packages, especially ones that have dependencies on other packages. Until the creation of setuptools, distutils was the primary way of creating installable Python packages. setuptools is a library that enhances distutils. Eggs refers to the final...

Threads in Python

Threads could be described as a necessary evil to some people, although many people dislike threads, to solve many problems that require dealing with multiple things at once. Threads are different than processes because they all run inside of the same process and share state and memory. That is both the thread's greatest advantage and disadvantage. The advantage is that you can create a data structure that all threads can access without creating an IPC, or interprocess communication mechanism....

Using a tarfile Module to Examine the Contents of TAR Files

Now that we have a tool that creates TAR files, it only makes sense to examine the TAR file's contents. It is one thing to blindly create a TAR file, but if you have been a systems administrator for any length of time, you have probably gotten burned by a bad backup, or have been accused of making a bad backup. To put this situation in perspective and highlight the importance of examining TAR archives, we will share a story about a fictional friend of ours, let's call it The Case of the Missing...

Using Zenoss to Manage Windows Servers from Linux

If you have the unfortunate task of managing one or more Windows servers, the task just became a little less unpleasant. Zenoss is an amazing tool that will help us out here. We talk about Zenoss in the Chapter 7, SNMP. In addition to being an industry-leading SNMP tool, Zenoss also provides the tools to talk to a Windows server via WMI from Linux We still get the giggles when thinking about the practical implications of this as well as the technology used to make it possible. From a discussion...

Why Python

If you are a system administrator, it is likely that you have encountered Perl, Bash, ksh, or some other scripting language. You may have even used one or more yourself. Scripting languages are often used to do repetitive, tedious work at a rate and with an accuracy that far surpass what you could accomplish without them. All languages are tools. They are simply a means to get work done. They have value only insofar as they help you get your job done better. We believe that Python is a valuable...

Working with Files

Learning to deal with files is key to processing textual data. Often, text that you have to process is contained in a text file such as a logfile, config file, or application data file. When you need to consolidate the data that you are analyzing, you often need to create a report file of some sort or put it into a text file for further analysis. Fortunately, Python contains an easy-to-use built-in type called file that can help you do all of those things. It may seem counterintuitive, but in...

Google App Engine

Google App Engine was released as a beta service, and it was massively buzzworthy from the day it was announced. It lets you run your application on Google's infrastructure for free. App Engine applications have a strictly Python API for now, but that could change at some point. One of the other interesting things about App Engine is that it also integrates with Google's other services. CELEBRITY PROFILE GOOGLE APP ENGINE TEAM Kevin Gibbs is the technical lead for Google App Engine. Kevin...

Working with Paths Directories and Files

One can't talk about dealing with data without taking into account paths, directories, and files. Every sysadmin needs to be able to, at the very least, write a tool that walks a directory, searches for a condition, and then does something with the result. We are going to cover some interesting ways to do just that. As always, the Standard Library in Python has some killer tools to get the job done. Python doesn't have a reputation for being batteries included for nothing. Example 6-5 shows how...

Brief Introduction to SNMP

The 10,000 foot view of SNMP is that it is a protocol for managing devices on an IP network. Typically, this is done via UDP ports 161 and 162, although it is possible, but rare, to use TCP as well. Just about any modern device in a data center supports SNMP this means it is possible to manage not only switches and routers, but servers, printers, UPSs, storage, and more. The basic use for SNMP is to send UDP packets to hosts and to wait for a response. This is how monitoring of devices occurs...

Python Alias In Linux

The first feature of a Python Unix shell bridge that we will look at is the alias magic function. With alias you can create an IPython shortcut to execute system commands. To define an alias, simply type alias followed by the system command and any arguments for that command . For example In 1 alias nss netstat -lptn Not all processes could be identified, non-owned process info will not be shown, you would have to be root to see it all. Active Internet connections only servers Proto Recv-Q...

Scapy

If you like writing network code, you are going to love Scapy. Scapy is an incredibly handy interactive packet manipulation program and library. Scapy can discover networks, perform scans, traceroutes, and probes. There is also excellent documentation available for Scapy. If you like this intro, you should buy the book for even more details on Scapy. The first thing to figure out about Scapy is that, as of this writing, it is kept in a single file. You will need to download the latest copy of...

Linux System Administration Language

Being a sysadmin often means that you get thrown to the wolves. Rules, a predictable schedule, or even choice of an operating system is often out of your control. To be even a marginally effective sysadmin nowadays, you need to know it all, and we mean all the operating systems. From Linux, to Solaris, to OS X, to FreeBSD, it needs to be in your toolbelt. Although only time will tell, it does seem as if the proprietary operating systems, such as AIX and HP-UX, won't last forever, but they still...

Using Return Codes with Subprocess

One interesting thing to note about subprocess.call is that you can use return codes to determine the success of your command. If you have experience with programming in C or Bash, you will be quite at home with return codes. The phrases exit code and return code are often used interchangeably to describe the status code of a system process. Every process will have a return code when it exits, and the status of the return code can be used to determine what actions a program should take....

Creating a Custom Bootstrapped Virtual Environment

The release of virtualenv 1.0, which is current as of the writing of this book, includes support to create bootstrap scripts for virtualenv environments. One method of doing that is to call What this does is create a bootstrap script, which is like virtualenv, but with additional features to extend option parsing, adjust_options, and use after_install hooks. Let's go over how easy it is to create a custom bootstrap script that will install virtualenv and a custom set of eggs into a new...

Processing Module

So what is this processing module we have hinted at, anyway As of the printing of this book, processing is a package for the Python language which supports the spawning of processes using the API of the standard library's threading module One of the great things about the processing module is that it maps to the threading API, more or less. This means that you don't have to learn a new API to fork processes instead of threads. Visit http pypi.python.org pypi processing to find out more about...

Building an Apache Log Viewer Using Curses

Curses is a library that facilitates the creation of interactive text-based applications. Unlike GUI toolkits, curses does not follow an event handling and callback approach. You are responsible for getting input from the user and then doing something with it, whereas in GTK, the widget handles getting input from the user and the toolkit calls a handler function when an event occurs. Another difference between curses and GUI toolkits is that with GUI toolkits you are adding widgets to some...

Cross Platform Unix Programming in Python

While there are some significant differences between nix operating systems, there is much more in common that not. One way to bring the different versions of nix back together is to write cross-platform tools and libraries that bridge the divide between the operating system differences. One of the most basic ways to accomplish this is to write conditional statements that check for the operating systems, platform, and version in code that you write. Python takes the batteries included philosophy...

Graphical Images

The following two examples will continue the example of parsing an Apache logfile for the client IP address and the number of bytes that were transferred. In the previous section, our example generated a shelve file that contained some information that we want to share with other users. So, now, we will create a chart object from the shelve file to make the data easy to read usr bin env python shelve_file shelve.open 'access.s' items_list i l , i 0 for i in shelve_file.items bytes_sent i 0 for...

OS X Scripting APIs

Often, with OS X it is a requirement for a sysadmin to know a bit about high-level scripting that interacts with the actual UI itself. With OS X Leopard, Python, and Ruby, we're given first-class access to the Scripting Bridge. Refer to this documentation for more information One of the options for accessing the OSA, or Open Scripting Architecture, is py-appscript, which has a project page here Using py-appscript is quite fun and powerful, as it gives Python the ability to interact with the...

Solaris Systems Administration

From the late '90s to the early 2000s Solaris was a preferred, Big Iron distribution of Unix. Linux's metioric rise in the early 2000s rapidly cut into Solaris' and Sun was in some real trouble. However, recently, a lot of sysadmins, developers, and enterprises are talking about Sun again. Some of the interesting developments in Sun's future are a 6-month release cycle, just like Ubuntu, with a 18-month support window. It is also copying the single CD approach of Ubuntu as well and ditching the...

Daemonizer

Dealing with daemons is a given for anyone who has spent more than a cursory amount of time on Unix. Daemons do everything from handling requests to sending files to a printer such as lpd , fielding HTTP requests, and serving up files such as Apache's httpd . So, what is a daemon A daemon is often thought of as a background task that doesn't have a controlling terminal. If you are familiar with Unix job control, you may think that running a command with an amp at the end of it will make it a...

Extending NetSNMP

As we have discussed earlier, Net-SNMP is installed as an agent on most nix machines. There is a default set of information that an agent can return, but it is also possible to extend an agent on a machine. It is reasonably straightforward to write an agent that collects just about anything and then returns the results via the SNMP protocol. The EXAMPLE.conf file is one of the best sources for information on extending Net-SNMP, and it is included with Net-SNMP. Doing a man on snmpd.conf is also...

Building a Simple PyGTK

We'll create a simple piece of code which uses the already-mentioned gtk.Window and gtk.Button classes. Following is a simple GUI application that doesn't do anything useful except show some of the basic tenets of GUI programming. Before being able to run this example or write your own PyGTK app, you'll have to install PyGTK. This is pretty simple if you're running a relatively modern Linux distribution. It even looks pretty easy for Windows. If you're running Ubuntu, it should already be...