The scope of the system administrator role has changed dramatically over the years. The number of systems supported by a single engineer has also increased. As such, it is impractical to handcraft each installation, and there is a need to automate as many tasks as possible. The structure of systems varies from organization to organization, therefore system administrators must be able to create their own management tools. Historically, the most popular programming languages for these tasks were UNIX shell and Perl. They served their purpose well, and I doubt they will ever cease to exist. However, the complexity of current systems requires new tools, and the Python programming language is one of them.
Python is an object oriented programming language suitable for developing large-scale applications. Its syntax and structure make is very easy to read, so much so that the language is sometimes referred to as "executable pseudocode." The Python interpreter allows for interactive execution, so in some situations, you can use it instead of a standard UNIX shell. Although Python is primarily an object-oriented language, you can easily adopt it for procedural and functional styles of programming. Given all that, Python makes a perfect fit as a new language for implementing system administration applications. There are a large number of Linux system utilities already written in Python, such as the Yum package manager and Anaconda, the Linux installation program.
This book is about using the Python programming language to solve specific system administration tasks. We will look at the four distinctive system administration areas: network management, web server and web application management, database system management, and system monitoring. Although I will explain most of the technologies used in this book in detail, bear in mind that the main goal of this book is to show you the practical application of the Python libraries to solve rather specific issues. Therefore, I'm assuming that you are a seasoned system administrator.
As we go along with the examples, you will be asked to install additional packages and libraries. In most cases, I provide the commands and instructions to perform these tasks on a Fedora system, but you should be ready to adopt these instructions to the Linux distribution that you are going to use. Most of the examples work without many modifications on a recent OS X release (10.6.X) too.
I also assume that you have a background in the Python programming language. I will be focusing on introducing the specific libraries that are used in system administration tasks as well as some lesser known or less-often-discussed language functionality, such as the generator functions or the class internal methods, but the basic language syntax is not explained. If you want to refresh your Python skills I would recommend Beginning Python: From Novice to Professional, Second Edition by Magnus Lie Hetland (Apress, 2008).
All examples presented in this book assume the Python version 2.6 and will not work correctly with the latest Python 3 without additional modifications. Most of the examples rely on additional modules that have not yet been ported to this version of Python.
■Note Because of the line length limitations of the printed page, some lines of the code had to be split into two lines. This is indicated by the backslash character (\) at the end of the split line. When you use the code examples, you can either leave the structure as it is (i.e., with the wrapped lines), or you can join the two lines together, in which case you'll have to remove the backslash character from the code.
This book contains 13 chapters, and each chapter solves a distinctive problem. Some examples span multiple chapters, but even then, each chapter deals with a specific aspect of the particular problem.
In addition to the chapters, several other organizational layers span this book. First, I grouped the chapters by the problem type. Chapters 1 to 4 deal with network management issues; Chapters 5 to 7 talk about the Apache web server and web application management; Chapters 8 to 11 are dedicated to monitoring and statistical calculations; and finally, Chapters 12 and 13 focus on database management issues.
Second, I am maintaining a common pattern in all chapters. I start with the problem statement and then move on to gather requirements and through the design phase before going into the implementation section.
Third, each chapter focuses on one or more technologies and the Python libraries that provide the language interface to the particular technology. Examples of such technologies could be the SOAP protocol, application plug-in architecture, or cloud computing concepts.
Chapter 1: Reading and Collecting Performance Data Using SNMP
Most network attached devices expose the internal counters via the Simple Network Management Protocol (SNMP). This chapter explains basic SNMP principles and the data structure. We then look at the Python libraries that provide the interface to SNMP-enabled devices. We also investigate the Round Robin database, which is the de facto standard for storing the statistical data. Finally, we'll look at the Jinja2 template framework, which allows us to generate simple web pages.
Chapter 2: Managing Devices Using the SOAP API
Complicated tasks, such as managing the device configuration, cannot be easily done by using SNMP, because the protocol is too simplistic. Therefore, advanced devices, such as the Citrix Netscaler load balanacers, provide the SOAP API interface to the device management system. In this chapter, we'll investigate the SOAP API structure and the libraries that enable the SOAP-based communication from the Python programming language. We'll also look at the basic logging functionality using the built-in libraries.
Chapter 3: Creating a Web Application for IP Address Accountancy
In this chapter, we will build a web application that maintains the list of the assigned IP addresses and the address ranges. We will learn how to create web application using the Django framework. I'll show you the way Django application should be structured, how to create and configure the application settings and the URL structure. We'll also investigate how to deploy the Django application using the Apache web server.
Chapter 4: Integrating the IP Address Application with DHCP
This chapter expands on the previous chapter, and we will implement the DHCP address range support. We will also look at some advanced Django programming techniques such as customizing the response MIME type as well as serving AJAX calls.
Chapter 5: Maintaining a List of Virtual Hosts in an Apache Configuration File
This is another Django application that we are going to develop, but this time, our focus will be the Django administration interface. While building the Apache configuration management application, you'll learn how to customize the default Django administration interface with your own views and functions.
Chapter 6: Gathering and Presenting Statistical Data from Apache Log Files
In this chapter, our goal is to build an application that parses and analyses the Apache web server log files. Instead of taking the straightforward but inflexible approach of building a monolithic application, we'll look at the design principles of building plug-in based applications. You will learn how to use the object and class type discovery functions and how to perform a dynamic module loading.
Chapter 7: Performing Complex Searches and Reporting on Application Log Files
This chapter also deals with the log file parsing, but this time I'll show you how to parse complex, multiline log file entries. We are going to investigate the functionality of the open source log file parser tool called Exctractor, which you can download from http: //exctractor. sourceforge. net/.
Chapter 8: A Web Site Availability Check Script for Nagios
Nagios is one of the most popular open source monitoring systems, because its modular structure allows users to implement their own check scripts and thus customize the monitoring tool to their needs. In this chapter, we are going to create two scripts that check the functionality of a web site. We're going to investigate how to use the Beautiful Soup HTML parsing library to extract the information from the HTML web pages.
Chapter 9: Management and Monitoring Subsystem
This chapter starts a three chapter series in which we'll build a complete monitoring system. The goal of this chapter is not to replace mature monitoring systems such as Nagios or Zenoss but to show you the basic principles of the distributed application programming. We'll look at database design principles such as data normalization. We're also going to investigate how to implement the communication mechanisms between network services using the RPC calls.
Chapter 10: Remote Monitoring Agents
This is the second chapter in the series where we'll implement the remote monitoring agent components. In this chapter, I also describe how to decouple the application from its configuration using the ConfigParser module.
Chapter 11: Statistics Gathering and Reporting
This is the last part of the monitoring series, where I'll show you how to perform basic statistical analysis on the collected performance data. We're going to use scientific libraries—NumPy to perform the calculations and matplotlib to create the graphs. You'll learn how to find which performance readings fall into the comfort zone and how to calculate the boundaries of that zone. We'll also do the basic trend detection, which provides a good insight for the capacity planning.
Chapter 12: Automatic MySQL Database Performance Tuning
In this chapter, I'll show you how to obtain the MySQL database configuration variables and the internal status indicators. We'll build an application that makes a suggestion on how to improve the database engine performance based on the obtained data.
Chapter 13: Amazon EC2/S3 as a Data Warehouse Solution
This chapter shows you how to utilize the Amazon Elastic Compute Cloud (EC2) and offload the infrequent computation tasks to it. We're going to build an application that automatically creates a database server where you can transfer data for further analysis. You can use this example as a basis to build an on-demand data warehouse solution.
The Example Source Code
The source code of all the examples in this book, along with any applicable sample data, can be downloaded from the Apress web site at http://apress.com/book/view/1430226056. The source code stored at this location contains the same code that is described in the book.
Most of the prototypes described in this book are also available as open source projects. You can find these projects at the author's web site http://www.sysadminpy.com/.
Was this article helpful?