Contents

■About the Author xvi

■About the Technical Reviewer xvii

■Acknowledgements xviii

■Introduction xix

■Chapter 1: Reading and Collecting Performance Data Using SNMP 1

Application Requirements and Design 1

Specifying the Requirements 1

High-Level Design Specification 2

Introduction to SNMP 2

The System SNMP Variables Node 4

The Interfaces SNMP Variables Node 5

Authentication in SNMP 7

Querying SNMP from the Command Line 7

Querying SNMP Devices from Python 11

Configuring the Application 11

Using the PySNMP Library 13

Implementing the SNMP Read Functionality 18

Storing Data with RRDTool 18

Introduction to RRDTool 18

Using RRDTool from a Python Program 20

Creating a RoundRobin Database 20

Writing and Reading Data from the RoundRobin Database 22

Plotting Graphs with RRDTool 25

Integrating RRDTool with the Monitoring Solution 28

Creating Web Pages with the Jinja2 Templating System 31

Loading Template Files with Jinja2 32

The Jinja2 Template Language 33

Generating Web Site Pages 35

Summary 39

■Chapter 2: Managing Devices Using the SOAP API 41

What Is the SOAP API? 41

The Structure of a SOAP Message 42

Requesting Services with SOAP 42

Finding Information about Available Services with WSDL 44

SOAP Support in Python 45

Converting WSDL Schema to Python Helper Module 46

Defining Requirements for Our Load Balancer Tool 47

Basic Requirements 48

Code Structure 48

Configuration 49

Accessing Citrix Netscaler Load Balancer with the SOAP API 50

Fixing Issues with Citrix Netscaler WSDL 50

Creating a Connection Object 51

Logging In: Our First SOAP Call 54

Gathering Performance Statistics Data 59

SOAP Methods for Reading Statistical Data and Their Return Values 59

Reading System Health Data 60

Reading Service Status Data 62

Automating Administration Tasks 65

Device Configuration SOAP Methods 66

Setting a Service State 66

A Word About Logging and Error Handling 68

Using the Python logging Module 68

Handling Exceptions 72

Summary 73

■Chapter 3: Creating a Web Application for IP Address Accountancy 75

Designing the Application 75

Setting Out the Requirements 75

Making Design Decisions 76

Defining the Database Schema 77

Creating the Application Workflow 79

The Basic Concepts of the Django Framework 80

What Is Django? 81

The Model/View/Controller Pattern 81

Installing the Django Framework 82

The Structure of a Django Application 83

Using Django with Apache Web Server 85

Implementing Basic Functionality 88

Defining the Database Model 88

URL Configuration 91

Using the Management Interface 93

Viewing Records 96

Using Templates 99

Deleting Records 102

Adding New Records 103

Modifying Existing Records 105

Summary 106

■Chapter 4: Integrating the IP Address Application with DHCP 107

Extending the Design and Requirements 107

Extending the Database Schema 109

Additions to the Workflow 110

Adding DHCP Network Data 110

Defining Data Models 110

Additional Workflows 111

The Add Function 111

The Modify Function 113

The Delete Function 113

Extending DHCP Configuration with Address Pools 113

The Address Pool Data Model 114

Displaying DHCP Network Details 114

The Add and Delete Functions 116

Reworking the URL Structure 116

Generating URLs in the Model Class 117

Reverse Resolution of URLs 117

Assigning Names to URL Patterns 118

Using URL References in the Templates 118

Adding Client Classification 120

Additions to the Data Model 120

Using Template Inheritance 121

Class Rules Management 122

Generating the DHCP Configuration File 127

Other Modifications 131

Resolving IPs to Hostnames 131

Checking Whether the Address Is In Use 132

Summary 135

■Chapter 5: Maintaining a List of Virtual Hosts in an Apache Configuration File 137

Specifying the Design and Requirements for the Application 137

Functional Requirements 138

High-Level Design 138

Setting Up the Environment 138

Apache Configuration 139

Creating a Django Project and Application 139

Configuring the Application 140

Defining the URL Structure 141

The Data Model 142

The Basic Model Structure 143

Modifying the Administration Interface 147

Improving the Class and Object Lists 147

Adding Custom Object Actions 152

Generating the Configuration File 154

Summary 157

■Chapter 6: Gathering and Presenting Statistical Data from Apache Log Files 159

Application Structure and Functionality 159

Application Requirements 160

Application Design 160

Plug-in Framework Implementation in Python 160

The Mechanics of a Plug-in Framework 161

Creating the Plug-in Framework 163

Log-Parsing Application 167

Format of Apache Log Files 167

Log File Reader 169

Calling the Plug-in Methods 173

Plug-in Modules 177

Installing the Required Libraries 177

Writing the Plug-in Code 179

Summary 180

■Chapter 7: Performing Complex Searches and Reporting on Application Log Files 181

Defining the Problem 181

Why We Use Exceptions 184

Are Exceptions Always a Bad Sign? 184

Why We Should Analyze Exceptions 184

Parsing Complex Log Files 186

What Can We Find in a Typical Log File? 186

The Structure of an Exception Stack Trace Log 187

Handling Multiple Files 189

Handling Multiple Files 189

Using the Built In Bzip2 Library 193

Traversing Through Large Data Files 193

What Are Generators, And How Do We Use Them? 193

Detecting Exceptions 195

Detecting Potential Candidates 195

Filtering Legitimate Exception Traces 196

Storing Data in Data Structures 197

The Structure of Exception Stack Trace Data 198

Generating an Exception Fingerprint for Unknown Exceptions 198

Detecting Known Exceptions 201

Producing Reports 206

Grouping Exceptions 207

Producing Differently Formatted Output for the Same Dataset 208

Calculating Group Statistics 208

Summary 210

■Chapter 8: A Web Site Availability Check Script for Nagios 211

Requirements for the Check System 211

The Nagios Monitoring System 212

Nagios Plug-In Architecture 212

The Site Navigation Check 213

Installing the Beautiful Soup HTML Parsing Library 214

Retrieving a Web Page 214

Parsing the HTML Pages with Beautiful Soup 216

Adding the New Check to the Nagios System 222

Emulating the User Login Process 224

Summary 229

■Chapter 9: Management and Monitoring Subsystem 231

Design 231

The Components 231

The Data Objects 233

The Data Structures 234

Introduction to Data Normalization 235

Configuration Data 237

Performance Data 240

Scheduling 241

Site Configuration 242

Communication Flows 243

XML-RPC for Information Exchange 244

CherryPy 246

The Server Process 247

Storing Data in a SQLite3 Database 247

Actions 252

The Scheduler 255

Actions 255

Running Multiple Processes 255

Running Methods at Equal Intervals 260

A Cron-Like Scheduler 263

Ticket Dispatcher 264

Summary 267

■Chapter 10: Remote Monitoring Agents 269

Design 269

The Passive Component 269

Architecture 269

Actions 269

The Security Model 271

Configuration 271

The ConfigParser Library 271

The Configuration Class Wrapper 277

The Sensor Design 281

Running External Processes 282

Using the subprocess Library 282

Controlling the Running Processes 286

Communicating with External Processes 289

Automatically Updating Sensor Code 292

Sending and Receiving Binary Data with XML-RPC 292

Working with Files and Archives (TAR and BZip2) 293

Summary 296

■Chapter 11: Statistics Gathering and Reporting 297

Application Requirements and Design 297

Using the NumPy Library 297

Installing NumPy 298

NumPy Examples 298

Representing Data with matplotlib 309

Installing matplotlib 309

Understanding the Library Structure 310

Plotting Graphs 311

Saving Plots to a File 316

Graphing Statistical Data 317

Collating Data from the Database 317

Drawing Timescale Graphs 318

Summary 328

■Chapter 12: Automatic MySQL Database Performance Tuning 329

Requirements Specification and Design 329

Basic Application Requirements 330

System Design 330

Modifying the Plug-in Framework 332

Changes to the Host Application 332

Modifying the Plug-in Manager 333

Writing the Producer Plug-ins 334

Accessing the MySQL Database from Python Applications 334

Querying the Configuration Variables 339

Querying the Server Status Variables 341

Collecting the Host Configuration Data 342

Writing the Consumer Plug-ins 343

Checking the MySQL Version 344

Checking the Key Buffer Size Setting 346

Checking the Slow Queries Counter 347

Summary 348

■Chapter 13: Using Amazon EC2/S3 as a Data Warehouse Solution 349

Specifying the Problem and the Solution 349

The Problem 349

Our Solution 350

Design Specifications 350

The Amazon EC2 and S3 Crash Course 351

Authentication and Security 351

The Simple Storage System Concepts 352

The Elastic Computing Cloud Concepts 354

User Interfaces 358

Creating a Custom EC2 Image 359

Reusing Existing Images 359

Making Modifications 360

Bundling the New AMI 362

Controlling the EC2 Using the Boto Python Module 364

Setting Up the Configuration Variables 364

Initializing the EC2 Instance Programmatically 365

Transferring the Data 369

Destroying the EC2 Instance Programmatically 369

Summary 372

Was this article helpful?

0 0

Post a comment