Storing and Applying Filters

All exception detection and classification rules are going to be stored in an array. Each array element is a dictionary that contains precompiled regular expressions, both group and description fields ,and finally an ID string, which is just an MD5 hash of regular expression strings. This ID can be used later in referencing particular exception groups and will remain unique as long as the rules are not changed.

Using precompiled regular expressions increases search speed significantly, because they are already validated and converted to bytecode ready for execution.

Configuration parsing and importing are done during the class initialization, as you can see from the example in Listing 7-12.

Listing 7-12. Class initialisation and configuration import class ExceptionContainer:

config = minidom.parse(CONFIG_FILE)

for et in config.getElementsByTagName('exception_types'): for e in et.getElementsByTagName('exception'): m =

m.update(e.attributes['logline'].value) m.update(e.attributes['headline'].value) m.update(e.attributes['body'].value) self.filters.append({ 'id' : m.hexdigest(),


re.compile(e.attributes['headline'].value), 'bl_re':

re.compile(e.attributes['body'].value), 'group': e.attributes['group'].value, 'desc' : e.attributes['desc'].value, })

When the insert method (described in detail earlier) is called, it will loop through the list of filters and attempt to search for matching strings. When such a string is found, the exception details are either stored or the running counter for the group is increased, depending on whether this exception has already been encountered in the log file. If no matches were found, the heuristic categorization method will be executed as shown in Listing 7-13.

Listing 7-13. Code to match custom categorisation rules def insert(self, suspect_body, f_name=""):

if self.is_exception(lines[1]): self.count += 1

logged = False for f in self.filters:

if f['ll_re'].search(log_l) and f['hl_re'].search(hd_l) and f['bl_re'].search(bd_l): logged = True if f['id'] in self.exceptions:

break if not logged:

# ... unknown exception, try to automatically categorise

1, log_l, hd_l, bd_l, f_name, f['desc'], f['group'], }

Was this article helpful?

0 0

Post a comment