Day 1: The case for using PowerShell to Analyze Your Log Files.
Over the years, I have worked with a variety of products that utilized some type of text based log file. I’ve had to use these log files for various tasks. One of my favorite examples of automation came from an engineering company that I once worked for.
At this particular company, we had a small problem. When I arrived as the Network Administrator, I noticed that single licensed software was freely passed around without any controls. When I requested to lock up all the copies, I was told not to worry about it. A year later, we received a letter from the creator of the software informing us that we are being audited. This is a standard practice in the industry so normally I would consider this not to be a big deal. Oh, but wait. I have no control over this software.
At the conclusion of my audit, I discovered that we were now 6 figures in debt to this company. All of the sudden, it became my problem. After I successful removed myself from the trail of accountability using a few emails that I saved, I began the work of cleaning up this mess while management began the process of how to pay for the mess that they created. My part was simple. Once I was able to confiscate all the software and all copies, we simply removed the software from all systems. Now, the fun part.
When I was presented with the plan for future licensing from the the regional managers, they decided that they only needed one license per 3 users. I informed them that our new license server for the product would be depleted of all available license in under 4 hours. I was reminded that the regional managers studied this issues thoroughly and that my concerns were not warranted. OK, so I had them sign off that they made the decision with my protest duly noted and I implemented their grand scheme. Once we turned it on, the entire pool of licenses were depleted in just 2 hours.
The individuals using this product were each working on multimillion dollar projects. Each one of them were calling me demanding that I provide them with a license. This placed me in the situation where I had to continuously decipher a text based log file and start calling people and ask them to release their license. Most of the time their answer was “no.”
Well, this became too much of a burden so off I went and started scripting. In the end, I developed a script that deciphered the log file and determined who had a license and for how long. This information was then further processed into HTML code and then placed online. I then integrated the script with Active Director so this internal web site would also provide the license holders cell phone number. This stopped nearly all calls to me and created what I called “Peer Accountability”. Under this model of Peer Accountability, if one user was keeping a license to long they would hear about it from their peers. It also allowed me to create a clear and concise report of every failed attempt to get a license from each region and present some concrete numbers to the regional managers. In the end, they had to purchase one license per user. in the end, it made economic sense to spend a few hundred thousand dollars once to make a few million dollars each month.
Since different products have different log file formats, it is difficult to create a script that can read them all. This series of posts will walk you through the steps needed to create scripts to read your text based log files and create PowerShell objects out of each record. Here is a list of steps that we will need to accomplish:
1. Identify a rule to separate records
2. Identify a rule to separate property names from property values
3. Create an object that contains all possible property names.
4. Read the log records into the object.
5. Send to the pipeline.
We will first work on a log file that has one record per line and then build a script to handle logs that span multiple lines. This code will not work for your specific text based log. You will have to create the exact code for your specific log so used these examples as general steps in the process. In the end, you will have PowerShell objects that you can then send into the pipeline to utilize all the automation that PowerShell has to offer.