Sawmill - Universal log file analysis and reporting

2008-08-28: Improved "Process Logs" Output coming in 7.2.16

We have made several improvements to the flexibility of the output of the "process logs" action in Sawmill, which is typically used to convert any log data to comma-separated format. With the new options, the delimiter is customizable, the header is optional, and the output date/time format is customizable. By specifying tabs between fields, a timestamp in the format "YYYY-MM-DD HH-MM-SS", and no header, this makes Sawmill's output compatible with BCP. One of our customers is using this to quickly import their Windows Media, Flash Media, Quicktime Streaming Server, and other media logs into their Microsoft SQL Server database, using Sawmill for the log conversion stage, and FreeBCP for the import. This new functionality will be available in Sawmill 7.2.16.

2008-08-27: Professional Services For Streaming Media, continued.

This week Flowerfire is again on site with a large streaming media hosting provider, helping them to implement billing and reporting systems with Sawmill. Sawmill is used to parse the logs from their Flash Media Servers, HTTP media servers, Real Media Servers, Quicktime Streaming Servers, and Microsoft Media Servers, integrating the per-customer bandwidth information from the disparate formats into a unified Microsoft SQL table which can be queried for billing purposes. If you have a similar need for Sawmill's log parsing, conversion, and import capabilities, we can get you up to speed quickly. Contact support@sawmill.net for a Professional Services quote.

2008-08-26: Professional Services For Streaming Media

Today, engineers from Flowerfire are on-site at a streaming media hosting provider, working to integrate Sawmill into their existing Microsoft SQL Server environment for billing and reporting. Sawmill is used to parse logs from their Flash Media, Microsoft Media, and Quicktime Streaming servers, and with a small amount of additional scripting, the parsed data is to be injected into their Microsoft SQL Server for use in billing calculations. If you would like to use Sawmill to parse, import, and report on your log data, our Sawmill Professional Services experts can help you get your system up and running quickly and efficiently.

2008-08-25: August Newsletter::Using Sawmill to query log data with SQL

The August Sawmill newsletter goes through the step-by-step process of creating a profile which uses MySQL as the back-end database. There is an example of analyzing a 5000 line log file, and how to query it for specific hostname information. There are other examples of getting the top 10 hostnames, and how to generate a report from these. If you didn't get the newsletter and would like to read it, or if you would like to join the newsletter mailing list, please email support@sawmill.net.

2008-08-22: Use uppercase file types in log filters

Just a reminder, if you are trying to filter on ASP or HTML file types, make sure that you use uppercase letters for the file types, creating a string with "asp" or "html" will not filter the files.

2008-08-21: New support for Firebox X Core e-Series

We have added support in Sawmill for analyzing log files, and generating reports, from Watchguard Firebox X Core e-Series devices. The new plug-in is available for free through technical support, and will be included in future releases of Sawmill.

2008-08-20: Upgrading is a breeze!

If you are using Sawmill version 7.2x, you can upgrade to the latest version at any time without losing data. Merely download the latest version, and install it over your existing version. There is no need to rebuild the database! Also, if you were using a trial license and have purchased a full license, all of your configuration files, logs and setup will be preserved, nothing is lost from your trial version.

2008-08-19: See us at Streaming Media West!

We will be at Streaming Media West, September 23-25, in San Jose, CA. Come see us at booth 420, we will be showing off our newest Sawmill version, giving away t-shirts and having show specials.

2008-08-18: Enhanced support for Windows Event Log reporting

We have enhanced support in Sawmill for reporting on Microsoft Windows Event Logs exported using the dumpevt tool. The enhanced plug-in now supports variable time formats, and Directory Services logs. The new plug-in is available for free through technical support, and will be included in future releases of Sawmill.

2008-08-15: New support for Atlassian JIRA log format

We have added support in Sawmill for analyzing log files, and generating reports, from Atlassian JIRA. The new plug-in is available for free through technical support, and will be included in future releases of Sawmill.

2008-08-14: New Article discusses Page Tagging vs. Log Analysis

An independent article in Search Engine Journal discusses the downsides of page tagging (e.g., Google Analytics), and the benefits of log analysis (e.g., Sawmill). We suggest you read it, to understand some of the benefits of log analysis. And while you're at it, why not look at our own Page Tagging White Paper, on the same subject?

2008-08-11: Professional Services Get You Reports Faster And Better

Sawmill Professional Services can help you get the most out of Sawmill. Sawmill is a highly flexible product, really an application server with its own built-in language, database, and reporting engine, which makes it possible to do almost anything with Sawmill. Many customizations can be done by novice users, but really advanced custom work requires a high level of Sawmill expertise. If there's something you'd like Sawmill to do, and you don't know how to configure it, we can help. Contact us at support@sawmill.net, and ask for Professional Services. For a fee, our experts can make Sawmill do whatever type of analysis you require. The most frequent paid service is the creation of a new plug-in, so Sawmill can analyze new log formats, even proprietary ones, the possibilities are unlimited. Want a new type of report? Want Sawmill's reports to integrate with an external data source? Want advanced graphs or charts? Our experts can do it all.

2008-08-08: Sawmill Training from GeniusDV.com

Our trainer, Derrick Freeman, will be offering Sawmill Professional one day training classes for the PC and Mac. All classes are being taught in Orlando, FL. The class for the PC is Sept. 11, with another class on Nov. 14th. The Mac classes will be Oct. 10th and then Dec. 17th. Please see: http://www.geniusdv.com/video_streaming_training/ for more details.

2008-08-07: The Apache Custom log format plug-in works with other access logs too!

The Apache Custom plug-in is designed to allow you to paste the format string that you used to create a custom Apache log format into a field in Sawmill's profile wizard to create a custom profile for parsing your Apache Custom logs. Actually, this plug-in will work with any access log that is made up of the same kinds of elements as Apache allows, such as protocol, method, url and cookie. Other fields can be named and added with the %{Field}n directive. (If there are any valid directives that we don't support, this should cover them also, but we'd appreciate hearing from you if you find any.) Apache's directives are here http://httpd.apache.org/docs/2.2/mod/mod_log_config.html. Note that modifiers < and > are not supported by Sawmill, but leaving them out causes no problems at all in your reports. To select the Apache Custom format, choose "Apache Custom Log Format" from the manual selection list in the profile creation wizard, click Next and follow instructions from there.

2008-08-04: Difference between database rebuild versus update?

We recently received this question in our forum and we wanted to mention it here... clarification for others. An update adds to the existing database, any changes will be seen as new data. If you have made changes, say to a log filter, then you will want to rebuild the database, all the data will be processed again, and the database will be rebuilt as "new".

2008-08-01: Welcome CompuCom!

We would like to welcome our latest reseller, CompuCom! CompuCom is located in Texas and has won many awards for their outstanding work in IT Infrastructure, Application, Software and Hardware Services. You can read more about them at www.compucom.com.

2008-07-31: Welcome Realmobile!

We would like to welcome one of our newest resellers, Realmobile! They are based in Seoul, Korea and will be doing the Korean translation for Sawmill. For more information about them, visit www.realmobile.net.

2008-07-30: Customer Testimonial

We would like to share with you a recent testimonial from Mark at Digital Marketing & Imagery:"Sawmill is remarkable...A true software utility that has been thought out from top to bottom. Anyone who works with server logs, which is everyone at some time or another, should have this program. I'll make sure I give my counterparts a heads up!" Thanks Mark!

2008-07-29: Sawmill Sale, Ending Soon!

Attention all Sawmill customers, new and old! Our July sale is ending on the 31st, please act soon for discounted prices. All upgrades are 50% off, act now!

2008-07-28: Use Professional Services To Supercharge Sawmill

Over the past few weeks, we've been working with a Fortune 1000 company, configuring their Sawmill installation to analyze their JBoss server logs. With just a few hours of professional services, we have enhanced their version of the JBoss plug-in to extract the JVM name from the filename, track and report exceptions more precisely, extract and report on specific additional fields from the message, and add a link to Log Detail to pop up their custom script for viewing additional information about a particular event. In addition, we spent a few hours helping them debug and tune their NAS and virtual machines to work best with Sawmill, and to set up a Sawmill cluster for high-availability. We've done this in a fraction of the time it would have taken them to learn to do it themselves, while training them to maintain it and make similar modifications in the future, which has made our Professional Services a highly cost-effective choice for them. If you'd like something different from Sawmill, we can do it--Sawmill is infinitely customizable, and our experts can design and implement any customization you want. So if Sawmill does most of what you want, but not all, why not get the rest, using Professional Services? Contact sales@sawmill.net for more information.

2008-07-25: Adobe Flash vs. Macromedia Flash

Sawmill 7.2.15 (the current shipping version) expected Flash Media Server to report itself as "Macromedia Flash." That's right for older versions, but with Adobe's acquisition of Macromedia, the latest Flash Media Server logs now report themselves as "Adobe Flash." Sawmill's plug-in will be updated to reflect this in version 7.2.16, but until then, you'll need to manually select Flash as the format, when analyzing and reporting very recent Adobe Flash logs. You can also contact Sawmill support if you'd like an updated plug-in immediately.

2008-07-24: Sawmill 8 Licensing Tiers

If you're considering purchasing Sawmill 7 with the intent of upgrading to Sawmill 8 when it becomes available, please note that Sawmill 8 will have four license tiers (Lite, Professional, Advanced, and Enterprise), rather than the three tiers in Sawmill 7 (Lite, Professional, and Enterprise). The Sawmill 8 "Advanced" tier is the equivalent of the Sawmill 7 "Enterprise" tier, and a new high-end Enterprise tier has been added with the new and powerful Sawmill 8 enterprise features (Oracle, Microsoft SQL Server, ODBC, RBAC, real-time, and more). So the upgrade path from Sawmill 7 Enterprise will be to Sawmill 8 Advanced (which has all the features of Sawmill 7 Enterprise)--there will be an additional licensing charge to upgrade from Sawmill 7 Enterprise to the more powerful Sawmill 8 Enterprise.

2008-07-23: Sawmill 8 Status

Sawmill 8 alpha phase continues at a nice clip; we are getting ever closer to "beta." Sawmill 8 is nearly ready for internationalization, and we continue to resolve all major issues in preparation for beta. We have a very limited alpha test list; if you'd like to be on it, and if you have experience testing software, please email alpha@sawmill.net.

2008-07-22: Running Sawmill As An Ubuntu Service

We have created an Ubuntu startup script (init.d script) for Sawmill. This allows users of the popular Ubuntu Linux distribution to easily install Sawmill to run when the system starts, and turn down cleanly when it shuts down. This script will be part of all future distributions of Sawmill; if you want it immediately, please contact Sawmill Support.

2008-07-21: July Newsletter: Adding Users Automatically

The Sawmill Newsletter, July 2008 edition, discusses adding users to a Sawmill installation automatically, which is useful when integrating Sawmill into a larger automated multi-user environment. The newsletter includes a script written in Salang (the Sawmill Language), and includes a detailed line-by-line explanation of the script's functionality, so it is also useful for anyone interested in learning Salang (Sawmill is almost infinitely configurable using Salang). If you didn't get the newsletter and would like to read it, or if you would like to join the newsletter mailing list, please email support@sawmill.net.

2008-07-18: Sawmill 8 Reaches Alpha Stage

We are pleased to announce that Sawmill 8, the next major release of Sawmill, has reached the "alpha" stage of development. This means that in our internal version of Sawmill 8, all features have been implemented. Now, we begin work on beta, which will become publicly available when all major bugs in Sawmill 8.0.0a1 are fixed. Major features of Sawmill 8 include support for Oracle and MS SQL, role-based authentication, real-time reporting, and a completely redesigned web interface. If you'd like a very early look at Sawmill 8, contact support@sawmill.net.

2008-07-17: Improving the performance of database updates

There are ways that you can improve the performance of Sawmill during the nightly updates. If your dataset gets very large or has a large number of files, you can make this update faster by: Using a local log file source, use a local drive, turn on "skip processed files on update by pathname", keep the new data in a separate directory, turn off active virus scanning of Sawmill's directories, and of course, the faster the CPU, the better! For more information, please email support@sawmill.net.

2008-07-16: Spider Traffic

We get questions all the time about filtering out certain types of traffic. One the main types of traffic is from spiders. You may want to exclude spider traffic from your reports, to get real data showing the number of humans who access your website. You can do this by using a log filter or a report filter. More information about how to do this can be found within our Technical Manual, under "Using Log Filters".

2008-07-15: Monthly newsletters

On the 15th of every month, we send out a newsletter to our customers. These newsletters deal with many topics, including questions from our forum and support emails. Recent newsletters have addressed: Showing usernames in reports, percentages for unique values, excluding your own traffic, using the zoom feature and how to create a rolling database, to name a few. If you would like to be added to the newsletter, please email newsletter@sawmill.net, with the subject line of "SUBSCRIBE".

2008-07-14: Enhancement to Microsoft Exchange Server 2007 CSV reporting

We have enhanced our Microsoft Exchange Server 2007 CSV plug-in for Sawmill. The new version reports multiple recipient information, making it possible to see the total number of recipients for a particular message, for a particular domain, or for anything else. The new plug-in is available for free through technical support, and will be included in future releases of Sawmill.

2008-07-11: Let us host and manage your Sawmill

To get the best performance out of Sawmill, you need a high-end, 64-bit, Linux- or Unix-based, multiprocessor system; and to get the best information out of it, you need someone experienced in Sawmill configuration. If you want to run Sawmill at full power, but don't have the budget for the dedicated hardware or the dedicated expert, consider letting us host and manage it for you. Our state-of-the-art servers are ready to deliver Sawmill statistics faster and better than your own in-house systems. We'll manage the server for you, and help you configure your Sawmill installation to get all the information you want and need. Our Sawmill experts can coax amazing reports from Sawmill in a short time, using its sophisticated customization features. Why stick with the standard reports when you can have exactly the reports you want? To purchase Managed Sawmill, contact sales@sawmill.net.

2008-07-10: How many profiles do I need?

We often get this question from our customers, before they purchase Sawmill. What is a profile and how many do I need? A profile is one type of log format and this builds into a report with information about that log format. If you want to track two different log formats, they would each have two different profiles. If you are an ISP managing multiple sites, you will create a profile for each site, with each customer seeing only their site's statistics. In this case, you would use the "create many profiles" feature, to set up each profile for a particular domain. There is more in depth information on our website, about cloning profiles and working with templates. But hopefully, you will have a better idea of where to start answering the question of how many profiles do I need for my organization.

2008-07-09: Upgrade to Enterprise at half price, July only!

If you're using Sawmill Lite, or Sawmill Pro, but want some of the advanced features provided by Sawmill Enterprise, the Sawmill July Sale is your ticket to ride. Sawmill Professional is vastly more powerful than Lite, due to the extensive configuration capabilities available only in Professional; and Sawmill Enterprise supports MySQL, multi-processor builds for higher performance, and "open source" customization of the entire user interface through Salang. If you yearn for features more advanced than your current tier provides, July is the month to buy--because in July, all upgrades are 50% off.

2008-07-08: July Sale

Don't miss the Sawmill July sale! For the month of July only, all upgrades are 50% off. Upgrading from an earlier version of Sawmill, to Sawmill 7, is half price for this month, and so is upgrading from a smaller license to a larger one (e.g., adding more profiles, or moving up a tier). If you're considering expanding the capabilities of your Sawmill installation by adding more profiles or more features, this is the month to do it.

2008-07-07: PCI compliance via email alerting

We've heard recently from several customers asking about PCI compliance with Sawmill. As described in our PCI white paper (in the White Papers section of the Downloads page on sawmill.net), Sawmill can be used to implement the network-related components of PCI compliance. Sawmill can examine log data (like Windows Event Log data) as it is generated, looking for violations based on rules you specify, and sending email when the rules are triggered. This is a specific application of Sawmill's general capability to do alerting based on a real-time log data stream.

2008-07-04: Enhancement to Microsoft Exchange Server 2000 W3C parsing

We have enhanced our Microsoft Exchange Server 2000 W3C plug-in for Sawmill. The new version handles email addresses better when they contain apostrophes and other special characters, providing better support for non-English log data. The new plug-in is available for free through technical support, and will be included in future releases of Sawmill.

2008-07-03: Sawmill July Sale!

During the month of July, we are offering a sale to our current customers.... We are cutting the cost in half for upgrading, adding new profiles and upgrading from Lite to Professional or Professional to Enterprise. If you are receiving our monthly newsletters, you will be receiving an email from us. If you're not on our mailing list, but would like to take advantage of this offer, please email us at sales@sawmill.net.

2008-07-02: IronPort C-Series Reporting Enhanced

We have enhanced support for IronPort C-Series logs, to track aborted messages better. Previously, these messages were tracked with single events in Sawmill's database, but did not include information about the recipients. Now, the plug-in adds a separate event for each recipient, tracking it with a new numerical field, "message deliveries aborted." This allows for easy reporting of recipient information for messages filtered by the anti-spam or anti-virus filters on IronPort devices.

2008-07-01: Adjusting for Daylight Savings Time

We have added a new advanced Log Filter example to the Sawmill Technical Manual (version 7.2.16 or later, or prerelease). This example illustrates the power of the built-in Salang language, by computing the DST date range for each year in the log data, and automatically offsetting the date/time value for each line of log data, depending on whether it's during Daylight Savings Time. The "Using Log Filters" chapter in the technical manual contains other advanced examples of log filtering using Salang.

2008-06-30: New Support For Radware DefensePro

We have added support in Sawmill for analyzing log files, and generating reports, from Radware DefensePro. The new plug-in is available for free through technical support, and will be included in future releases of Sawmill.

2008-06-27: Enhanced support for Kerio Mailserver

In response to many requests for improved Kerio Mailserver support, we have enhanced our Kerio plug-in. The new version supports Mailserver 6.5 logs, which provide more detailed reporting of multi-recipient messages, including the number of messages delivered, delayed, bounced, spam-filtered, etc. The new plug-in also recognizes and reports on the security and spam logs (previous versions reported only the main mail log). The new plug-in is available for free through technical support, and will be included in future releases of Sawmill.

2008-06-26: Sawmill works with SafeSquid

We have joined forces with SafeSquid to augment their enterprise-caliber HTTP firewall with our enterprise-caliber Sawmill reporting solution. If you use SafeSquid and want reports on your firewall traffic, Sawmill is your solution.

2008-06-25:

Sawmill now supports processing of Microsoft Internet Information Services (IIS) web server log files, which have been forwarded to a syslog server. Sawmill has always supported standard IIS logs, but for those who are forwarding their IIS log data to a separate syslog server (we recommend Kiwi Enterprises Kiwi Syslog Daemon on Windows), Sawmill will now recognize the format as IIS W3C logged through syslog, and recognize the particular syslog server (Sawmill recognizes and supports all major syslog servers), and generate reports containing all syslog fields, and web server fields and reports.

2008-06-24:

We have enhanced Sawmill's existing support for analyzing log files and generating reports from Symantec Mail Security. The new plug-in handles a new type of log file from Symantec Mail Security version 5, and reports on additional fields, including returned disposition, policy query, filtering policy name, description, and destination. The new plug-in is available for free through Sawmill technical support, and will be included in future releases of Sawmill.

2008-06-23: Sawmill can be your SaaS solution!

Sawmill can be your software as a service solution. We offer hosting services, support and customization. We can reduce your up-front costs for hardware, training and ongoing maintenance by allowing us to host your Sawmill configuration. Sawmill is easily scalable and can serve you and your customers' needs. For more information, please contact support@sawmill.net.

2008-06-20: New Support For TrendMicro InterScan VirusWall 6

We have added support for analyzing log files from TrendMicro InterScan VirusWall 6 with Sawmill, and generating reports. The new plug-in reports on the urlaccesslog format, with other formats to be implemented if requested. The new plug-in is available for free through technical support, and will be included in future releases of Sawmill.

2008-06-19:

We have added support in Sawmill, for analyzing the logs of SmarterTools SmarterMail, and generating reports from them. The new plug-in reports on sender, recipient, server domain, number of messages and bytes queued and delivered, number of messages failed (and associated error messages) and more. The new plug-in is available for free through Sawmill technical support, and will be included in future releases of Sawmill.

2008-06-18: Support for Astaro Security Gateway

In response to multiple requests, we have added support for Astaro Security Gateway format, to analyze the logs from Astaro Security Gateway devices, and generate reports. The new plug-in reports on both proxy and firewall events. The new plug-in is available for free through technical support, and will be included in future releases of Sawmill.

2008-06-17: Sawmill Training at Wichita State University

Our trainer, Derrick Freeman will be offering Sawmill training during June at Wichita State University. If you're going to be in the area between the 19th or 20th, contact Derrick for more information. You can email him at derrick@geniusdv.com. Derrick will be doing his Video Streaming and Compression class and show you how you can use Sawmill to analyze your log data.

2008-06-17: Enhanced support for Ipswitch IMail

We have significantly enhanced Sawmill's support for Ipswitch IMail log analysis and reporting. The new version of our Imail plug-in now reports delivery errors in a separate Error Messages report, reports relay hosts and IPs, and includes the number of errors in every report (for instance, the number of errors per day in the Days report). It also reports separately on messages (and bytes) delivered and relayed.

2008-06-16: Using Sawmill Professional Services

Sawmill Professional Services can help you get the most out of Sawmill. Sawmill is a highly flexible product, really an application server with its own built-in language, database, and reporting engine, which makes it possible to do almost anything with Sawmill. Many customizations can be done by novice users, but really advanced custom work requires a high level of Sawmill expertise. If there's something you'd like Sawmill to do, and you don't know how to configure it, we can help. Contact us at support@sawmill.net, and ask for Professional Services. For a fee, our experts can make Sawmill do whatever type of analysis you require. The most frequent paid service is the creation of a new plug-in, so Sawmill can analyze new log formats, even proprietary ones, but the possibilities are unlimited. Want a new type of report? Want Sawmill's reports to integrate with an external data source? Want advanced graphs or charts? Our experts can do it all.

2008-06-12: Become a Sawmill Reseller!

If you are a Sawmill user, you already know the benefits....but if you are also a reseller, then add Sawmill to your product line. We have worldwide opportunities for resellers, and sales incentives to keep you and your customers happy. For more information please email us, sales@sawmill.net.

2008-06-11: Accessing log data from remote drives

A customer asked us about using UNC paths (or mapped drives) to access log data which is on a different Windows system. Sawmill supports that in several ways. You can use a mapped drive letter, but if Sawmill is running as a service, that is not recommended--Microsoft recommends against using mapped drives from a service. Instead, you can use UNC paths (\\server\share\...). If the share is password-protected, Sawmill's "Manage Drives" window lets you authenticate to it from the service. From then on, you can refer to the log files by UNC path, for seamless processing of remote logs.

2008-06-10: Page Tagging vs. Log Analysis White Paper

Please see our new white paper on the pros and cons of page tagging versus using log file analysis. All of our white papers can be found under the menu, Download.

2008-06-09: Intel or SPARC?

A customer contacted us, asking whether they should use an UltraSPARC Solaris system, or an Intel Solaris system. Sawmill runs on either, and if you have a SPARC system already, it can be an acceptable system for Sawmill. But if you have a choice, there's no comparison: Intel (or AMD) always beats SPARC for Sawmill. SPARC systems are great for certain types of tasks, especially for slamming through huge numbers of transactions per second. They are highly parallel, and for multi-threaded integer-based work, they outperform Intel. But Sawmill's work involves floating point calculations, and tends to come in huge batches, not many small transactions; and the work is not easily split across multiple processors. For that type of work, the slower clock rate and shared floating point units severely hurt performance, versus a comparably priced Intel system. Intel is always the better choice for Sawmill, and the performance difference is often 4x.

2008-06-06: Internal database, or MySQL?

Which database should you use, the internal or MySQL database? If you don't have Enterprise licensing, you must use the internal database. But if you do, should you always use MySQL? No, in fact, you should usually use the internal database. Contrary to expectation, MySQL is slower than the internal database, so if it's performance you're after, go internal. MySQL is necessary if you're on a 32-bit system, and your database is too large to fit into a 32-bit address space (100GB+). It's also necessary if you need to run SQL queries against the collected data. But MySQL is slow, especially for building xref tables. So if you have 64-bit, or a small dataset, internal is usually best.

2008-06-05: Continuous disk defragmentation can interfere with Sawmill

We frequently see customers using Windows, whose antivirus software is mangling Sawmill's database, by modifying files there or by deleting them, or even just by pulling out of the way long enough that Sawmill can't find them when it needs them. Recently, we've seen a similar situation with the continuous defragmentation feature of Diskeeper. Like antivirus software, Diskeeper apparently grabs Sawmill's database files and makes them inaccessible to Sawmill while it's defragmenting them, resulting in errors ("file does not exists," "permission denied," and others) during database builds or report generation. If you're using Diskeeper or another automated defragmentation system, configure it to ignore Sawmill's installation directory (e.g., C:\Program Files\Sawmill 7), to avoid these errors. This affects only Windows installations of Sawmill.

2008-06-04: Merging databases

A customer asked how they could "archive" old data, when keeping a 12-month database. One possibility is to just keep the log data forever (always a good idea anyway, and relatively small if compressed); if the old statistics are needed, the log data can then be imported into a new profile. But for this dataset (over 1TB of log data), that could be hours or days of import time. A more sophisticated option is to keep a profile or database for each month of data, compress the one-month databases, and merge them together to create a complete history, when archived statistics are required. Merging databases is faster than re-processing the log data, so this would allow for quicker turnaround. The fastest solution is to simply keep a profile with all data in it. That will be a large and slow profile, but it can be queried immediately when required. For past-12-month queries, a separate profile with a separate database provides the quicker queries.

2008-06-03: Session Analysis for Flash Media Server Logs

Session Analysis has been added to the Flash Media Server log format plug-in and is available in Sawmill 7.2.15. We have been adding session analyis to media server plug-ins as they are requested. The reason for the request is most often an interest in the maximum number of concurrent connections to the server. This is tricky with most media server log formats, including Flash, because they don't have a session id or any other way to distinguish among simultaneous connections by the same user. The way around this, so far, has been to examine the disconnect event and use the session duration to determine when the session began, ignoring the actual connect event even if it is available.

2008-06-02: Train the Trainer series

We are now offering training in Sawmill! We have a new Train the Trainer series going on, for experienced Sawmill customers. If you are interested, please send us an email, with your specific experience administering and using Sawmill. You will be certified in using Sawmill, and we will work with you in setting up local trainings with our customers.

2008-05-30: Train the Trainer series

2008-05-30: Sawmill Trainer

We would like to introduce Derrick Freeman, our Sawmill trainer, located in Florida. Derrick's expertise is with streaming video and he uses Sawmill to analyze his log files. Derrick will be running Sawmill 2 day courses in June. Please stay posted for the dates. Derrick can also come to your business to help you set up or learn more about Sawmill. Derrick's company is geniusdv.com.

2008-05-29: Instability and poor performance due to antivirus scanning

We regularly see customers with error or performance problems due to conflicts between Sawmill and anti-virus programs. Active-scanning anti-virus programs keep hold of Sawmill's database files, cause errors when Sawmill tries to write to them; sometimes they even change or delete the files, corrupting the database. The solution is to disable scanning of Sawmill's installation folder (C:\Program Files\Sawmill 7). It is also worth noting that only Windows has this problem, so using Linux, MacOS, or some other non-Windows operating system eliminates the anti-virus problem entirely.

2008-05-28: Fixing database bloat with "maximum main table segment size"

We saw another customer today who was seeing a huge database from a not-so-huge dataset. Something like 200GB of database, from 50GB of log data (uncompressed). This happens with the internal database when the main table segment size (in Database Tuning) is set to too small a value. The default value (100MB) is too small for really large datasets, so if your dataset is more than 10GB or so, you should set this value to a higher value, like 2GB (or the size of physical RAM, if you have less than 2GB). That will keep the database closer to the expected size (which is about the size of the uncompressed log data). If this value is set too low, the database will become exponentially larger as the log data gets larger, due to an inefficiency in the way Sawmill 7 represents indices. This inefficiency will be fixed in Sawmill 8, but for now, make sure that value is set high. However, don't set it above 2GB if you're using Sawmill 7.2.14 or earlier on 64-bit Windows, because there's a bug which causes problems when Sawmill deals with files over 2GB, in some cases. So on 64-bit Windows with 7.2.14, set it to 2GB; with later versions, you can set it higher, if you have that much RAM.

2008-05-22: Sawmill 8 Status

Sawmill 8 is nearing alpha stage (i.e., almost all features are complete now). After a period of internal alpha testing, we will reach beta stage (all crashing and major bugs fixed). At that point, we will begin to seed our beta program. If you are interested in becoming a beta tester, please contact beta@sawmill.net and we can give you seeding instructions. Major features in 8.0 include support for Microsoft SQL Server and Oracle, as back-end databases, major improvements to the web interface, real-time log analysis, and role-based authentication.