See the I/O You Caused by Getting Your Diagnostic Tracing into Process Monitor

Recently I was working on a performance bug for a client where our hypothesis was that there was too much file I/O going on in the process, which was causing unnecessary page faults. As this application is a kiosk application running on limited hardware with limited memory we needed to know exactly what the I/O patterns were especially since there were third party numerous components in the product we didn’t control. There’s only one tool for watching I/O and that’s the amazing Process Monitor from Sysinternals.

To help isolate down which actions caused what I/O we added tracing to the application to report when operations started and ended so we could see what operations were causing the most I/O. To get both the I/O stream and your tracing you have to run both Process Monitor and another Sysinternals tool, Debug View. We were able to narrow down the problem and recode some operations to eliminate the unnecessary I/O. However, having I/O in one program and tracing in another, it was very tedious to sit there and match up traces with the start of I/O in Process Monitor.

What I really wanted was for my trace statements to be part of the Process Monitor viewing so that way it would be trivial mapping the I/O activity to operations in my code. Fortunately, I have a personal developer at my disposal that is keen to tackle these kinds of challenges. He’s a very nice guy named Mark Russinovich who happens to be the author of Process Monitor. Mark is always eager to hear feature requests for his tools and I think he’s implemented at least 30 features in Sysinternals tools over the years that I thought would be great to have. Don’t hesitate to email Mark with feature ideas so he can be your personal developer as well.

Mark and I bounced a couple of emails back and forth discussing how the trace statements should get into Process Monitor. Originally, I thought merging the functionality of Debug View and Process Monitor would be the way to go but Mark thought adding the OutputDebugString capturing to Process Monitor would add lots of extraneous lines of output that would get in most people’s way. We settled on the idea that getting your tracing into Process Monitor should be something a developer specifically opts into by calling an API.

You can download the API I put together for both native and managed code to get your trace statements into Process Monitor here. For native code, you’ll need to include the header file ProcMonDebugOutput.h and link against ProcMonDebugOutputx86.lib or ProcMonDebugOutputx64.lib as appropriate. The API you’ll call is, appropriately named, ProcMonDebugOutput which takes a single parameter of a UNICODE string. Obviously, you’ll need to add ProcMonDebugOutputx86.DLL or ProcMonDebugOutputx64.DLL as part of your distribution.

For managed code, the API is wrapped up in a TraceListener derived class, ProcessMonitorTraceListener, in Sysinternals.Debug.DLL. That means you can add ProcessMonitorTraceListener through configuration files like any TraceListener you’ve ever used. With your application you’ll need to include Sysinternals.Debug.DLL as well as both ProcMonDebugOutputx86.DLL and ProcMonDebugOutputx64.DLL. The ProcessMonitorTraceListener works with both 32-bit and 64-bit code and calls the appropriate native DLL as necessary.

Once you’ve added calls to push your tracing to Process Monitor, you’ll see the trace output as part of the Profiling Events so make sure that the red circled button below is enabled to see them. In the screen shot, any line where the operation is “Debug Output Profiling” is one of your trace statements going through the API.

The new tracing API in Process Monitor will be a huge boon to developers trying to get a real picture of registry, file, and network I/O going on with individual operations. I’ve had the Process Monitor build that supports the API for a couple of weeks and it’s been fascinating to watch what really happens under the hood when certain operations happen in your code. I’m now much more cognizant of my I/O operations than ever before. Half the battle debugging is just seeing what’s going on and how you caused it. Process Monitor’s new API will make your debugging considerably easier!