Review of Application Monitoring

Review of Application Monitoring

Here is an article by Andy McBride which explains how to track down the cause of sluggish application response as seen by the users.

The Time Honoured Argument:

Application Geek: "My code is fine, the network is slow – that’s the problem."

Network Supremo: "The network has plenty of bandwidth, something must be wrong with your application."

Stop the Madness

As you can see, the above dialogue could be classified as less than productive.  Both the network engineer and the application developer had some tools to diagnose what was happening in their area of expertise, but they each had little to no visibility into how the network and applications were working together.  The two departments were experiencing the effects of technology silos.  Something had to change in how we defined and measured applications.  In the mid to late 1990’s that change happened, and management vendors on both sides of the fence began to offer solutions that included application and network monitoring features.  These included:

• Hardware probes that sniff traffic and analyze what is on-the-wire.
• Synthetic transaction software that inserts and measures artificial transactions to model actual user transactions Distributed software agents to measure user transactions.
• Flow analyzers to discover what applications were consuming bandwidth, application vendor modules focused on managing their own application.
• Network management system (NMS)-based application monitoring, which combined application and management packages.

Server Processes/Service and Daemon Monitoring

Windows servers – By far the most common method of monitoring processes and services is Windows Management Instrumentation (WMI).  WMI is Microsoft’s implementation of a couple of standards: CIM and WBEM.  WMI offers monitoring of counters on a Windows system to track machine-level items like memory, processors, etc., and OS information like server work queues, processes, services and connection status.

Functionally, WMI is a lot like SNMP in its use of counters, polling and information bases.  The best way to get to know WMI is to give it a spin.  Download and try the free SolarWinds WMI Monitor tool.  Make sure to spend some time with the WMI browser included in the free tool.  It’s a good way to see firsthand the types of objects you can manage with WMI and how WMI operates.

Linux or Unix servers – These servers have been around for decades and have mature implementations of SNMP.  Like WMI, SNMP on these servers monitors machine-level and OS components.  Unlike WMI, SNMP is a completely open standard.  You can use a MIB browser and walk through a server’s MIB much the way you do in a WMI browser.  As an example, if we want to see that an FTP server is able to operate correctly we should monitor the FTP process on that server.  This can be accomplished by setting up a WMI monitor and monitoring the FTP process WMI counter on the FTP server.  We could also do this using an SNMP monitor for the FTP process.

Protocol Port Monitoring Protocols can be thought of as two-way radios: they allow bidirectional information flow between two or more points.  The protocol ports in this analogy are the radio frequencies.  They offer multiple channels of communication within dedicated spectra.  So TCP port 20 is like TCP channel 20.  If you were able to tune in and listen to port 20 you’d be hear computers delivering files, as TCP port 20 is reserved for File Transfer Protocol (FTP).  If you tuned to port 21 you’d hear computers asking for file transfers that will be sent on port 20.  These two ports work together to supply FTP control (port 21) and an FTP file carrier (port 20).  So if you just want to see if files are able to be transferred, you only need to setup a monitor for port 20 on the FTP server.  On the other hand, if you want to make sure the FTP network service is able to function, you would monitor both ports.  There’s actually a trick we can do to monitor port 21 directly and 20 indirectly.  We’ll see that below.

"10.1.1.1 calling 10.1.1.2, please ship me file1.txt on port 20"

"Hi 10.1.1.1, this is 10.1.1.2, I’ll meet you on port 20 with file1.txt"

This example is a bit more elaborate than actual protocol port monitoring.  In protocol port monitoring what we do is look to make sure the server is able to use the specified port rather than listening to the conversations on that port.  Examining the conversations is accomplished by flow analyzers and sniffers.  That level of monitoring is overkill for the type of application monitoring we are discussing here.

Now we can monitor ports and determine FTP is working port-wise, but that is only two components of the FTP network service.  There is also an FTP service running on the server OS that receives the requests and serves the files.  As we learned in the previous section we could monitor the FTP process using WMI or SNMP. 

Now that we have combined the FTP service monitoring with FTP port monitoring we are close to claiming we are monitoring the FTP application.  But what if the service and ports are operational and someone has deleted the target file to be served? While the components of FTP are operational, the FTP network service cannot complete user requests for the absent file.  We should test the service by requesting the file.

Guy Recommends SolarWinds’ Free Network Monitor

Thus utility makes it easy to check the health of a router or firewall.  Check the real-time performance, and availability statistics, for any device on your network.  Get started with an extensive collection of "out-of-the-box" monitors for popular network devices. Give Network Monitor a whirl – it’s free. Download your free Network Device Monitor

User Experience Monitoring

This component of application monitoring involves making an automated request of an application to model a real user request.  These are called synthetic transactions because they mimic a user transaction but don’t originate from a user.  Typically they are initiated from the NMS.  Let’s add a user experience monitor for FTP to the process and protocol monitors above. 

The FTP user monitor in this case will make an FTP request to the FTP server from the NMS.  The request can be for a file normally requested as part of a business transaction or a test file.  When the NMS makes the request for the file it can record the following:

• Time to complete transfer of the file.
• File integrity via checksum.

Here’s the trick I mentioned: If we monitor TCP port 21, we examine FTP requests.  Now by placing a user experience monitor on FTP, we are also testing TCP port 20 indirectly and monitoring the actual ability to serve a file.  After all, if TCP port 20 is not functioning the user experience transfer will not happen.  Let’s review how we’ve created a holistic application monitor using each of the component monitors above:

What application do we want to monitor?

• FTP of file1.txt from FTP server001. 

What components define this application instance?

• Service monitor – The FTP service on server001
• Protocol port monitor – TCP ports 20 and 21 on server001
• User experience monitor – File1.txt delivery

How will we measure each one of these?

Create an application monitor for FTP which includes:

• Status of the FTP service on server001 (WMI or SNMP service/ process monitor)
• Status of TCP ports 20 and 21 on server001 (protocol monitor)
• Delivery of file1.txt (user experience monitor)

Often these component monitors are grouped to make an application template.  This can help to save a great deal of time when there are multiple instances of an application to monitor.  Instead of having to create a new application monitor for FTP on server002, wouldn’t it be nice to apply the template created for server00 instead?

Simple and Complex

Applications

For our purposes, I’ll define these as:

• Simple application = an application that operates as a process, independently from other applications or processes.
• Complex application = an application that requires multiple processes and services and interacts with other applications to complete its functions.

This does not imply anything about size or ease of use of the applications, it just refers to application and process/service interaction.  In our example we created an application monitor for a simple application, FTP, which happens to be a network service.  Let’s take SolarWinds Orion Network Performance Monitor with the Orion NetFlow Traffic Analyzer module as an example of a complex application.  Here are some of the components that we could monitor to make a complete application monitor:

Orion NPM Server Services Monitors – WMI

• SNMP service, SNMP trap service, SolarWinds alerting engine, SolarWinds job engine, SolarWinds NetFlow service, SolarWinds information service, IIS, WWW publishing service. 
• Orion NPM server port monitors. 
• TCP port 80 – HTTP for web console.

User Experience Monitors

HTTP monitor of Orion home page.  This monitor will log in to the Orion home page automatically and test both authentication and delivery of the http home page. 

SQL monitor.  This monitor will automatically query the SQL server and return the results. 

That makes 10 individual monitors to monitor the application, and you could probably add a few more.  These monitors cover all three of the component areas we defined in Section 3 – service monitors, port monitors and user experience monitors.  Here is where application templates are crucial.  With 10 or more components being used to monitor an application, there is plenty of opportunity for mistakes if you had to create the application monitor from scratch each time.  If you apply a verified, saved application monitor template, the risk of applying monitors with mistakes is eliminated.  Another option would be to create an application monitor template for the SQL server and one for the DNS server that NetFlow uses to resolve endpoint IP addresses to domains.

See spotlight on SolarWinds SAM »

Summary of Application Monitoring

We have seen the following points in the above sections:

1) An application is not just a service or process running on a server. 

2) Applications depend on multiple resources to operate correctly, including:

• Services/processes – both the actual application services and associated services which support the application Server resources (RAM, CPU, hard drives, etc.)
• Protocol ports
• Secondary applications such as SQL in section 4

3) Applications can be effectively monitored by:

• Examining the application components and applying the proper monitors, including: WMI and SNMP component monitors for processes/services
• Port component monitors for protocol ports
• User experience monitors to demonstrate the application functionality.  I hope this has been a helpful overview. 

If you have any questions or comments, participate in our online community Thwack, made of over 58,000 users of SolarWinds products

Additional Free and Trial SolarWinds Network Software

Here are nifty gadgets which I have evaluating on my network.  Some are completely free, while other downloads are trial versions of the full product.  I think SolarWinds have a great strategy, namely, supplying a free utility, yet providing a big-brother suite of programs for larger organizations.

• SolarWinds Permissions Analyzer  • Review PC Engineers Toolset   • LanSurveyor  • Home

• SolarWinds Netflow   • Review SolarWinds TFTP Server  • Review of Network Topology Mapper

• Mobile IT Admin App   • Netflow Monitoring   • Free NetFlow Analyzer   • CSVDE Import Tool