Metrics4j

Fine Grained Statistics for JDO

Copyright (c) 2009. Step Ahead Software Pty Ltd. All rights reserved.
This software is released under the Apache 2 license and can be
used in both free and commerical applications.

Download metrics4j source and binary

What is it?

The Metrics4j framework makes it easy to construct fine grained 'metrics accumulating' objects for an application that uses JDO for persistence. Metrics can be gathered on all persistent POJO domain model objects without changing their code. Metrics4j produces concise, summarized reports (dumps) of the persitent object instances that have been loaded or created for a paricular PM for a particular operation. This is helpful in understanding your application's runtime characteristics and can help pinpoint bottlenecks or operations that cause excessive memory usage and memory leaks.

This lightweight and simple to use framework consists of only a few classes and only one is needed to perform metrics gathering and reporting.

Why is Metrics4j helpful?

To help you judge the helpfulness of Metics4j consider the options you have for tracking instance loading/creation.

1. MBeans - gives "whole of JVM" instance summary - hard to track changes relating to a single PM or operation.
2. JVisualVM, jhat - analysis of heap dumps of the JVM tomcat is running in. Again it's hard to track changes relating to a single PM or operation and if you've ever tried to perform a heap dump of a live, production server you'll know how much of a bad idea that is.
3. Turn on persistence logging. When you switch on logging for your particular JDO imlpementation it can, depending on how you set it up, output many detailed lines per instance loaded or created. If you have hundreds of objects being loaded or created this can result in thousands
of lines of log output flashing past your eyes at 1000 miles/hour. In addition to this, logging is indiscriminate, outputing logging for every instance in every PM in the application regardless of where it is instantiated.

Compare this to the sample output from Metrics4j below:

-----------------------------------

Metrics4j - Object Instances snapshot:

Instances: 40 Loaded: 40 Created: 0 Class: class com.sas.av.model.product.Dvd

Instances: 23 Loaded: 23 Created: 0 Class: class com.sas.av.model.product.BlueRay

Instances: 1 Loaded: 1 Created: 0 Class: class com.sas.av.model.product.XboxGame

Instances: 1 Loaded: 1 Created: 0 Class: class com.sas.av.model.employee.Employee

Instances: 2 Loaded: 1 Created: 1 Class: class com.sas.av.model.customer.Customer

Total: 67 Loaded: 66 Created: 1

----------------------------------

This is a sample of a Metrics4j instance dump. It shows the number of instances loaded, created and a total (loaded+created) for each instance type, ordered by total (largest first) which makes it very easy to see if you have any excessive object instatiations.

As you can imagine the above is much easier to interpret and analyse than the other techniques mentioned and because it is a summary it is orders of magnitude smaller than the logging that would have been output to represent the instance loading and creation that has occured.

How does it work? Do I have to add code to my POJOs?

You can insert a metrics accumulating object into a persistence manager in a very fine grained way to help pin point a memory problem or excessive object loading or creation and you can dump the metrics at any time. There is no need to change any of your domain model POJOs. JDO 2 has a clever LifecycleListener interface that allows Metrics4j to hook into JDO's internals and get notified of object instance loading and creation.

What sort of applications can it be used in?

Metrics4j can be used in both client, server and web applications.

How to I add it to my web application?

Typical usage in a web application context would be to instantiate and place a MetricsTracker object in a persistence manager at the start of a HTTP request and perform a snapshot dump at the end of HTTP request processing. This is easily done in the doFilter method of any filter that you have set up. If you use any kind of dependency injection framework or 'open session per persistence manager' framework then that framework should already have a filter that will allow you to insert the MetricsTracker and perform a dump() at the end of HTTP processing.

The snapshot dump will provide a list of the objects either created or loaded during the processing of that HTTP request - this can be *very* revealing as we discovered the first time we used it.
It has helped us pin point a couple of flaws in some designs that caused execessive numbers of objects to be loaded which would eventually lead, over time to memory depletion and excessive garbage collection activity.

What are the internal details?

Metrics4j is very simple and works by a process of metrics accumulation. You can see all the details in the open source but here's an overall summary.

There are two classes that can be used for metrics accumulation:

• MetricsAccumulator

MetricsAccumulator is a low level, raw accumulator, which is the implementation of JDO's LifecycleListener interfaces. This can be used on its own to perform a wide variety of metrics gathering tasks although, for simple metrics snapshots, we recommend that you use the convenience class, MetricsTracker.

• MetricsTracker

MetricsTracker is a higher level, convenience class that uses a MetricsAccumulator to perform its metrics accumulation for the purpose of creating a dump of a metrics snaphot. An application simply instantiates a MetricsTracker passing in the PM that it should listen to and then call dump() whenever a dump of the instance snapshot is required.

If the application is only interested in how many persistent instances were created or loaded since the last dump then it simply calls reset() after each call to dump() and the next dump will be effectively a 'delta' summary since the last dump.

There are two classes available for tailoring the output to your specific needs. Tailoring output is optional: you can use metrics4j without knowing about these two classes.

• MetricsWriter

Metrics4j provides an interface called MetricsWriter that abstracts the writing of dump data. There is no need to use this interface directly: by default MetricsTracker writes to standard output but MetricsTracker implements MetricsWriter so to output to your favourite logger simple extend MetricsTracker and override writeLine(String line) to write dump lines using log4j or your logging framework of choice.

• MetricsFormatter

Provides ability to customize the header/footer lines of a metrics dump. There is no need to override this interface if you are happy with the default metrics dump format.