Smoothing Data

Yesterday, someone contacted me and asked me how to quickly smooth a bunch of data measured over time. While I did this many times in various projects before, I discovered just now, that I never published a sample program which shows how to perform smoothing of data using the SDL Component Suite. So here we are...

This small program allows to create a time series simulating the measured data. The data may be "poisoned" by arbitrary levels of (Gaussian) noise and up to 10 spikes. The resulting data series is typical for measured data, which usually contains some amount of noise, and occasionally one or two spikes (e.g. when a refrigerator in the vicinity of the measurement device switches off).

Experimenting with this little program quickly shows that normally distributed noise can be easily reduced by most of the smoothing algorithms, provided that the window width of the smoothing algorithm is large enough to cover a sufficient number of random fluctuations, and, of course, small enough not to interfere with low-frequency signal components.

A big problem to all integrating filters are spikes. We know from system theory that the response of a filter to a dirac impulse resembles its own transfer function. Thus a moving average filter will generate rectangular shapes, a polynomial filter will generate parabolic artefacts. A very good solution for getting rid of the spikes (at least if they have a width of only a few data points) is to use a moving median filter.

The images above show the effect of smoothing a noisy signal containing a a spike.

If you want to experiment with these kinds of filters yourself, you can download the program here. The archive contains the executable as well as the Delphi XE4 sources, so you can immediately try out the various options. For adjusting the code to your own needs you have to install the SDL Component Suite (the Light Edition is free of charge).

Last but not least two remarks: you may be interested in experimenting with penalized splines, as well. They are basically not intended to automatically filter data, however, the resulting smoothed curves are free of noise and thus nice to look at. A sample program to experiment with penalized splines may be downloaded from the VIAS Web site. Another way (and a quite efficient one) to perform smoothing is to use fast Fourier transform to remove unwanted parts of the signal.