Souvik Bhattacharyya - Noise Filtering for Big Data Analytics

Signal as a noise confined to small scales or high frequency can be removed preferentially from noisy signals if some prior information is available. According to the uncertainty principle, the energy dispersion of a function and its Fourier Transform (FT) cannot be made simultaneously small. Filters in the FT domain applied for signal processing involve a tradeoff between signal-to-noise ratio (SNR) and spatial resolution of the signal processed. Waveforms with a minimal dispersion in a time-frequency surface can be decomposed to retrieve the information about time and frequency. A windowing technique for analyzing non-stationary signals was developed by French geophysicist J. Morlet in the late 1970s to overcome the fixed windowing technique of the short-time Fourier transform (STFT). In lieu of assigning equal width pulses, shorter waveforms at high frequencies are essentially incurred by scaling a single function known as wavelet. An abstraction about conjugated quadrature filters that make a new wavelet transform known as continuous wavelet transform (CWT) was adopted in order to research multiscale image processing in the field of mathematics and computer world. A wavelet transform (WT) can decompose a signal into various spectral components in a time-frequency plane much like STFT, except it has different time-frequency resolution. The digitally implementable counterpart of CWT known as discrete wavelet transform (DWT) provides an octave-band filter bank structure. It is a computationally efficient arrangement of compactly supported orthonormal sequences of a single function localized in both position and scale. Therefore, DWT and its extension packet versionbased noise elimination or suppression exploits the proficiency of WT in denoising largely by providing sparse representations of data. Sparsity entails that most of the smaller value wavelet coefficients will be discarded as noise to yield a noise-free reconstruction and larger value data contains the true signal information. In this wavelet-based shrinkage technique, a noise contaminated signal can be decomposed by operating appropriate orthogonal mother filters of a multiresolution filter bank and soft thresholding is performed to shrink the retained decomposed data. It implies that from each coefficient, a constant value is subtracted. Therefore, this approach has been applied to eliminate noise effectively without affecting the useful information correlated with the high-frequency noise. Also, the coefficients with smaller amplitude than threshold are set to zero but the remaining coefficients are then shrunk to reduce the noise effect presumed to corrupt all the wavelet coefficients. This shrinkage-based signal estimation technique with proper threshold value on the wavelet coefficients would significantly minimize noise.

The selection of proper wavelet and optimal levels of decomposition in signal denoising are important criteria. Denoising performance-based selection of mother filter is time-consuming and cumbersome. Therefore, an alternative algorithm to select optimal wavelet, minimum description length (MDL) method provides the advantages of Bayesian information criteria incorporating the dependence structure within the cluster with penalty function. In this method, initially, model encoding method is applied for data encoding and finally, model is used to encode it. Then the minimized model from the total two-stage description length (DL) of the data should be selected within a few specific model classes. A model data that optimizes the trade-off between model complexity and goodness-of-fit can be selected in this way. The fundamental presumption of this approach is that the cost function reaches a minimum indicating the optimal number of coefficients. The entropy method furnishes a compact mathematical form for calculating the expected uncertainty associated with coefficients decomposed by wavelet transform to select optimal levels.

In this chapter, MDL method is implemented to select optimal mother filter. Shannons entropy is contemplated here to select the proper levels of decomposition in discrete wavelet transform and best basis selection technique for discrete wavelet packet transform. Initially, the formulation of the wavelet-based multiresolution filtering is applied for one-dimensional noisy signal obtained from lab-scale liquid-level plant as well as analogous real-time plant and this algorithm is developed in MATLAB level 2s function. The denoised response is then compared to the conventional filtering technique. It is seen that wavelet filters perform well in terms of improving signal to noise ratio (SNR) and reducing mean squared error (MSE) than traditional technique.