Help designing very low pass filters (sub-mHz) for long time series

I am relatively new to filter design so patience is appreciated.

I have an extremely long time series (e.g. ~100 million data points) with a sample rate of 0.2 Hz and I want to apply a series of lowpass filters to the signal at query frequencies of e.g. 1/10 Hz, 1/100 Hz, 1/1000 Hz, 1/10000 Hz, and so on. However, the filtered datasets look roughly identical regardless of what lowpass filter I use.

Fig 1: Zoom view of section of time series to show small differences in filtered time series.

The spectrum of the signal suggests that there is plenty of power at the lower frequencies, so I would think that the results should be different depending on which lowpass cutoff I use.

Fig 2: Spectrum of signal

I wonder if it has something to do with trying to filter sub-mHz levels. Most of the filter questions/answers that I have found are dealing with audio sampling and much higher frequencies. Are there some defaults in MATLAB which design filters to be better behaved at higher frequencies and these assumptions break down at lower frequencies?

I have not done filter design before. All my previous filtering experiences has just used plug-and-play MATLAB functions like lowpass, highpass, bandpass. How would I go about designing an adequate filter for these low frequency signals?

Alternatively, is there nothing wrong with the result and my intuition about what the filter results should look like is just wrong?

Any help is appreciated.

Answer 1

[Umar]

Hi Darcy,

You sound like a pro and impressive job with filter design. Patience is the key when you are trying to learn something new. Don’t give up, keep learning from the experts at Mathworks such as image analyst, Paul, Voss, Torsten, Rik, and especially Walter Robertson, they are all great contributors and excellent coaches. Kudos to their efforts and helping us grow in this community. I would like to win Mathworks T-shirt and you can participate if they draw raffle ticket. Hope they recognize efforts of people that I mentioned by giving them matlab contributor of the year and you as well.

Based on the information provided, it seems that you are facing challenges in designing very low pass filters (sub-mHz) for long time series data in MATLAB. Your goal is to apply a series of lowpass filters at specific query frequencies but are finding that the filtered datasets look roughly identical regardless of the filter used.

In your case, where you are dealing with an extremely long time series (~100 million data points) and low frequencies such as 1/10 Hz, 1/100 Hz, 1/1000 Hz, and so on, it is essential to consider the limitations and characteristics of digital filters at such low frequencies.

The phenomenon you are experiencing where the filtered datasets look similar could be attributed to various factors. One possible explanation could be related to the resolution and precision of your data at these extremely low frequencies. The numerical precision of the filter coefficients and calculations in MATLAB may impact the differentiation between filter outputs at sub-mHz levels.

Moreover, designing filters for such low frequencies requires careful consideration of filter specifications, such as filter order, transition bandwidth, and passband ripple. MATLAB does provide default parameters for filter design functions that are optimized for typical audio sampling frequencies, which may not be suitable for your specific application at sub-mHz levels.

To address this issue and design adequate filters for low frequency signals, you can consider customizing the filter design process in MATLAB using tools like 'fdesign' or 'designfilt' functions.

For more information regarding designfilt, please refer to

https://www.mathworks.com/help/signal/ref/designfilt.html

Note: it is not compatible with R2021b.

For more information on fdesign, please refer to

https://www.mathworks.com/help/dsp/ref/fdesign.html

By specifying the desired filter characteristics accurately, such as passband frequency, stopband frequency, and attenuation requirements, you can tailor the filter design to suit your specific needs at sub-mHz frequencies.

Additionally, it would be beneficial to analyze the frequency response of your designed filters and compare them to understand how they affect your signal in the frequency domain. Visualizing the spectrum of the filtered signals can provide insights into how different filters shape the frequency content of your data.

Good luck with your future endeavors. Please let me know if I can provide further assistance.

Answer 2

[Umar]

Hi Darcy,

I do highly appreciate your recent comment. Instead of appreciating my contribution towards resolving your problem, you basically insulted me. However, I would like to express my knowledge by sharing an example code with you to further enhance your knowledge about functions mentioned above by providing my code snippet.

>> % Designing a custom low-pass filter for low-frequency signals fs = 1000; % Sampling frequency fc = 100; % Cutoff frequency for low-pass filter order = 4; % Filter order

% Design the filter using 'fdesign' function d = fdesign.lowpass('N,F3dB', order, fc, fs); Hd = design(d, 'butter');

% Visualize the frequency response of the designed filter freqz(Hd); title('Frequency Response of Low-Pass Filter for Low-Frequency Signals');

Please see attached plot along with code snippet.

The purpose of this code snippet is to showcase the process of designing a custom low-pass filter in MATLAB for attenuating high-frequency components in signals, particularly suited for low-frequency applications. By setting the sampling frequency, cutoff frequency, and filter order, the code tailors the filter characteristics to meet specific signal processing requirements.

The visualization of the frequency response using freqz aids in understanding how the designed filter affects different frequency components. It will allows you to analyze the filter's behavior, such as passband ripple, stopband attenuation, and phase response, crucial for signal processing tasks.

Please let me know if you need further assistance or help.