## Beyond 2'nd order statistics; higher order statistics

### Beyond 2'nd order statistics; higher order statistics

I have been trying to understand and classify how much
information is lost when we model signals using Gaussian models
and throw away higher order statistical information content.
Clearly this is fine when signals are Gaussian, but if
when signals are not Gaussian.

Example : if I want to estimate a variable x (Multi-dimensional)
from observations of a related signal y (Multi-dimensional),
assuming the signals are Gaussian best linear estimation of x
is x_estimate=Rxy * Inverse(Ry) * y   where Ry=E(xx*) is Cov Matrix of
y and similarly Rxy=E(xy*). The estimate of x is optimal in the
least square sense; but if we estimate x using higher order
statistical information content (cumulants beyond 2nd order) then
estimate is not optimal in the least square sense, yet it is
more consistent with the statistical properties of the signals.

When should be use higher order statistics and when does it actually
make a difference and how much beyond the usual 2n'd order estimation
techniques?

Any references would be greatly appreciated.

Ben

Ben

### Beyond 2'nd order statistics; higher order statistics

Quote:> I have been trying to understand and classify how much
> information is lost when we model signals using Gaussian models
> and throw away higher order statistical information content.
> Clearly this is fine when signals are Gaussian, but if
> when signals are not Gaussian.

> Example : if I want to estimate a variable x (Multi-dimensional)
> from observations of a related signal y (Multi-dimensional),
> assuming the signals are Gaussian best linear estimation of x
> is x_estimate=Rxy * Inverse(Ry) * y   where Ry=E(xx*) is Cov Matrix of
> y and similarly Rxy=E(xy*). The estimate of x is optimal in the
> least square sense; but if we estimate x using higher order
> statistical information content (cumulants beyond 2nd order) then
> estimate is not optimal in the least square sense, yet it is
> more consistent with the statistical properties of the signals.

> When should be use higher order statistics and when does it actually
> make a difference and how much beyond the usual 2n'd order estimation
> techniques?

regarding dither and audio and we found out that rectangular dither (of size
1 LSB of the ideal quantizer) totally decorrelated the mean of the error
from the input signal and that triangular dither (or size 2 LSBs) totally
decorrelated the mean *and* the variance (i.e. the first 2 moments) of the
error from the input signal (but not the higher moments).  it was determined
with reasonably controlled listening tests (i think Stanley Lipshitz did
this and he's a stickler for *double* blind testing) that nobody heard the
higher moments.

Quote:> Any references would be greatly appreciated.

that will be painful to find but i'll try.

r b-j

### Beyond 2'nd order statistics; higher order statistics

> > I have been trying to understand and classify how much
> > information is lost when we model signals using Gaussian models
> > and throw away higher order statistical information content.
> > Clearly this is fine when signals are Gaussian, but if
> > when signals are not Gaussian.

> > Example : if I want to estimate a variable x (Multi-dimensional)
> > from observations of a related signal y (Multi-dimensional),
> > assuming the signals are Gaussian best linear estimation of x
> > is x_estimate=Rxy * Inverse(Ry) * y   where Ry=E(xx*) is Cov Matrix of
> > y and similarly Rxy=E(xy*). The estimate of x is optimal in the
> > least square sense; but if we estimate x using higher order
> > statistical information content (cumulants beyond 2nd order) then
> > estimate is not optimal in the least square sense, yet it is
> > more consistent with the statistical properties of the signals.

> > When should be use higher order statistics and when does it actually
> > make a difference and how much beyond the usual 2n'd order estimation
> > techniques?

As an example, I would say that training based signal or pilot based
signal
(as we often see in GSM/EDGE mobile system) can estimate CIR or learn
the signal
using 2nd order statistics like correlation/covariance properties. If
the signal is not fed with such training sequences or pilot symbols it
has to depend heavily on the obderved signal statistics i.e it has to
exploit higher order cumulants for better estimation.

I guess if we need to study phase properties of such signals we can
not use 2nd order statistics since it is phase blind.

Look much more excellent information in the following paper:

J M Mendel Tutorial on Higher Order Statistics (Spectra) in signal
processing and system theory: theoretical results and some
applications Proceedings of the IEEE, 79(3), pp 278-305, March, 1991.

Regards,
Santosh

- Show quoted text -

Quote:

> regarding dither and audio and we found out that rectangular dither (of size
> 1 LSB of the ideal quantizer) totally decorrelated the mean of the error
> from the input signal and that triangular dither (or size 2 LSBs) totally
> decorrelated the mean *and* the variance (i.e. the first 2 moments) of the
> error from the input signal (but not the higher moments).  it was determined
> with reasonably controlled listening tests (i think Stanley Lipshitz did
> this and he's a stickler for *double* blind testing) that nobody heard the
> higher moments.

> > Any references would be greatly appreciated.

> that will be painful to find but i'll try.

> r b-j

### Beyond 2'nd order statistics; higher order statistics

> Look much more excellent information in the following paper:

> J M Mendel Tutorial on Higher Order Statistics (Spectra) in signal
> processing and system theory: theoretical results and some
> applications Proceedings of the IEEE, 79(3), pp 278-305, March, 1991.

> Regards,
> Santosh

Thanks, Santosh, for providing this link. I'll certainly check the article
out. Unfortunately, I must wait till after I have figured out why I can't

Rune

### Beyond 2'nd order statistics; higher order statistics

Just thought I would throw in my 2 cents...

According to Tong, they proved that the channel is identifiable from the
channel output 2nd order statistics if the channel is nonzero for all z.
Have a look at:

Tong, Xu and Kailath, "Blind Identification and Equalization Based on
Second-Order Statistics: A Time Domain Approach", IEEE Trans.
on Info Theory, March 1994.

Moulines has also shown the channel can be identified using a subspace
method similar to MUSIC:

Mouline and et al., "Subspace methods for the blind identification of
multichannel FIR filters", IEEE Trans on Signal Processing, Feb. 1995.

cf

> > Look much more excellent information in the following paper:

> > J M Mendel Tutorial on Higher Order Statistics (Spectra) in signal
> > processing and system theory: theoretical results and some
> > applications Proceedings of the IEEE, 79(3), pp 278-305, March, 1991.

> > Regards,
> > Santosh

> Thanks, Santosh, for providing this link. I'll certainly check the article
> out. Unfortunately, I must wait till after I have figured out why I can't

> Rune

### Beyond 2'nd order statistics; higher order statistics

> Just thought I would throw in my 2 cents...

> According to Tong, they proved that the channel is identifiable from the
> channel output 2nd order statistics if the channel is nonzero for all z.
> Have a look at:

> Tong, Xu and Kailath, "Blind Identification and Equalization Based on
> Second-Order Statistics: A Time Domain Approach", IEEE Trans.
> on Info Theory, March 1994.

> Moulines has also shown the channel can be identified using a subspace
> method similar to MUSIC:

> Mouline and et al., "Subspace methods for the blind identification of
> multichannel FIR filters", IEEE Trans on Signal Processing, Feb. 1995.

While I subscribe (or at least intended to subscribe) to Proc. IEEE
through ieeexplore, I don't have access to Trans. Inf. Theory. The abstract
of the Tong et al. paper says the method is based on "cyclostationary"
signals. Lots of papers on blind source and/or channel estimation are
based on the signal being "cyclostationary".

What does "cyclostationary" mean?

Rune

- Show quoted text -

> > > Look much more excellent information in the following paper:

> > > J M Mendel Tutorial on Higher Order Statistics (Spectra) in signal
> > > processing and system theory: theoretical results and some
> > > applications Proceedings of the IEEE, 79(3), pp 278-305, March, 1991.

> > > Regards,
> > > Santosh

> > Thanks, Santosh, for providing this link. I'll certainly check the article
> > out. Unfortunately, I must wait till after I have figured out why I can't

> > Rune

### Beyond 2'nd order statistics; higher order statistics

> I have been trying to understand and classify how much
> information is lost when we model signals using Gaussian models
> and throw away higher order statistical information content.
> Clearly this is fine when signals are Gaussian, but if
> when signals are not Gaussian.

> Example : if I want to estimate a variable x (Multi-dimensional)
> from observations of a related signal y (Multi-dimensional),
> assuming the signals are Gaussian best linear estimation of x
> is x_estimate=Rxy * Inverse(Ry) * y   where Ry=E(xx*) is Cov Matrix of
> y and similarly Rxy=E(xy*). The estimate of x is optimal in the
> least square sense; but if we estimate x using higher order
> statistical information content (cumulants beyond 2nd order) then
> estimate is not optimal in the least square sense, yet it is
> more consistent with the statistical properties of the signals.

> When should be use higher order statistics and when does it actually
> make a difference and how much beyond the usual 2n'd order estimation
> techniques?

> Any references would be greatly appreciated.

> Ben

> Ben

I'm not  a big fan of higher order statistics.  A few years ago I was
looking at approximating a pdf.
I was looking at the output pdf of a 3 dimensional, correlated , Gaussian
pdf, fed into a memoryless nonlinearity .    The characteristic functions
were easy to calculate, so the moment generating function was
relatively easy to get.  The thing that surprised me was the number of
moments that were necessary to recover the pdf.  I ended up abandoning
the moment approach.  One needs to be very cautious with moments because
they don't really work like a Taylor series. The specific problem was
similar to the difficulty of approximating  an "AR" polynomial with an
"MA" polynomial.  This was essentially an analytic exercise so issues
related to consistent estimators vs sample size of higher order momements
(or cumulents) didn't enter the problem.  If higher order moments were
that informative, a lot more folks would be doing Gram-Challier  kinds of
expansions.

### Beyond 2'nd order statistics; higher order statistics

>  This was essentially an analytic exercise so issues
> related to consistent estimators vs sample size of higher order momements
> (or cumulents) didn't enter the problem.  If higher order moments were
> that informative, a lot more folks would be doing Gram-Challier  kinds of
> expansions.

I have noted that higher order statistics isn't very popular. To the little
extent I have read about it, higher order statistics seems to be presented
as quite useful. However, the main reason I am aware of cumulant analysis
(and also an important reason why I stay away from it) is the paper

Dwyer: "Spectra and fourth order cumulant spectra from broadband beamformed
data", Journ, Ac. Soc. Am., v 102, p 1696, 1997.

The appendix of that paper is two and a half pages long and contains one
- 1 - sole equation...

Rune

### Beyond 2'nd order statistics; higher order statistics

> >  This was essentially an analytic exercise so issues
> > related to consistent estimators vs sample size of higher order momements
> > (or cumulents) didn't enter the problem.  If higher order moments were
> > that informative, a lot more folks would be doing Gram-Challier  kinds of
> > expansions.

> I have noted that higher order statistics isn't very popular. To the little
> extent I have read about it, higher order statistics seems to be presented
> as quite useful. However, the main reason I am aware of cumulant analysis
> (and also an important reason why I stay away from it) is the paper

> Dwyer: "Spectra and fourth order cumulant spectra from broadband beamformed
>        data", Journ, Ac. Soc. Am., v 102, p 1696, 1997.

> The appendix of that paper is two and a half pages long and contains one
> - 1 - sole equation...

> Rune

Almost everything is presented as being useful.  I wish there were more authors who would
say that they really don't know if their algorithm was useful but nevertheless, they think
it is interesting and given a larger readership, a useful application may be found.

### Beyond 2'nd order statistics; higher order statistics

> > Just thought I would throw in my 2 cents...

> > According to Tong, they proved that the channel is identifiable from the
> > channel output 2nd order statistics if the channel is nonzero for all z.
> > Have a look at:

> > Tong, Xu and Kailath, "Blind Identification and Equalization Based on
> > Second-Order Statistics: A Time Domain Approach", IEEE Trans.
> > on Info Theory, March 1994.

> > Moulines has also shown the channel can be identified using a subspace
> > method similar to MUSIC:

> > Mouline and et al., "Subspace methods for the blind identification of
> > multichannel FIR filters", IEEE Trans on Signal Processing, Feb. 1995.

> While I subscribe (or at least intended to subscribe) to Proc. IEEE
> through ieeexplore, I don't have access to Trans. Inf. Theory. The abstract
> of the Tong et al. paper says the method is based on "cyclostationary"
> signals. Lots of papers on blind source and/or channel estimation are
> based on the signal being "cyclostationary".

> What does "cyclostationary" mean?

> Rune

Hi Rune,

"Cyclostaionary" is one  class of blind channel identification method.
"Cyclostationary blind equalisation methods exploit the fact that,
sampling the received signal at a rate higher than the transmitted
signal symbol rate, the received signal becomes cyclostationary. In
general, cyclostationary blind equalisers can identify a channel with
less data than higher-order statistics (HOS) methods"

Among  HOS, most famous is CMA approach by Godard(may be spellig
mistake!),1980.

For a better mathematical coverage of all these algorithms you can
look into

Regards,
Santosh

- Show quoted text -

> > > > Look much more excellent information in the following paper:

> > > > J M Mendel Tutorial on Higher Order Statistics (Spectra) in signal
> > > > processing and system theory: theoretical results and some
> > > > applications Proceedings of the IEEE, 79(3), pp 278-305, March, 1991.

> > > > Regards,
> > > > Santosh

> > > Thanks, Santosh, for providing this link. I'll certainly check the article
> > > out. Unfortunately, I must wait till after I have figured out why I can't

> > > Rune

### Beyond 2'nd order statistics; higher order statistics

> > What does "cyclostationary" mean?

> > Rune

> Hi Rune,

> "Cyclostaionary" is one  class of blind channel identification method.
> "Cyclostationary blind equalisation methods exploit the fact that,
> sampling the received signal at a rate higher than the transmitted
> signal symbol rate, the received signal becomes cyclostationary. In
> general, cyclostationary blind equalisers can identify a channel with
> less data than higher-order statistics (HOS) methods"

Thanks Santosh.

From what you say above, I assume "cyclostationarity" is some property
that somehow is interconnected with transmission systems, code books
and modulation schemes. Which means that these kinds of algorithms
don't suit my types of data...

Quote:> Among  HOS, most famous is CMA approach by Godard(may be spellig
> mistake!),1980.

> For a better mathematical coverage of all these algorithms you can
> look into
> Simon Haykin's "Adaptive filter theory".

I think I'll do that. Now, where did I put my copy...

Rune

### Beyond 2'nd order statistics; higher order statistics

...

Quote:> Thanks Santosh.

> From what you say above, I assume "cyclostationarity" is some property
> that somehow is interconnected with transmission systems, code books
> and modulation schemes. Which means that these kinds of algorithms
> don't suit my types of data...

A cyclostationary process is one which, while not strictly stationary,
can be treated as such because of some repetitive aspect.
http://www.mai.liu.se/~tikos/Wold1.pdf might shed a bit more light.

Jerry
--
Engineering is the art of making what you want from things you can get.

### Beyond 2'nd order statistics; higher order statistics

Rune,

using HOS for estimation, trying looking at the book by Nandi:

"Blind Estimation using Higher-Order Statistics", Kluwer, 1999.

hth,

cf

> > > What does "cyclostationary" mean?

> > > Rune

> > Hi Rune,

> > "Cyclostaionary" is one  class of blind channel identification method.
> > "Cyclostationary blind equalisation methods exploit the fact that,
> > sampling the received signal at a rate higher than the transmitted
> > signal symbol rate, the received signal becomes cyclostationary. In
> > general, cyclostationary blind equalisers can identify a channel with
> > less data than higher-order statistics (HOS) methods"

> Thanks Santosh.

> From what you say above, I assume "cyclostationarity" is some property
> that somehow is interconnected with transmission systems, code books
> and modulation schemes. Which means that these kinds of algorithms
> don't suit my types of data...

> > Among  HOS, most famous is CMA approach by Godard(may be spellig
> > mistake!),1980.

> > For a better mathematical coverage of all these algorithms you can
> > look into
> > Simon Haykin's "Adaptive filter theory".

> I think I'll do that. Now, where did I put my copy...

> Rune

Hi,

I am enquiring as to whether there are Higher Order Statistics toolboxes for
MATLAB, available on the web, which anybody recommends.

Thanks,
Luca