## question about fixed-integer precision in digital signal processing?

### question about fixed-integer precision in digital signal processing?

Dear all,

I am studying 2D DCT for an image. The 2D DCT can be summarized as computing
Y=T*X*T', where X is the input matrix, T is the 2D DCT basis matrix, Y is
the output.

X is integer 0-255. T is real-valued, 0 - 1.

In order to convert real-valued T to integers, I multiply T with 256(left
shift), and round it to integer, and use these new integer values in my
computing.

Now Y will be larger than its correct value by a factor of 256*256, so I
will divide it by 65536(right shift the result by 16 bits to get correct
result).

My question is, is my approach correct?

How many bits I need to allocate for internal matrix computation and the
output?

Is there any better approach to save some datawidth?

In general, are there any resources to study for dealing with
real-to-integer conversion?

Thanks a lot,

-Walala

### question about fixed-integer precision in digital signal processing?

Quote:> Y=T*X*T'

> X is integer 0-255. T is real-valued, 0 - 1.

Hi,
by prepared T you convert to int as T'=(int)(T+0.5).
Y1=(T*X+1)>>1.
Y=Y1*T will have 15 bit exactitude, scaler 128 in mind.

Cheers

### question about fixed-integer precision in digital signal processing?

Hi Vic,

Quote:> by prepared T you convert to int as T'=(int)(T+0.5).

T is a matrix with element in [0, 1]... so do you mean T'=round(T*2^b),
where 2^b is a scaling factor? But it is unclear to me how to choose this
scaling factor optimally?

Quote:> Y1=(T*X+1)>>1.

The above expression is quite unfamiliar to me... why  +1 then >>1?

Quote:> Y=Y1*T will have 15 bit exactitude, scaler 128 in mind.

If you choose b=7, scaling 128, then the final result will be scaling
128*128=16384(14 bits), right?

Sorry I may need more help on this, can you explain to me a little more in
details?

Thanks a lot,

-Walala

### question about fixed-integer precision in digital signal processing?

Hi Walala,
i answered fastly but with errors as see now.
Quote:> > by prepared T you convert to int as T'=(int)(T+0.5).

sorry my negligance: T'=(int)(T>>8+0.5) .

Quote:> > Y1=(T*X+1)>>1.
> The above expression is quite unfamiliar to me... why  +1 then >>1?
> '+1' is round up for fixed number by /2. There is because my CCS bad work with unsigned numbers, for next round up use bit 14.
> > Y=Y1*T will have 15 bit exactitude, scaler 128 in mind.

Must be Y=(Y1*T+0x4000)>>8;

Quote:> If you choose b=7, scaling 128, then the final result will be scaling
> 128*128=16384(14 bits), right?

As you see, b=8 for T & b=7  for result.

Cheers

### question about fixed-integer precision in digital signal processing?

Again bad hurry: shift rezult only >>8, thus Y=(T*X*T+0x100)>>9
Shame for me

### question about fixed-integer precision in digital signal processing?

Walala & all have read above: i am sorry & never shall write after birthday's wine.

dear newgroup member,

first of all, thank for replying and i really appreciate your
help. recently i plan to built a audio  equalizer using one of the
digital signal processor.  in order to choose a cost effective
processor , i'm told to calculate the mips consume in my application.
my question is :

General EQ question ?

0) can you please tell me that which type of equalizer (i.e parametric
or graphic equalizer) is more worth to implement ?

0-1) do you know the ISO frequency standard frequency band for the
audio equalizer or the exact ISO number ?

0-2) can you suggest how many frequency band and the center frequency
for each band i should use for my project ?

Question on how much mips require for my application
1) how to calculate the mips need for my application ? what factor do
i need to consider when   carry up this process...

Testing equipment/software for EQ
2) since i'm building grahic equalizer, is that any equipment/software
i might require to use for testing my EQ. (in terms of testing
frequency response of EQ) ?

3) can somebody suggest a good site i have a full range of tutorial
regarding how to test the EQ frequency response?

Question Spectrum analyzer for EQ
4) beside building EQ, one other major things involve in my project is
building the spectrum analyzer for this EQ. do anybody know how to
implement it using a digital signal processor and graphical LCD
module? or else can somebody suggest some good website, books,
articles, tutorial or other related topics .

Commercial EQ product question
5) one last question is about the commercial EQ found in the market.
is it most of them are implement using digital signal processor or
made by digital filter IC or even made using anlogue electronics (i.e
using opamp and some rc circuit for some filter cct) ? what is the
normal price for it.

JMAX/MATLAB for EQ simulation Question
6) i'm planning to use some computer software to do simulation. what
i'd in my mind are either using JMAX or matlab to do my simuilation ,
my question is which one is good for those simulation and can somebody
suggest some good site that contain tutorial, sample code.

that'all my questions, i wish you can help me ,thank you.

best regards,

goh