question about fixed-integer precision in digital signal processing?

question about fixed-integer precision in digital signal processing?

Post by walal » Mon, 18 Aug 2003 01:41:15



Dear all,

I am studying 2D DCT for an image. The 2D DCT can be summarized as computing
Y=T*X*T', where X is the input matrix, T is the 2D DCT basis matrix, Y is
the output.

X is integer 0-255. T is real-valued, 0 - 1.

In order to convert real-valued T to integers, I multiply T with 256(left
shift), and round it to integer, and use these new integer values in my
computing.

Now Y will be larger than its correct value by a factor of 256*256, so I
will divide it by 65536(right shift the result by 16 bits to get correct
result).

My question is, is my approach correct?

How many bits I need to allocate for internal matrix computation and the
output?

Is there any better approach to save some datawidth?

In general, are there any resources to study for dealing with
real-to-integer conversion?

Thanks a lot,

-Walala

 
 
 

question about fixed-integer precision in digital signal processing?

Post by Vi » Tue, 19 Aug 2003 20:08:00


Quote:> Y=T*X*T'

> X is integer 0-255. T is real-valued, 0 - 1.

Hi,
by prepared T you convert to int as T'=(int)(T+0.5).
Y1=(T*X+1)>>1.
Y=Y1*T will have 15 bit exactitude, scaler 128 in mind.

Cheers

 
 
 

question about fixed-integer precision in digital signal processing?

Post by walal » Wed, 20 Aug 2003 00:32:30


Hi Vic,

Quote:> by prepared T you convert to int as T'=(int)(T+0.5).

T is a matrix with element in [0, 1]... so do you mean T'=round(T*2^b),
where 2^b is a scaling factor? But it is unclear to me how to choose this
scaling factor optimally?

Quote:> Y1=(T*X+1)>>1.

The above expression is quite unfamiliar to me... why  +1 then >>1?

Quote:> Y=Y1*T will have 15 bit exactitude, scaler 128 in mind.

If you choose b=7, scaling 128, then the final result will be scaling
128*128=16384(14 bits), right?

Sorry I may need more help on this, can you explain to me a little more in
details?

Thanks a lot,

-Walala

 
 
 

question about fixed-integer precision in digital signal processing?

Post by Vi » Wed, 20 Aug 2003 23:56:07


Hi Walala,
i answered fastly but with errors as see now.
Quote:> > by prepared T you convert to int as T'=(int)(T+0.5).

sorry my negligance: T'=(int)(T>>8+0.5) .

Quote:> > Y1=(T*X+1)>>1.
> The above expression is quite unfamiliar to me... why  +1 then >>1?
> '+1' is round up for fixed number by /2. There is because my CCS bad work with unsigned numbers, for next round up use bit 14.
> > Y=Y1*T will have 15 bit exactitude, scaler 128 in mind.

Must be Y=(Y1*T+0x4000)>>8;

Quote:> If you choose b=7, scaling 128, then the final result will be scaling
> 128*128=16384(14 bits), right?

As you see, b=8 for T & b=7  for result.
Sorry my bad hurry.

Cheers

 
 
 

question about fixed-integer precision in digital signal processing?

Post by Vi » Thu, 21 Aug 2003 00:03:50


Again bad hurry: shift rezult only >>8, thus Y=(T*X*T+0x100)>>9
Shame for me
 
 
 

question about fixed-integer precision in digital signal processing?

Post by Vi » Thu, 21 Aug 2003 20:33:03


Walala & all have read above: i am sorry & never shall write after birthday's wine.
 
 
 

1. Digital Audio Processing , EQ question, Spectrum analyzer question

dear newgroup member,

        first of all, thank for replying and i really appreciate your
help. recently i plan to built a audio  equalizer using one of the
digital signal processor.  in order to choose a cost effective
processor , i'm told to calculate the mips consume in my application.
my question is :

General EQ question ?

0) can you please tell me that which type of equalizer (i.e parametric
or graphic equalizer) is more worth to implement ?

0-1) do you know the ISO frequency standard frequency band for the
audio equalizer or the exact ISO number ?

0-2) can you suggest how many frequency band and the center frequency
for each band i should use for my project ?

Question on how much mips require for my application
1) how to calculate the mips need for my application ? what factor do
i need to consider when   carry up this process...

Testing equipment/software for EQ
2) since i'm building grahic equalizer, is that any equipment/software
i might require to use for testing my EQ. (in terms of testing
frequency response of EQ) ?

3) can somebody suggest a good site i have a full range of tutorial
regarding how to test the EQ frequency response?

Question Spectrum analyzer for EQ
4) beside building EQ, one other major things involve in my project is
building the spectrum analyzer for this EQ. do anybody know how to
implement it using a digital signal processor and graphical LCD
module? or else can somebody suggest some good website, books,
articles, tutorial or other related topics .

Commercial EQ product question
5) one last question is about the commercial EQ found in the market.
is it most of them are implement using digital signal processor or
made by digital filter IC or even made using anlogue electronics (i.e
using opamp and some rc circuit for some filter cct) ? what is the
normal price for it.

JMAX/MATLAB for EQ simulation Question
6) i'm planning to use some computer software to do simulation. what
i'd in my mind are either using JMAX or matlab to do my simuilation ,
my question is which one is good for those simulation and can somebody
suggest some good site that contain tutorial, sample code.

that'all my questions, i wish you can help me ,thank you.

best regards,

goh

2. firewire question/ Nikon LS 8000

3. WTB: Multirate Digital Signal Processing / Rabiner & Crochiere

4. Making sendmail 8.8.7 work under Solaris 2.5.1

5. DIGITAL SIGNAL PROCESSING DESIGN BOOK FOR $35.00

6. Nested hash creation help !

7. Two Workshops on Digital Signal Processing for Audio at Stanford

8. 5mx hangs during IR comms

9. Senior Digital Signal Processing Engineer

10. Looking for "Digital Signal Processing" function pack for Mathcad

11. Biological signals digital processing

12. Any good digital signal process course website