Very Computer

Hi,

my company's business relies heavily on the gzip compression algorithm.
I was thinking about a PCI-card, with a small driver for let's say: linux, which
does only gzip, but really fast !!
Does anybody know a company or somebody who sells or can develop such a thing ?
However it must be x-times faster than today's top pentium !

--Andy

Quote:> Hi,

> my company's business relies heavily on the gzip compression algorithm.
> I was thinking about a PCI-card, with a small driver for let's say:
> linux, which does only gzip, but really fast !!  Does anybody know a
> company or somebody who sells or can develop such a thing ?  However it
> must be x-times faster than today's top pentium !

Sorry,  I fail to see what this has to do with Perl.

If you want to do compression in a Perl program use Compress::ZLib

/J\
--
Jonathan Stowe
http://www.gellyfish.com
http://www.tackleway.co.uk

I don't think you will find anything like this because it isn't really
possible. Compression is just number crunching and number crunching needs a
powerful computer.
As an example, you may think that your 3D graphics card draws 3D graphics X
times faster than your CPU, but this appearance is mainly simply because the
3D card has its own dedicated processor. You effecively have dual processor
system, which has more bandwidth for the 3D graphics. (There are lots of
other optimisations, such as faster memory access, but this is the main
one).
A 'gzip card' would simply be a computer. You might as well get a dual
processor system to start with.

What you can do (these are just ideas- I don't know how effective they will
be):
- Use a dedicated system with a fast network connection to do your gzipping.
It should preferably run a non-multitasking OS. This way, all the system's
power is going into the compression and not being wasted on other things,
e.g. talking to you.
- Use a RISC processor- some of these will go at over 1Ghz, and for this
sort of work, most of the extra instructions in a CISC processor will
probably not be much use.
- Use more than one processor: I don't know if there is any gzipping code
about that supports parallel processing, but it should be possible to get
some written. Alternatively, you may be able to do several operations at
once, one per processor.
- Possibly use a 64-bit processor- these have faster data throughput, but
since compression works at the byte or even bit level, a lot of this
advantage may be lost.
- Why do you need to do so much gzipping anyway? Might it not be more
straightforward just to invest in a large hard disk, DVD-RAM, or whatever?

-- Jamie Webb

Quote:> Hi,

> my company's business relies heavily on the gzip compression algorithm.
> I was thinking about a PCI-card, with a small driver for let's say: linux,
which
> does only gzip, but really fast !!
> Does anybody know a company or somebody who sells or can develop such a
thing ?
> However it must be x-times faster than today's top pentium !

> --Andy

On Fri, 07 Apr 2000 10:06:05 +0200, Andreas Vierengel

        I'm sure someone could develop an ASIC that could compress and
decompress gzip files quite effectively, but really, I think you'ld
find that the amount of time your system spent doing I/O to this card
would negate any performance advantage you might see.  Your best bets
for improving things would probably be to get a faster processor
and/or multiple processors and a faster hard drive, possibly some more
memory.

Anthony Hill

Quote:> >my company's business relies heavily on the gzip compression algorithm.
> >I was thinking about a PCI-card, with a small driver for let's say: linux, which
> >does only gzip, but really fast !!
> >Does anybody know a company or somebody who sells or can develop such a thing ?
> >However it must be x-times faster than today's top pentium !
>         I'm sure someone could develop an ASIC that could compress and

Or more realistically, a FPGA.  

Quote:> decompress gzip files quite effectively, but really, I think you'ld
> find that the amount of time your system spent doing I/O to this card
> would negate any performance advantage you might see.  Your best bets

Indeed; the first and most prudent thing would be to find out where the
bottleneck exists.  Is it I/O?  Memory?  Or is it the CPU?  I don't know
how Gzip compares with Winzip or other comparable programs which exist
on PC's, but I know that the PC-based programs tend to be I/O, and not
CPU bound.

Quote:> for improving things would probably be to get a faster processor
> and/or multiple processors and a faster hard drive, possibly some more
> memory.

Good advice, and a hell, (and I mean a hell) of a lot less expensive
than developing hardware to perform the task.

Thanks for the answers !

First, we do not do archiving or compressing large files.
We generate data on-the-fly, compress it with zlib (compression-level 1) and
send it over network. All in memory...
We don't have disk i/o for this purpose. Our bottleneck is CPU power...
So if we had an PCI-Busmaster DMA gzip-card with a kernel-module for Linux (:-),
we surely would benefit from this...

I have 18 Servers (single Processor 500MHz PIII) for this task and will upgrade
these with dual machines from VA-Linux...
If we do the same without compression we would need only 4 servers, but suffer
more traffic :-) and bandwith allocation...
Simply more servers is not a solution !

I thought about ASIC's, but currently don't know anybody who can develop such a
thing...
Maybe someone have adresses or telephone numbers ?

--Andy

Quote:> We generate data on-the-fly, compress it with zlib (compression-level 1) and
> send it over network. All in memory...
> We don't have disk i/o for this purpose. Our bottleneck is CPU power...

Have you had a look at LZO?

<http://wildsau.idv.uni-linz.ac.at/mfx/lzo.html>

AFAIR this was designed for on-the-fly-compresssion tasks.

XL
--
|____ Axel Schwenke,  Admin/Entwickler,  Jobs & Adverts Online AG ____|
|___  Linux is like a wigwam:  no Windows, no Gates, Apache inside ___|

Quote:>Thanks for the answers !

>First, we do not do archiving or compressing large files.
>We generate data on-the-fly, compress it with zlib (compression-level 1) and
>send it over network. All in memory...
>We don't have disk i/o for this purpose. Our bottleneck is CPU power...
>So if we had an PCI-Busmaster DMA gzip-card with a kernel-module for Linux (:-),
>we surely would benefit from this...

>I have 18 Servers (single Processor 500MHz PIII) for this task and will upgrade
>these with dual machines from VA-Linux...
>If we do the same without compression we would need only 4 servers, but suffer
>more traffic :-) and bandwith allocation...
>Simply more servers is not a solution !

>I thought about ASIC's, but currently don't know anybody who can develop such a
>thing...
>Maybe someone have adresses or telephone numbers ?

>--Andy

Try asking is sci.electronics.design
Some people tehre are involved very much with designing chips.
No guarantee for a solutuon though.
Jan

: Thanks for the answers !
:
: First, we do not do archiving or compressing large files.
: We generate data on-the-fly, compress it with zlib (compression-level 1) and
: send it over network. All in memory...
: We don't have disk i/o for this purpose. Our bottleneck is CPU power...

Hmm, why do you compress the data at all, if you don't have an i/o bottleneck?
Or use another compression scheme, which needs less CPU-power, e.g. LZW aka compress?
BTW it should be much easier to get compression hardware for LZW as it
is used in modems for compression as far as I remember.

: So if we had an PCI-Busmaster DMA gzip-card with a kernel-module for Linux (:-),
: we surely would benefit from this...
:
: I have 18 Servers (single Processor 500MHz PIII) for this task and will upgrade
: these with dual machines from VA-Linux...
: If we do the same without compression we would need only 4 servers, but suffer
: more traffic :-) and bandwith allocation...
: Simply more servers is not a solution !
:
: I thought about ASIC's, but currently don't know anybody who can develop such a
: thing...
: Maybe someone have adresses or telephone numbers ?
:
: --Andy

Contact the next university to you, which has an institute, that deals with
development of integrated circuits. They should have contacts to make
low volume chips of any sort. However, this definitely would not be cheap.
Probably the cheapest would be the programming of some sort of FPGA, since
you can buy these boards off the shelf, but the programming is quite complicated.

- Matthias Meixner

--

Technische Universit?t Darmstadt
Rechnerbetriebsgruppe                          Telefon (+49) 6151 16 6670
Wilhelminenstra?e 7, D-64283 Darmstadt, Germany    Fax (+49) 6151 16 4701

Thanks again for the answers.

I'm definetely stuck with gzip, since another compression algorithm is not
supported by the client program and we have no control over this programs :-(

--Andy

A thought....
GZip has a variety of compression levels.
Does it help if you use the "gzip -0" option (ie no compression,
basically it puts a header and footer and calculates cheksums but does
no actual compression - much faster).  A different compression level
would increase the data size but would save a _lot_ of CPU time.

The default with Gzip isusually maximum compression, even turning it
down a couple of steps would save heaps of CPU time without much impact
on the output size.

Good luck.

Alan.

--
* Hi! I'm a replicating .sig virus! Join the fun and copy me into yours!
:)

Today's subliminal thought is:

I'm still unclear as to what this has to do with Perl - if a hardware
solution is what is required then a high level programming language is not
going to help.  If you want to do compression with Perl then you can use
Compress::ZLib.  Once the specification for your hardware is available I am
sure we would be delighted to have the public interface so someone can
create a module that can use it.  Until that time it has nothing to do with
Perl.

/J\
--
No jokes, no taunting--That kid's got bosoms! Somebody get me a wet
towel! C'mere you butterball.
--
fortune oscar homer

How are your servers connected to "the network"? I'm somewhat sure
a Pentium III 500 class CPU could generate deflate compressed data
by a rate that is in the same order of magnitude as is 100Base.
You might consider to replace zlib by a self-hacked equivalent
that has hardcoded paths especially in the LZ77 search section
(while the zlib code already is quite effective using a 3-octet-
hash for sequence matching) and maybe even the occasional assembly
crafted inner loop. At least you should profile zlib to see where
exactly you lose cycles. I'd expect a gain factor of 1.5 to 2 by
such work (as it is cosmetic, not algorithmic). Maybe that is already
enough? If not, you may fire up a Pentium III specific tuner to
get te best out of the cache accesses and pipelines, and finally,
if nothing else helps, you might try whether some operations of
the compressor could benefit from parallelism coded through ISSE,
eventually using quite different algorithms than the original zlib
code does. After all, deflate is a standard for a format, not
really for an implementation (however zlib is the only existing
one I know of, and *is* effective, in the way you can be effective
in portable HLL code).

BTW, are you sure your hardware is well selected so to not lose
cycles that you could better use in deflate for silly waiting
and housekeeping stuff? I.e. do you have fast SCSI disks with
appropriate HA and especially a non-stupid network card (no
RTL8139 *[to be silent of NE2k and such])? Servers hope-
fully not running X with silly eyecandy? The other software
and the stuff APIing to zlib well engineered so cycles aren't
burned away there, either? You originally included a Perl group
in your crosspost, so one might have an explanation ;->

--

 "the big bang: the ultimate hero of low frequency;
  the divine intergalactical bassdrum" -- Yello, "solar driftwood"

+-o-+--------------------------------------------------------+-o-+
| o |               \\\- Brain Inside -///                   | o |
| o |                   ^^^^^^^^^^^^^^                       | o |
| o | Andre' Beck (ABPSoft)   AB10-RIPE    Xlink PoP Dresden | o |
+-o-+--------------------------------------------------------+-o-+