zlib and checking CRC - how?

zlib and checking CRC - how?

Post by jim_marshall2.. » Wed, 23 Aug 2000 04:00:00



I'm in the midst's of writing a program that can be updated via the
internet.  The program will download gzip'd versions of shared libraries
it uses. I found 'zlib' and have been using the file operation function
(gzopen, gzread) to decompress the file.

However; there doesn't appear to be any API in zlib to determine the
original CRC of the '.gz' file.  Additionally if the '.gz' file is
corrupted performing a gzopen, gzread will not cause any problems (at
least they return success values).

The question is, while using zlib how can I determine if the '.gz' file
is not corrupted?

Thanks
-Jim

Sent via Deja.com http://www.deja.com/
Before you buy.

 
 
 

zlib and checking CRC - how?

Post by jbrick.. » Wed, 23 Aug 2000 04:00:00


Zlib API is designed to be a compression API, not a full package....
However a CRC is not that difficult to implement yourself, and I don't
know about you, **I** feel safer when I know what CRC was used.  I
generally use two different CRC methods when I am truly worried about
data corruption.  The standard CRC-32 polynomial you will find source
code for on the internet, plus a checksum-sum (16-bit check sum, and
16-bit sum of the sum).  If you want to go REALLY far, you can apply a
32-bit CRC to the compressed data form to check BEFORE you de-compress
it and 32-bit CRC to check after, as much as you truly want to check,
but Zlib doesn't do it for you, you must do it yourself.
Just me,
jeff



> I'm in the midst's of writing a program that can be updated via the
> internet.  The program will download gzip'd versions of shared
libraries
> it uses. I found 'zlib' and have been using the file operation
function
> (gzopen, gzread) to decompress the file.

> However; there doesn't appear to be any API in zlib to determine the
> original CRC of the '.gz' file.  Additionally if the '.gz' file is
> corrupted performing a gzopen, gzread will not cause any problems (at
> least they return success values).

> The question is, while using zlib how can I determine if the '.gz'
file
> is not corrupted?

> Thanks
> -Jim

> Sent via Deja.com http://www.deja.com/
> Before you buy.

Sent via Deja.com http://www.deja.com/
Before you buy.

 
 
 

zlib and checking CRC - how?

Post by jim_marshall2.. » Thu, 24 Aug 2000 04:00:00


Jeff,
  Thanks for the reply.  yes I know I could do the CRC myself, but I'm
tryng to avoid adding that level of complexity to both my application
and the server app.  The main issue is changing how I get the updated
file from the server (it will need to pass a CRC to me - which isn't the
end of the world but I know getting the plp writing the server to add
this will be a fight...).

Besides - the '.gz' file has a CRC within it.  Why wouldn't 'libz' give
you access to that existing CRC value?  'gunzip' uses it, or uses some
information to verify the files integrity.

-Jim


> Zlib API is designed to be a compression API, not a full package....
> However a CRC is not that difficult to implement yourself, and I don't
> know about you, **I** feel safer when I know what CRC was used.  I
> generally use two different CRC methods when I am truly worried about
> data corruption.  The standard CRC-32 polynomial you will find source
> code for on the internet, plus a checksum-sum (16-bit check sum, and
> 16-bit sum of the sum).  If you want to go REALLY far, you can apply a
> 32-bit CRC to the compressed data form to check BEFORE you de-compress
> it and 32-bit CRC to check after, as much as you truly want to check,
> but Zlib doesn't do it for you, you must do it yourself.
> Just me,
> jeff



> > I'm in the midst's of writing a program that can be updated via the
> > internet.  The program will download gzip'd versions of shared
> libraries
> > it uses. I found 'zlib' and have been using the file operation
> function
> > (gzopen, gzread) to decompress the file.

> > However; there doesn't appear to be any API in zlib to determine the
> > original CRC of the '.gz' file.  Additionally if the '.gz' file is
> > corrupted performing a gzopen, gzread will not cause any problems
(at
> > least they return success values).

> > The question is, while using zlib how can I determine if the '.gz'
> file
> > is not corrupted?

> > Thanks
> > -Jim

> > Sent via Deja.com http://www.deja.com/
> > Before you buy.

> Sent via Deja.com http://www.deja.com/
> Before you buy.

Sent via Deja.com http://www.deja.com/
Before you buy.
 
 
 

zlib and checking CRC - how?

Post by nom.. » Sun, 27 Aug 2000 04:00:00


according to the zlib spec (RFC1950), zlib data streams are supposed to
contain
an adler32 32-bit checksum....

i'm not sure if/how this is actually implemented, though.... try corrupting
a file on purpose, then see if the decompression process fails or if it
works anyway.......

the decompress functions will _probably_ return with some kind of error
code......?

 
 
 

zlib and checking CRC - how?

Post by jim_marshall2.. » Tue, 29 Aug 2000 04:00:00


There are no errors returned from the zlib functions if the file is
corrupted.  Specifically I am using the gzopen, gzread, gzclose
functions.  zlib does provide a mechanism to calculate a CRC on data
that has been read in (gzread) but it doesn't have a way to know the
original CRC, so I can't see a way to check the computed CRC.....

-Jim


> according to the zlib spec (RFC1950), zlib data streams are supposed
to
> contain
> an adler32 32-bit checksum....

> i'm not sure if/how this is actually implemented, though.... try
corrupting
> a file on purpose, then see if the decompression process fails or if
it
> works anyway.......

> the decompress functions will _probably_ return with some kind of
error
> code......?

Sent via Deja.com http://www.deja.com/
Before you buy.
 
 
 

zlib and checking CRC - how?

Post by nrav » Thu, 31 Aug 2000 04:00:00


so gzread and gzclose never return with the value -1?

if they do, you could use gzerror to get the type of error....
_____

.....have a look at
http://www.info-zip.org/pub/infozip/zlib/manual.html#Introduction

- "The 'zlib' compression library provides in-memory compression and
decompression functions, __including integrity checks of the uncompressed
data.__ "

- "__The decoder checks the consistency of the compressed data__, so the
library should never crash even in case of corrupted input."


> There are no errors returned from the zlib functions if the file is
> corrupted.  Specifically I am using the gzopen, gzread, gzclose
> functions.  zlib does provide a mechanism to calculate a CRC on data
> that has been read in (gzread) but it doesn't have a way to know the
> original CRC, so I can't see a way to check the computed CRC.....

> -Jim


> > according to the zlib spec (RFC1950), zlib data streams are supposed
> to
> > contain
> > an adler32 32-bit checksum....

> > i'm not sure if/how this is actually implemented, though.... try
> corrupting
> > a file on purpose, then see if the decompression process fails or if
> it
> > works anyway.......

> > the decompress functions will _probably_ return with some kind of
> error
> > code......?

> Sent via Deja.com http://www.deja.com/
> Before you buy.

 
 
 

zlib and checking CRC - how?

Post by jim_marshall2.. » Thu, 31 Aug 2000 04:00:00


Not to keep this thread going for ever, but...

I believe the quote you mention is referring to the compression methods
in the library, not the file access methods.

regardless - the gzread, gzopen & gzclose functions do not seem to
return an error condition if the file is invalid.

Specifically the test I did (and it was an unintentional test) was the
HTTP server failed to send the file, so it created a 0 length file. When
I did a gzopen, gzread, gzclose I got no error messages.

I also took a gz file and cut out a 1/4 of the file (in the middle),
again no errors came from these functions.

-Jim


> so gzread and gzclose never return with the value -1?

> if they do, you could use gzerror to get the type of error....
> _____

> .....have a look at
> http://www.info-zip.org/pub/infozip/zlib/manual.html#Introduction

> - "The 'zlib' compression library provides in-memory compression and
> decompression functions, __including integrity checks of the
uncompressed
> data.__ "

> - "__The decoder checks the consistency of the compressed data__, so
the
> library should never crash even in case of corrupted input."



> > There are no errors returned from the zlib functions if the file is
> > corrupted.  Specifically I am using the gzopen, gzread, gzclose
> > functions.  zlib does provide a mechanism to calculate a CRC on data
> > that has been read in (gzread) but it doesn't have a way to know the
> > original CRC, so I can't see a way to check the computed CRC.....

> > -Jim


> > > according to the zlib spec (RFC1950), zlib data streams are
supposed
> > to
> > > contain
> > > an adler32 32-bit checksum....

> > > i'm not sure if/how this is actually implemented, though.... try
> > corrupting
> > > a file on purpose, then see if the decompression process fails or
if
> > it
> > > works anyway.......

> > > the decompress functions will _probably_ return with some kind of
> > error
> > > code......?

> > Sent via Deja.com http://www.deja.com/
> > Before you buy.

Sent via Deja.com http://www.deja.com/
Before you buy.
 
 
 

zlib and checking CRC - how?

Post by jim_marshall2.. » Thu, 31 Aug 2000 04:00:00


In case anyone cares, I figured out the problem. I had a loop which did
the gzread, I was only checking if gzread returned a negative number.
However; in the case of a zero length file, gzread will return zero.
Once I changed the code to detect the case of zero bytes read all was
well.



> I'm in the midst's of writing a program that can be updated via the
> internet.  The program will download gzip'd versions of shared
libraries
> it uses. I found 'zlib' and have been using the file operation
function
> (gzopen, gzread) to decompress the file.

> However; there doesn't appear to be any API in zlib to determine the
> original CRC of the '.gz' file.  Additionally if the '.gz' file is
> corrupted performing a gzopen, gzread will not cause any problems (at
> least they return success values).

> The question is, while using zlib how can I determine if the '.gz'
file
> is not corrupted?

> Thanks
> -Jim

> Sent via Deja.com http://www.deja.com/
> Before you buy.

Sent via Deja.com http://www.deja.com/
Before you buy.
 
 
 

1. OT - Cyclic Redundency Check (CRC)

I'm building on an app to configure a new model alarm system.  I need a CRC
algorithm or (preferably) a snippet of VB code to generate a CRC.  If any of you
good folks have VB source code or other info that would be helpful, it would be
greatly appreciated.

TIA

Regards,
Robert L Bass

=============================>
Bass Home Electronics
The Online DIY Alarm Store
http://www.Bass-Home.com
4883 Fallcrest Circle
Sarasota, FL 34233
877-722-8900 Sales & Tech Support
941-925-9747 voice (Florida)
941-926-9857 fax

=============================>

2. Cannot seem to clean ZA Pro off of the system.

3. NEWBIE question : What is the length for efficient CRC checking ?

4. Need -- Business Analyst -- Newark, NJ -- 12 months

5. "incorrect header check" using ZLIB Compression

6. DOS programs under Win2K

7. CRC Check Polynomial -> Byte presentation question

8. opera

9. CRC check

10. CRC Checks on microcontrollers

11. CRC Error Checking

12. PKZip W95 2.5 & check CRC

13. Q: are there competent fixers for zip files that fail CRC check ?