compressing a file on NT

compressing a file on NT

Post by Nitin Pu » Thu, 24 Jan 2002 03:45:41



I need to have a command line utility on NT to compress a file on the
fly as the size of the file is going beyond 2GB.
Please suggest a solution.

Regards

 
 
 

compressing a file on NT

Post by Michael L » Thu, 24 Jan 2002 05:02:09



Quote:>I need to have a command line utility on NT to compress a file on the
>fly as the size of the file is going beyond 2GB.
>Please suggest a solution.

>Regards

You can use NT filesystem compression, but I HIGHLY recommend against it.

Is this for a checkpoint, work file, or for a datafile?

For a datafile, spread the table over several locations, even if they
are on the same disk.

For a work file, define several work locations to "stripe" the temporary
file.

For a checkpoint, you need to modify cktmpl.def to use a compression
utility like gzip, zip, pkzip, etc.  I have used them all quite
successfully.

Hope this helps,


Caribou Lake Software  http://www.cariboulake.com  Ingres, JDBC

Specializing in Ingres/Oracle integration and conversion

 
 
 

compressing a file on NT

Post by Nitin Pu » Thu, 24 Jan 2002 05:22:55


Hi Michael,

   Thanks so much for the reply.
   It's is a datafile, particularly a database dump file, which I want to compress.
   Your suggestion for the datafile may not work over here, as it is a generated file
   and I need to compress it on the fly.
   On Unix I don't have any problem, as I have compress command at my disposal.
   I need to port it to NT, which makes me particular about the command line
   DOS utility.
   I have tried COMPACT.exe which comes with WinNT, but the limitation is
   that this command works only on the files in NTFS partitions.
   I want to compress a file in MVFS(clearcase) partition.
   I hope I've been able to communicate my problem.

   Any suggestions please.

   Regards,

   Nitin

  | Date: Tue, 22 Jan 2002 13:41:59 -0600


  | Subject: Re: compressing a file on NT
  |

  | >I need to have a command line utility on NT to compress a file on the
  | >fly as the size of the file is going beyond 2GB.
  | >Please suggest a solution.
  | >
  | >Regards
  |
  | You can use NT filesystem compression, but I HIGHLY recommend against it.
  |
  | Is this for a checkpoint, work file, or for a datafile?
  |
  | For a datafile, spread the table over several locations, even if they
  | are on the same disk.
  |
  | For a work file, define several work locations to "stripe" the temporary
  | file.
  |
  | For a checkpoint, you need to modify cktmpl.def to use a compression
  | utility like gzip, zip, pkzip, etc.  I have used them all quite
  | successfully.
  |
  | Hope this helps,
  |
  |

  | Caribou Lake Software  http://www.cariboulake.com  Ingres, JDBC
  |
  | Specializing in Ingres/Oracle integration and conversion
  |

 
 
 

compressing a file on NT

Post by Bill Hoelz » Thu, 24 Jan 2002 05:40:14



Quote:>I need to have a command line utility on NT to compress a file on the
>fly as the size of the file is going beyond 2GB.
>Please suggest a solution.

You could use the WinZip utility.

http://www.winzip.com

This utility can be run from the command line or via a batch job in
addition to the familiar GUI.

 
 
 

compressing a file on NT

Post by Michael L » Thu, 24 Jan 2002 06:30:41



Quote:>Hi Michael,

>    Thanks so much for the reply.
>    It's is a datafile, particularly a database dump file, which I want to
> compress.
>    Your suggestion for the datafile may not work over here, as it is a
> generated file
>    and I need to compress it on the fly.
>    On Unix I don't have any problem, as I have compress command at my
> disposal.
>    I need to port it to NT, which makes me particular about the command line
>    DOS utility.
>    I have tried COMPACT.exe which comes with WinNT, but the limitation is
>    that this command works only on the files in NTFS partitions.
>    I want to compress a file in MVFS(clearcase) partition.
>    I hope I've been able to communicate my problem.

>    Any suggestions please.

>    Regards,

>    Nitin

Nitin,

Are you SURE this is a DUMP file?  (The file of transactions that occur
during an online checkpoint, named NNNNNNNN.dmp)

A 2GB dump file indicates EXTREMELY heavy transactions during an on-line
checkpoint.  I have a hard time believing that is the case.

If this is a CHECKPOINT file (the file generated when backing up
Ingres databases using ckpdb), then I could suggest using the public
ZIP utility, PKZIP (shareware/commercial), or GZIP (GNU, free).  You
will need to modify CKTMPL.DEF in the %II_SYSTEM%/ingres/files directory.

DON'T use COMPACT.  Evil.  Evil.  Evil.  Evil.  Why?  Dunno.  Experience
says I usually re-install the O/S one month after using it ;-}

If this is a file generated by UNLOADDB scripts, then you may need
to do some MAGIC and hack the unload script, unload through named pipes,
and compress on the end of that pipe.  Never done it on NT, but I hear it
can be done.

Hope that helps,

Cheers,


Caribou Lake Software  http://www.cariboulake.com  Ingres, JDBC

Specializing in Ingres/Oracle integration and conversion

 
 
 

compressing a file on NT

Post by Fernando Hev » Thu, 24 Jan 2002 06:36:44


I recommend you get a copy of PKZIP or a similar (old) compress utility.
Anyway, chances are you have Winzip already installed, in that case check
out this link for the command line parameters that Winzip supports:

http://www.winzip.com/wzcline.htm

You can void the graphic interfase using these parameters.




CC:
Asunto:   compressing a file on NT

I need to have a command line utility on NT to compress a file on the
fly as the size of the file is going beyond 2GB.
Please suggest a solution.

Regards

 
 
 

compressing a file on NT

Post by Gibson Jonath » Thu, 24 Jan 2002 20:40:24


I would like to know why you recommend against using NT filesystem
compression?  At my previous company we ran Ingres on an IBM Netfinity box
with compression on without ever hitting a problem.  We used to run it on an
HP NT server and that box gave us heaps of trouble with just about
everything so I'm wondering if this is related to hardware.  I only ask as I
still have friends their supporting the system.

Regards
Jon Gibson
Ingres Database Administrator
Hiscox Plc
+44 (020) 7448 6820

PS I'm now happily back on AIX and I'm off the valium at last!!!

-----Original Message-----

Sent: 22 January 2002 19:42

Subject: Re: compressing a file on NT


>I need to have a command line utility on NT to compress a file on the
>fly as the size of the file is going beyond 2GB.
>Please suggest a solution.

>Regards

You can use NT filesystem compression, but I HIGHLY recommend against it.

Is this for a checkpoint, work file, or for a datafile?

For a datafile, spread the table over several locations, even if they
are on the same disk.

For a work file, define several work locations to "stripe" the temporary
file.

For a checkpoint, you need to modify cktmpl.def to use a compression
utility like gzip, zip, pkzip, etc.  I have used them all quite
successfully.

Hope this helps,


Caribou Lake Software  http://www.cariboulake.com  Ingres, JDBC

Specializing in Ingres/Oracle integration and conversion

<font size ='1' face='arial'>**********************************************************************
This email and any files transmitted with it are confidential and
intended solely for the use of the individual or entity to whom they
are addressed. No one else is authorised to distribute, forward,
print, copy or act upon any information contained in this email.
If you have received this email in error, please notify the sender.

Hiscox plc is a company registered in England and Wales under
company registration number 2837811 and registered office at
1 Great St Helen's, London EC3A 6HX
**********************************************************************</font>

 
 
 

compressing a file on NT

Post by Michael L » Thu, 24 Jan 2002 23:58:27



Quote:>I would like to know why you recommend against using NT filesystem
>compression?  At my previous company we ran Ingres on an IBM Netfinity box
>with compression on without ever hitting a problem.  We used to run it on
>an HP NT server and that box gave us heaps of trouble with just about
>everything so I'm wondering if this is related to hardware.  I only ask as
>I still have friends their supporting the system.

That is exactly why!  I can't recommend something that only works
part of the time!  I am not willing to test something as basic as the
filesystem!

Quote:>Regards
>Jon Gibson
>Ingres Database Administrator
>Hiscox Plc
>+44 (020) 7448 6820

>PS I'm now happily back on AIX and I'm off the valium at last!!!

Yup.  That will do it!

Cheers,


Caribou Lake Software  http://www.cariboulake.com  Ingres, JDBC

Specializing in Ingres/Oracle integration and conversion

 
 
 

compressing a file on NT

Post by Nitin Pu » Fri, 25 Jan 2002 03:04:16


Ok, let's shift gear. I have decided that as I don't have anything as "a simple
Dos prompt command" for compressing on MVFS partition, I have decided to
move over to gzip.
This utility works just fine, but again with a limitation. I'm trying to compress
a file on the fly. The file is open and I write to it. This is, then that I
pipe it into the "gzip" command to compress it on the fly but I get the error
message "Permission denied", which I construe comes from the fact that the file
is being used by another process. We cannot, first write out the file and then
compress it because we exceed virtual memory in that case.
Any suggestions. Michael ?

Regards

Nitin Puri
Logic Design & Verification
Cadence Design Systems Inc.

  | Date: Wed, 23 Jan 2002 08:47:13 -0600


  | Subject: RE: compressing a file on NT
  |

  |
  | >I would like to know why you recommend against using NT filesystem
  | >compression?  At my previous company we ran Ingres on an IBM Netfinity box
  | >with compression on without ever hitting a problem.  We used to run it on
  | >an HP NT server and that box gave us heaps of trouble with just about
  | >everything so I'm wondering if this is related to hardware.  I only ask as
  | >I still have friends their supporting the system.
  |
  | That is exactly why!  I can't recommend something that only works
  | part of the time!  I am not willing to test something as basic as the
  | filesystem!
  |
  | >Regards
  | >Jon Gibson
  | >Ingres Database Administrator
  | >Hiscox Plc
  | >+44 (020) 7448 6820
  | >
  | >PS I'm now happily back on AIX and I'm off the valium at last!!!
  |
  | Yup.  That will do it!
  |
  | Cheers,
  |
  |

  | Caribou Lake Software  http://www.cariboulake.com  Ingres, JDBC
  |
  | Specializing in Ingres/Oracle integration and conversion

 
 
 

compressing a file on NT

Post by Michael L » Fri, 25 Jan 2002 03:43:02


Nitin,

You cannot use gzip in this fashion.

What are you trying to compress on the fly?  If you are trying
to get around the 2GB limitation of a single file, you cannot
do so until later versions of Ingres that support "large file access".

It doesn't matter that the OS compresses the file, Ingres cannot address
more than 2GB of data with the 32 bit file I/O interface.

Again, PLEASE tell us what you are trying to do, and why.  Please
tell us the EXACT files you are trying to compress.

Cheers,

Mike Leo


>Ok, let's shift gear. I have decided that as I don't have anything as "a
>simple
>Dos prompt command" for compressing on MVFS partition, I have decided to
>move over to gzip.
>This utility works just fine, but again with a limitation. I'm trying to
>compress
>a file on the fly. The file is open and I write to it. This is, then that I
>pipe it into the "gzip" command to compress it on the fly but I get the error
>message "Permission denied", which I construe comes from the fact that the
>file
>is being used by another process. We cannot, first write out the file and then
>compress it because we exceed virtual memory in that case.
>Any suggestions. Michael ?

>Regards

>Nitin Puri
>Logic Design & Verification
>Cadence Design Systems Inc.

>   | Date: Wed, 23 Jan 2002 08:47:13 -0600



>   | Subject: RE: compressing a file on NT
>   |

>   |
>   | >I would like to know why you recommend against using NT filesystem
>   | >compression?  At my previous company we ran Ingres on an IBM
> Netfinity box
>   | >with compression on without ever hitting a problem.  We used to run
> it on
>   | >an HP NT server and that box gave us heaps of trouble with just about
>   | >everything so I'm wondering if this is related to hardware.  I only
> ask as
>   | >I still have friends their supporting the system.
>   |
>   | That is exactly why!  I can't recommend something that only works
>   | part of the time!  I am not willing to test something as basic as the
>   | filesystem!
>   |
>   | >Regards
>   | >Jon Gibson
>   | >Ingres Database Administrator
>   | >Hiscox Plc
>   | >+44 (020) 7448 6820
>   | >
>   | >PS I'm now happily back on AIX and I'm off the valium at last!!!
>   |
>   | Yup.  That will do it!
>   |
>   | Cheers,
>   |
>   |

>   | Caribou Lake Software  http://www.cariboulake.com  Ingres, JDBC
>   |
>   | Specializing in Ingres/Oracle integration and conversion

 
 
 

compressing a file on NT

Post by Peter Ga » Sat, 26 Jan 2002 04:19:13


Gzip!!  In the same family as gtar!! That's the one with the
-ignore_failed_read parameter. Much loved of IBM AIX users where the
standard tar routine falls over (rather than produce a warning) if the
file changes size during the tar. -ignore_failed_read makes gtar (gzip)
ignore that problem and any other IO errors it comes across. In other
words it is not compatible with the OS. So what you ask? Well it's a
fundemental part of on-line checkpoints that we can continue to write to
the database whilst taking a backup. Thus it is possible for table files
to grow. Tar takes a snaphot of file sizes before starting the copy and
will moan quietly if the file changes size. Gtar (and gzip) both *
like hell if the file changes size and you get an error.

Why do I tell you this. Well because IMHO you should in most cases stick
to using the software that is provided with the OS when it comes to
copying and compressing files. With NT you should stick to using the
software it comes with period (Ingres does not come with NT (Think about
it!!), nor does O*****E etc)

I must be in a bad mood!!!

Peter Gale
Director
PJG Computer Services Ltd
+44 (0)1398 341491 (o)
+44 (0)7831 513181 (m)

-----Original Message-----


Sent: 23 January 2002 17:49


Subject: RE: compressing a file on NT

Ok, let's shift gear. I have decided that as I don't have anything as "a
simple Dos prompt command" for compressing on MVFS partition, I have
decided to
move over to gzip.
This utility works just fine, but again with a limitation. I'm trying to
compress a file on the fly. The file is open and I write to it. This is,
then that I pipe it into the "gzip" command to compress it on the fly
but I get the error message "Permission denied", which I construe comes
from the fact that the file is being used by another process. We cannot,
first write out the file and then compress it because we exceed virtual
memory in that case. Any suggestions. Michael ?

Regards

Nitin Puri
Logic Design & Verification
Cadence Design Systems Inc.

  | Date: Wed, 23 Jan 2002 08:47:13 -0600



  | Subject: RE: compressing a file on NT
  |

  |
  | >I would like to know why you recommend against using NT filesystem
  | >compression?  At my previous company we ran Ingres on an IBM
Netfinity box
  | >with compression on without ever hitting a problem.  We used to run
it on
  | >an HP NT server and that box gave us heaps of trouble with just
about
  | >everything so I'm wondering if this is related to hardware.  I only
ask as
  | >I still have friends their supporting the system.
  |
  | That is exactly why!  I can't recommend something that only works
  | part of the time!  I am not willing to test something as basic as
the
  | filesystem!
  |
  | >Regards
  | >Jon Gibson
  | >Ingres Database Administrator
  | >Hiscox Plc
  | >+44 (020) 7448 6820
  | >
  | >PS I'm now happily back on AIX and I'm off the valium at last!!!
  |
  | Yup.  That will do it!
  |
  | Cheers,
  |
  |

  | Caribou Lake Software  http://www.veryComputer.com/  Ingres, JDBC
  |
  | Specializing in Ingres/Oracle integration and conversion

 
 
 

compressing a file on NT

Post by Michael L » Sat, 26 Jan 2002 05:49:01


I've used AIX for almost 10 years, and I have had GOBS of
issues with AIX's tar as well.

I disagree.  Fear of non-vendor software is why this world is
in such an wonderful mess with Winblows ...

Mikey


>Gzip!!  In the same family as gtar!! That's the one with the
>-ignore_failed_read parameter. Much loved of IBM AIX users where the
>standard tar routine falls over (rather than produce a warning) if the
>file changes size during the tar. -ignore_failed_read makes gtar (gzip)
>ignore that problem and any other IO errors it comes across. In other
>words it is not compatible with the OS. So what you ask? Well it's a
>fundemental part of on-line checkpoints that we can continue to write to
>the database whilst taking a backup. Thus it is possible for table files
>to grow. Tar takes a snaphot of file sizes before starting the copy and
>will moan quietly if the file changes size. Gtar (and gzip) both *
>like hell if the file changes size and you get an error.

>Why do I tell you this. Well because IMHO you should in most cases stick
>to using the software that is provided with the OS when it comes to
>copying and compressing files. With NT you should stick to using the
>software it comes with period (Ingres does not come with NT (Think about
>it!!), nor does O*****E etc)

>I must be in a bad mood!!!

>Peter Gale
>Director
>PJG Computer Services Ltd
>+44 (0)1398 341491 (o)
>+44 (0)7831 513181 (m)

>-----Original Message-----


>Sent: 23 January 2002 17:49


>Subject: RE: compressing a file on NT

>Ok, let's shift gear. I have decided that as I don't have anything as "a
>simple Dos prompt command" for compressing on MVFS partition, I have
>decided to
>move over to gzip.
>This utility works just fine, but again with a limitation. I'm trying to
>compress a file on the fly. The file is open and I write to it. This is,
>then that I pipe it into the "gzip" command to compress it on the fly
>but I get the error message "Permission denied", which I construe comes
>from the fact that the file is being used by another process. We cannot,
>first write out the file and then compress it because we exceed virtual
>memory in that case. Any suggestions. Michael ?

>Regards

>Nitin Puri
>Logic Design & Verification
>Cadence Design Systems Inc.

>   | Date: Wed, 23 Jan 2002 08:47:13 -0600



>   | Subject: RE: compressing a file on NT
>   |

>   |
>   | >I would like to know why you recommend against using NT filesystem
>   | >compression?  At my previous company we ran Ingres on an IBM
>Netfinity box
>   | >with compression on without ever hitting a problem.  We used to run
>it on
>   | >an HP NT server and that box gave us heaps of trouble with just
>about
>   | >everything so I'm wondering if this is related to hardware.  I only
>ask as
>   | >I still have friends their supporting the system.
>   |
>   | That is exactly why!  I can't recommend something that only works
>   | part of the time!  I am not willing to test something as basic as
>the
>   | filesystem!
>   |
>   | >Regards
>   | >Jon Gibson
>   | >Ingres Database Administrator
>   | >Hiscox Plc
>   | >+44 (020) 7448 6820
>   | >
>   | >PS I'm now happily back on AIX and I'm off the valium at last!!!
>   |
>   | Yup.  That will do it!
>   |
>   | Cheers,
>   |
>   |

>   | Caribou Lake Software  http://www.veryComputer.com/  Ingres, JDBC
>   |
>   | Specializing in Ingres/Oracle integration and conversion

 
 
 

compressing a file on NT

Post by Chris Da » Fri, 01 Feb 2002 21:19:12


At one client, I found that they were not clearing out their temporary
tables, and so the list of filenames in one of the database data
directories became very long.
It exceeded the length of command-line which tar would handle, so tar
did nothing - and returned a success status!!
Infodb said that the checkpoints were valid, but the checkpoint files
were all empty.

I do not know what happens on other Unices (but there is now a DAR in
the system to check for zero-length checkpoint files).

Chris Dawe


>I've used AIX for almost 10 years, and I have had GOBS of
>issues with AIX's tar as well.

>I disagree.  Fear of non-vendor software is why this world is
>in such an wonderful mess with Winblows ...

>Mikey

[snip]
--

 
 
 

compressing a file on NT

Post by Peter Ga » Mon, 04 Feb 2002 20:34:11


Chris,

This can happen all unixes. To get around it replace the * with a .
(dot) In the tar command in the template file and hey-presto the
filename list problem goes away. Also you should get an error when tar
gets a parameter list that is too long. Are you piping the output
through something else (compress?). If you are then you need to pick up
the stderr output from the tar and check it some how.

One method is to modify the WSDD line like this (simplified)

cd %D;tar cf - . 2>tar.err.%M | compress %A.Z

Then check the contents of tar.err.* in the ESDD line. If there is
anything in the file then do an exit 1 in the checking script and the
checkpoint will be marked invalid.

HTH

Peter Gale
Director
PJG Computer Services Ltd
+44 (0)1398 341491 (o)
+44 (0)7831 513181 (m)

-----Original Message-----


Sent: 31 January 2002 11:26
To: Michael Leo

Subject: Re: compressing a file on NT

At one client, I found that they were not clearing out their temporary
tables, and so the list of filenames in one of the database data
directories became very long. It exceeded the length of command-line
which tar would handle, so tar did nothing - and returned a success
status!! Infodb said that the checkpoints were valid, but the checkpoint
files were all empty.

I do not know what happens on other Unices (but there is now a DAR in
the system to check for zero-length checkpoint files).

Chris Dawe


>I've used AIX for almost 10 years, and I have had GOBS of issues with
>AIX's tar as well.

>I disagree.  Fear of non-vendor software is why this world is in such
>an wonderful mess with Winblows ...

>Mikey
[snip]
--