compressing on the fly: father process seems to end before popened compress

compressing on the fly: father process seems to end before popened compress

Post by Fernando Hev » Thu, 28 Nov 2002 05:54:40



Hi all.

Got this weird problem. Program A starts a child process (no forking)
with popen to compress data written to disk on the fly. This seems to
work ok. The thing is that immediately after program A terminates, an
invoking script moves the new file to another location. It turns out
that the file gets truncated after it is moved. I get the sense that
compress didn't finish before the file was moved. I'm doing a pclose
on the pipe of course so this is really strange. I'm flushing every
write too, but to no effect.
The strangest thing is that this doesn't happen always. Sometimes
moved files are chopped sometimes they arent. Probably due to system
load. If I dont call mv in the script the whole process works just
fine, files are always created ok. Also if I introduce a "sleep 1"
before de mv in the script, no files get truncated. Of course I dont
like this workaround. Would rather have prog_a stop execution till
compress is completely finished. But I thought pclose would arrange
for that.

Any ideas of what can be happening? Any of you guys ever walked into a
similar behavior? Im working on Digital Unix if that matters for
anything.

This is an excerpt of my code:

*** script.sh ***
prog_a newfile.Z
mv newfile.Z /another_fs/another_dir/newfile.Z

*** progr_a ***
fout = popen("/usr/bin/compress - > newfile.Z", "w");
if (fout==NULL){ perror("popen didn't work"); exit(1); }

/* Write data and flush every time */
while(...){
   fprintf(fout, ...);
   fflush(fout);

Quote:}

if( pclose(fout)!= 0){
   /* never gets in here so pclose works fine */
   perror("oops!");

Quote:}

Thanks in advance for any help you can provide.

Fernando Hevia
Telecom Personal
Cordoba, Argentina

 
 
 

compressing on the fly: father process seems to end before popened compress

Post by Walter Brisco » Thu, 28 Nov 2002 07:54:50



Nov 2002 12:54:40 in comp.unix.programmer, Fernando Hevia

Quote:>Hi all.

>Got this weird problem. Program A starts a child process (no forking)
>with popen to compress data written to disk on the fly. This seems to
>work ok. The thing is that immediately after program A terminates, an
>invoking script moves the new file to another location. It turns out
>that the file gets truncated after it is moved. I get the sense that
>compress didn't finish before the file was moved. I'm doing a pclose
>on the pipe of course so this is really strange. I'm flushing every
>write too, but to no effect.

[snip]

Quote:>/* Write data and flush every time */
>while(...){
>   fprintf(fout, ...);
>   fflush(fout);

I would expect fflush() to be a bad idea as it increases the number of
I/O operations.
Quote:>}

I suspect you may expect a problem with disk access.
NFS can be funny in this area. Although it is not aesthetic, you can
tune down the delay with usleep(), select(), etc. sync() may work,
fsync() is inappropriate. I prefer bzip2 to compress as it is usually
more effective and it seems to be easy to absorb into a process. In that
case, an fsync() would be OK.
--
Walter Briscoe

 
 
 

compressing on the fly: father process seems to end before popened compress

Post by Dmitry Karasi » Fri, 29 Nov 2002 18:46:53


        Hi Fernando!


 Fernando> *** script.sh *** prog_a newfile.Z mv newfile.Z
 Fernando> /another_fs/another_dir/newfile.Z

I think /another_fs/ is the key of the problem, - since
the file move operation is only atomary within the same
filesystem, your mv is smart enough not to complain when
the filesystems are different, but actually copy the file
instead. You have to guarantee thus that /usr/bin/compress
is finished before you exit from prog_a, which you cannot
with pclose(). To do so, use explicit fork() and wait()
instead of popen()/pclose().

 Fernando> *** progr_a *** fout = popen("/usr/bin/compress - > newfile.Z",
 Fernando> "w"); if (fout==NULL){ perror("popen didn't work"); exit(1); }

 Fernando> /* Write data and flush every time */ while(...){ fprintf(fout,
 Fernando> ...); fflush(fout); }

 Fernando> if( pclose(fout)!= 0){ /* never gets in here so pclose works
 Fernando> fine */ perror("oops!"); }

 Fernando> Thanks in advance for any help you can provide.

 Fernando> Fernando Hevia Telecom Personal Cordoba, Argentina

--
Sincerely,
        Dmitry

--- www.karasik.eu.org ---

Life ain't fair, but the root password helps.
                      - BOFH

 
 
 

compressing on the fly: father process seems to end before popened compress

Post by Fernando Hev » Sat, 07 Dec 2002 05:21:27




> Nov 2002 12:54:40 in comp.unix.programmer, Fernando Hevia

> >Hi all.

> >Got this weird problem. Program A starts a child process (no forking)
> >with popen to compress data written to disk on the fly. This seems to
> >work ok. The thing is that immediately after program A terminates, an
> >invoking script moves the new file to another location. It turns out
> >that the file gets truncated after it is moved. I get the sense that
> >compress didn't finish before the file was moved. I'm doing a pclose
> >on the pipe of course so this is really strange. I'm flushing every
> >write too, but to no effect.
> [snip]

> >/* Write data and flush every time */
> >while(...){
> >   fprintf(fout, ...);
> >   fflush(fout);
> I would expect fflush() to be a bad idea as it increases the number of
> I/O operations.

Agreed on that. Just included the fflush() as a desperate measure.

Quote:> I suspect you may expect a problem with disk access.
> NFS can be funny in this area. Although it is not aesthetic, you can
> tune down the delay with usleep(), select(), etc. sync() may work,
> fsync() is inappropriate. I prefer bzip2 to compress as it is usually
> more effective and it seems to be easy to absorb into a process. In that
> case, an fsync() would be OK.

The delay is really not an option, it's too messy and unpredictable
for my taste. Nevertheless, sync seems to work Ok. Thanks
 
 
 

1. COMPRESS (how much does it compress?)

I was wondering if someone could tell me what the average compression
factor is for the compress command?  We are compressing some of the
filesystems in our nightly backups which makes it difficult to compute
how much data is actually being written to tape.  If I knew the average
compression factor then I could make better estimates.

The Sun man page says 50-60% for English text or source-code.  What
about for an entire filesystem?  What about for an oracle database?
What about other things like say binaries?

Does anyone know?

--
--
Alan W. McKay           | (902)542-2201.158     | Wolfville, N.S. Canada

2. swap partition deleted

3. Compressed filesystem or Compressed loop?

4. EMERGENCY (sendmail)

5. To compress or not to compress, that is the question.

6. Kernel Panic

7. Compressed 2.2 kernel about 35% larger than compressed 2.0.x?

8. gnome setup error

9. On-the-Fly disk compress

10. popen, pclose, and compress

11. scp/compress on fly

12. TAR/compress on-the-fly?