scp/compress on fly

scp/compress on fly

Post by rich for » Fri, 23 Feb 2001 14:47:50

I am trying to scp and compress on the fly:

compress<temp.dmp>file.Z &
rcp host:directory/file . > temp.dmp

This doesn't do the trick.


scp/compress on fly

Post by Logan Sh » Fri, 23 Feb 2001 16:10:49

Quote:>I am trying to scp and compress on the fly:

>compress<temp.dmp>file.Z &
>rcp host:directory/file . > temp.dmp

>This doesn't do the trick.

This doesn't work because the compress sees end of file whenever it
reaches the last byte of the file.  It doesn't care or know that
another program has the file open.  (In fact, when compress first
starts, the other file doesn't have the file open yet, so even if
compress did care, your script would only work some of the time.)

The solution is to use a pipe:

        #! /bin/sh

        ssh host cat directory/file | compress > file.Z

This uses ssh instead of scp, but that should be O.K.
in most cases.  If you want to use scp, I'm not sure if
it has the ability to send its output to its standard
output, so you might have to trick it, like this:

        #! /bin/sh

        scp host:directory/file /dev/fd/1 | compress > file.Z

/dev/fd/1 is, for any program, file #1 that *that* program
has open, so /dev/fd/1 is usually standard output.

Note that I've used "/bin/sh" and not "/bin/ksh" because
nothing that either you or I did requires ksh-specific
features, so you might as well make it portable.  (Assuming
you care, which you might not, but it is good practice.)

  - Logan
my  your   his  her   our   their   *its*
I'm you're he's she's we're they're *it's*


scp/compress on fly

Post by Philip Bro » Mon, 12 Mar 2001 07:15:51


>The solution is to use a pipe:

>    #! /bin/sh

>    ssh host cat directory/file | compress > file.Z

it would be more efficient on the network to do

ssh host compress -c directory/file >file.Z

[note that the ">" gets interpreted by the LOCAL shell, unless you quote
 it to send to the remote machine]

[Trim the no-bots from my address to reply to me by email!]
[ Do NOT email-CC me on posts. Pick one or the other.]

The word of the day is mispergitude


1. compressing on the fly: father process seems to end before popened compress

Hi all.

Got this weird problem. Program A starts a child process (no forking)
with popen to compress data written to disk on the fly. This seems to
work ok. The thing is that immediately after program A terminates, an
invoking script moves the new file to another location. It turns out
that the file gets truncated after it is moved. I get the sense that
compress didn't finish before the file was moved. I'm doing a pclose
on the pipe of course so this is really strange. I'm flushing every
write too, but to no effect.
The strangest thing is that this doesn't happen always. Sometimes
moved files are chopped sometimes they arent. Probably due to system
load. If I dont call mv in the script the whole process works just
fine, files are always created ok. Also if I introduce a "sleep 1"
before de mv in the script, no files get truncated. Of course I dont
like this workaround. Would rather have prog_a stop execution till
compress is completely finished. But I thought pclose would arrange
for that.

Any ideas of what can be happening? Any of you guys ever walked into a
similar behavior? Im working on Digital Unix if that matters for

This is an excerpt of my code:

*** ***
prog_a newfile.Z
mv newfile.Z /another_fs/another_dir/newfile.Z

*** progr_a ***
fout = popen("/usr/bin/compress - > newfile.Z", "w");
if (fout==NULL){ perror("popen didn't work"); exit(1); }

/* Write data and flush every time */
   fprintf(fout, ...);

if( pclose(fout)!= 0){
   /* never gets in here so pclose works fine */

Thanks in advance for any help you can provide.

Fernando Hevia
Telecom Personal
Cordoba, Argentina

2. Bug in NIS/SENDMAIL/Domains

3. On-the-Fly disk compress

4. "host to network long long"

5. TAR/compress on-the-fly?

6. repartition linux native

7. Recording radio and mpeg compressing on the fly?

8. poppassd

9. GNU tar compress on the fly?

10. Compressing logfiles on the fly?

11. trying to compress sybase dump on-the-fly

12. on-the-fly file-compressing?

13. compress and ftp on the fly