Copy data from one descriptor to another?

Copy data from one descriptor to another?

Post by Michael B Alle » Sat, 14 Jun 2003 16:47:07



Is there a standard function for copying data from one descriptor to
another? Meaning read from one and write that data to the other in a
loop for n bytes?

Thanks,
Mike

 
 
 

Copy data from one descriptor to another?

Post by David Schwart » Sat, 14 Jun 2003 19:30:41




Quote:> Is there a standard function for copying data from one descriptor to
> another? Meaning read from one and write that data to the other in a
> loop for n bytes?

    You have to 'read' from one and 'write' to the other unless 'sendfile'
fits the bill and exists on the platform you're using. I should note that
there's really no reason to do this if the source descriptor is a file,
using 'mmap' and writing from the 'mmap'ed region should avoid the copy,
which I presume is what you're trying to do.

    Think about how you'd write such a function, assuming both descriptors
are sockets for the moment. What if the 'read' generates an error? What if
the 'write' does? What if you read 10 bytes but couldn't write any more than
3 of them? Or would it always be blocking? But then if you read from a
socket, it could block forever.

    So it's not really possible to have one nice, clean, general version of
a function that does this. So code what makes sense for you.

    DS

 
 
 

Copy data from one descriptor to another?

Post by Marc Rochkin » Sat, 14 Jun 2003 23:53:03




[snip]

Quote:> You have to 'read' from one and 'write' to the other unless 'sendfile'
> fits the bill and exists on the platform you're using. I should note that
> there's really no reason to do this if the source descriptor is a file,
> using 'mmap' and writing from the 'mmap'ed region should avoid the copy,
> which I presume is what you're trying to do.

I can see how this would avoid some of the code in the app to do the copy,
but can't see how it would avoid the copy itself. The data from the source
file still has to go from disk to user memory, and then back to disk in the
destination file. I'm sure what you meant is that, given a decent
implementation, this approach to copying is more efficient than one that
uses a normal read. (But, I would like to see some numbers to prove it.)

[snip]

Quote:

> So it's not really possible to have one nice, clean, general version of
> a function that does this. So code what makes sense for you.

It's very difficult to prove a negative statement... I would bet that a
suitable funciton could be designed, but, as the operation is relatively
uncommon, there isn't much need for such a function.

As an example of how to package up some functions that are often used
together, while still preserving most of the needed flexibility, one could
look at posix_spawn.

--Marc

 
 
 

Copy data from one descriptor to another?

Post by Barry Margoli » Sun, 15 Jun 2003 00:38:37






>[snip]

>> You have to 'read' from one and 'write' to the other unless 'sendfile'
>> fits the bill and exists on the platform you're using. I should note that
>> there's really no reason to do this if the source descriptor is a file,
>> using 'mmap' and writing from the 'mmap'ed region should avoid the copy,
>> which I presume is what you're trying to do.

>I can see how this would avoid some of the code in the app to do the copy,
>but can't see how it would avoid the copy itself. The data from the source
>file still has to go from disk to user memory, and then back to disk in the
>destination file. I'm sure what you meant is that, given a decent
>implementation, this approach to copying is more efficient than one that
>uses a normal read. (But, I would like to see some numbers to prove it.)

If you use read(), it copies from the file to a kernel buffer and then from
the kernel buffer to the user memory.  If you use mmap(), the kernel buffer
is bypassed, because the user memory is attached directly to the file using
the virtual memory mechanism.

In either case, a copy is performed during write().  So the difference is
that a read/write loop results in three copies per iteration, while an
mmap/write loop results in only two copies per iteration.

--

Level(3), Woburn, MA
*** DON'T SEND TECHNICAL QUESTIONS DIRECTLY TO ME, post them to newsgroups.
Please DON'T copy followups to me -- I'll assume it wasn't posted to the group.

 
 
 

Copy data from one descriptor to another?

Post by Dragan Cvetkovi » Sun, 15 Jun 2003 01:04:43



> As an example of how to package up some functions that are often used
> together, while still preserving most of the needed flexibility, one could
> look at posix_spawn.

Posix_spawn? What's that?

Bye, Dragan

--
Dragan Cvetkovic,

To be or not to be is true. G. Boole      No it isn't.  L. E. J. Brouwer

 
 
 

Copy data from one descriptor to another?

Post by Marc Rochkin » Sun, 15 Jun 2003 02:20:33


On 13 Jun 2003 12:04:43 -0400, Dragan Cvetkovic



>> As an example of how to package up some functions that are often used
>> together, while still preserving most of the needed flexibility, one
>> could
>> look at posix_spawn.

> Posix_spawn? What's that?

> Bye, Dragan

A new function added around 1999, part of Advanced Realtime.

--Marc

 
 
 

Copy data from one descriptor to another?

Post by David Schwart » Sun, 15 Jun 2003 03:47:56



Quote:> I can see how this would avoid some of the code in the app to do the copy,
> but can't see how it would avoid the copy itself. The data from the source
> file still has to go from disk to user memory, and then back to disk in
the
> destination file. I'm sure what you meant is that, given a decent
> implementation, this approach to copying is more efficient than one that
> uses a normal read. (But, I would like to see some numbers to prove it.)

    Take a look at how 'cp' works. If you want to duplicate a file, for
example, mmap'ing both files and 'memcpy'ing right from one to the other is
faster than reading into a user-space buffer and then writing from the
user-space buffer.

    The idea is not so much to eliminate the copy in the app, but to
eliminate the copy from kernel-space to user-space itself. Copying data
across the kernel/user boundary is not cheap.

    Think about writing to a socket from an mmap'ed file. The data never
goes to user-space. So no data crosses the user/kernel boundary instead of
all the data twice.

    DS

 
 
 

Copy data from one descriptor to another?

Post by Dragan Cvetkovi » Sun, 15 Jun 2003 05:01:06



> > Posix_spawn? What's that?

> > Bye, Dragan

> A new function added around 1999, part of Advanced Realtime.

> --Marc

Indeed. Thanks Marc. Here is its prototype:

#include <spawn.h>

int posix_spawn(pid_t *restrict pid, const char *restrict path,
                const posix_spawn_file_actions_t *file_actions,
                const posix_spawnattr_t *restrict attrp,
                char *const argv[restrict], char *const envp[restrict]);

Bye, Dragan

--
Dragan Cvetkovic,

To be or not to be is true. G. Boole      No it isn't.  L. E. J. Brouwer

 
 
 

Copy data from one descriptor to another?

Post by Michael B Alle » Sun, 15 Jun 2003 05:15:26





>> Is there a standard function for copying data from one descriptor to
>> another? Meaning read from one and write that data to the other in a
>> loop for n bytes?

> You have to 'read' from one and 'write' to the other unless 'sendfile'
> fits the bill and exists on the platform you're using. I should note

Well I'm going in the other direction so I would need a 'recvfile'.

Quote:> that there's really no reason to do this if the source descriptor is a
> file, using 'mmap' and writing from the 'mmap'ed region should avoid the
> copy, which I presume is what you're trying to do.

This is a good idea. The arrangement is that a socket client is writing
to a file but I suspect mmap might still help.

Quote:> Think about how you'd write such a function, assuming both descriptors
> are sockets for the moment. What if the 'read' generates an error? What
> if the 'write' does? What if you read 10 bytes but couldn't write any
> more than 3 of them? Or would it always be blocking? But then if you
> read from a socket, it could block forever.

Exactly. That's why I wanted to find out if there's a standard function
for this operation.

Quote:> So it's not really possible to have one nice, clean, general version of
> a function that does this. So code what makes sense for you.

Well I actually overted the need for this entirely for reasons not worth
going into. But not after writing the necessary code (inlined below
for future reference perhaps).

Thanks,
Mike

static ssize_t
writen(int fd, const void *src, size_t n)
{
        size_t nleft;
        ssize_t nwritten;
        const char *ptr;

        ptr = src;
        nleft = n;
        while (nleft > 0) {
                if ((nwritten = write(fd, ptr, nleft)) < 0) {
                        return nwritten;
                }
                nleft -= nwritten;
                ptr += nwritten;
        }
        return n;

Quote:}

...
        ssize_t tot, n;
        tot = <amount to copy>
        while (tot > 0) {
                unsigned char buf[PAGE_SIZE];

                n = MIN(PAGE_SIZE, tot);
                if ((n = read(infd, buf, n)) == -1 ||
                                        writen(outfd, buf, n) == -1) {
                        return -1;
                }
                tot -= n;
        }

 
 
 

1. sending data from one file descriptor to many

I'm writing an application, where I'll need to read from one socket and
distribute that data to many others.  Is there a built-in way to do
this?  I want to avoid reading into a buffer, then looping through the
descriptors.

Jon

Sent via Deja.com http://www.deja.com/
Share what you know. Learn what you don't.

2. mmap semantics clarification

3. copying data from one file into some other file with VI

4. Unix Programming FAQ (v1.34)

5. How ? One file <----> One file descriptor

6. gs printing every second page ?

7. How to copy data from serial port to a file (data logging) ?

8. cdrecord

9. How ? One file <----> One file descriptor

10. I want to not copy exclude one file when i Copy

11. copying tape of one it schemes to another one

12. ksh: Copying files one by one

13. Outputting to screen and log; copying file descriptors