RPC Challenge -- pipelining

RPC Challenge -- pipelining

Post by Zhe » Sun, 19 Jun 1994 06:10:55



Hi Gurus:

I am in the process of designing an RPC procedure which scans a file
with typical size of 200MB+ upon client requests. One design is to let
the procedure returns 8K result data per-call (similar to NFS).
Because of the synchronous RPC nature, the client has to do something
like the following pseudo code:

        char    result[8192];
        while ((result = rpc_call(fileName, thisPageNum, ...)) != EOF)
        {
                result = rpc_call(fileName, nextPageNum++, ...);
        }

Each iteration has to send a request, wait for the 8K result sent back
across the network, then send the next request (two-way msg
delays). Significant sequential delays are introduced if the client
wants to scan the whole file.

What I really want to have is for the client to make a single RPC call
and get all 200MB+ data back. If using sockets, the client can send a
request, then repeatedly read from the connected TCP socket. This way
only one request is needed for 200MB+ data.

Any hints on how I could achieve the same thing using RPC? Will
threaded clients or servers help? (maybe 200MB/8K = 25K threads
needed?)

Regards,

/Jay Li
Dept. of Computer Science
Columbia University


are enough interests.

 
 
 

RPC Challenge -- pipelining

Post by Vernon Schryv » Mon, 20 Jun 1994 01:04:24



> ...
>I am in the process of designing an RPC procedure which scans a file
>with typical size of 200MB+ upon client requests. One design is to let
>the procedure returns 8K result data per-call (similar to NFS).
> ...
>What I really want to have is for the client to make a single RPC call
>and get all 200MB+ data back. If using sockets, the client can send a
>request, then repeatedly read from the connected TCP socket. This way
>only one request is needed for 200MB+ data.
> ...

Without knowing which of the many flavors of remote procedure call are
being used, it's hard to guess the constraints.  Sun's RPC allows the
use of "batching" over TCP.  I don't remember what the per-call data
limit is, but I thought it was well over 8KByte.

This seems like the wrong way to attack such a problem.  Unless the
network is at least as fast as FDDI, the network will (or should be)
the bottleneck.  Sending 200MByte over an Ethernet will take at least
3-4 minutes no matter what optimizations are used.  Instead, why not
define a remote procedure that takes as arguments whatever is being
scanned for, and does the scanning on the server?

Besides being much faster and saving network bandwidth, this kind of
approach will probably save the CPU cycles on the server.  The server
will probably spend fewer cycles sifting the data than sending it,
assuming the scanning is simple.  Of course, one could have the server
do the simplest part of the scanning, and let the client do any really
fancy pattern matching.



 
 
 

RPC Challenge -- pipelining

Post by Lance Viss » Mon, 20 Jun 1994 05:01:05



+>I am in the process of designing an RPC procedure which scans a file
+>with typical size of 200MB+ upon client requests. One design is to let
+>the procedure returns 8K result data per-call (similar to NFS).
+>Because of the synchronous RPC nature, the client has to do something
+>like the following pseudo code:

        Just as a suggestion, if your dealing with this size file,
its better to make a general scanner the RPC call and not transfer
the data at all.
        The RPC call could pass a regular expression as a parameter
to some sort of scanning server on the host where the file lives.
This approach will be faster and take fewer cycles on both
client & server than scanning locally.

        The only case where this would not work is if there is a huge
amount of local state needed to scan the file or you wanted a huge
percentage of the file moved to the local node anyway.

 
 
 

1. splitting (teeing) a pipeline into two further pipelines

Is there an easy way to split a pipeline, as by `tee', but where each
resulting output stream goes into its own pipeline, rather than one
going to a file?  I ended up writing a simple program that does this
(a more general tee, you could say).  If it would be useful, I can
post it.
 - David B Rosen  --  Cognitive and Neural Systems,  Boston University

2. GZIP required!!

3. How to use rpc.portmap, rpc.mountd, rpc.nfsd?

4. Where are the new modules??!!

5. Where are rpc.portmap, rpc.mountd, rpc.nfsd?

6. Need help configuring second network access

7. Dangers of rpc.portmap, rpc.mountd, rpc.nfsd

8. IDA sendmail under Solaris 2.2

9. RPC 4.0 vs RPC 4.1 vs TI RPC

10. Comparison bet soap rpc, dce rpc and sun rpc

11. UNIX RPC client, NT RPC server

12. RPC: Port mapper failure - RPC: Unable to receive

13. RPC.LOCKD cannot contact RPC.STATD. HELP!!