script for Gopher file retrieval

script for Gopher file retrieval

Post by Matthew Wayne Kr » Sun, 08 May 1994 02:02:04



I need help writing a c-shell script which runs telnet interactively to
retrieve files from a gopher site.  Pretty basic: I want to telnet
gopher-site.univ.edu 70, then issue a file path such as
0/.dirs/INSEO/AOFNWSXX/Apts, and concatenate >> the results to a file.

I'll have an input file which contains a list of each gopher-file-path that I
want to retrieve, one entry per line.  I understand how the gopher-retrival
works, but not how to run telnet interactively to retrive each file.

This all came about because I can't find a gopher client which can save all
files from a given gopher directory into one concatenated file.  This is
especially useful for gopher sites which maintain *listings* on gopher (my
alternative right now is to save each file, quit gopher, cat * > file, pretty
messy).

Thanks,
Matt Krom

 
 
 

script for Gopher file retrieval

Post by Stephen D. Holla » Sun, 08 May 1994 07:20:44



Quote:>I need help writing a c-shell script which runs telnet interactively to
>retrieve files from a gopher site.  Pretty basic: I want to telnet
>gopher-site.univ.edu 70, then issue a file path such as
>0/.dirs/INSEO/AOFNWSXX/Apts, and concatenate >> the results to a file.

What you'll need to do is write a script (maybe shell, maybe perl)
which for each of the specified files does effectively the following:
        echo >temporary_file <filepath>
        telnet gopher-site.univ.edu 70 <temporary_file >>output_file

That's all there is to it!
        -Steve
--
-------------------------------------------
Steve Holland N1RAO    "Vinum in Veritate"

-------------------------------------------

 
 
 

script for Gopher file retrieval

Post by Adrian V Maria » Sun, 08 May 1994 08:29:47




>>I need help writing a c-shell script which runs telnet interactively to
>>retrieve files from a gopher site.  Pretty basic: I want to telnet
>>gopher-site.univ.edu 70, then issue a file path such as
>>0/.dirs/INSEO/AOFNWSXX/Apts, and concatenate >> the results to a file.
>What you'll need to do is write a script (maybe shell, maybe perl)
>which for each of the specified files does effectively the following:
>    echo >temporary_file <filepath>
>    telnet gopher-site.univ.edu 70 <temporary_file >>output_file
>That's all there is to it!

Did you try this?  

I don't think you can use I/O redirection with telnet.  You need to
use 'expect' to drive the telnet process.

 
 
 

script for Gopher file retrieval

Post by Stephen D. Holla » Sun, 08 May 1994 11:19:58



Quote:>I don't think you can use I/O redirection with telnet.  You need to
>use 'expect' to drive the telnet process.

You're right. Telnet does not seem to support input redirection,
so you would have to use something such as expect. In any case, all
you have to do is wait until you connect and then transmit the
selector string followed by return.
        -Steve

--
-------------------------------------------
Steve Holland N1RAO    "Vinum in Veritate"

-------------------------------------------

 
 
 

script for Gopher file retrieval

Post by Sergio Gela » Mon, 09 May 1994 05:52:59



Quote:>I need help writing a c-shell script which runs telnet interactively to
>retrieve files from a gopher site.  Pretty basic: I want to telnet
>gopher-site.univ.edu 70, then issue a file path such as
>0/.dirs/INSEO/AOFNWSXX/Apts, and concatenate >> the results to a file.

Forget about using the standard telnet command, and try the following
perl script (I wrote it in the last five minutes, so it may not be
perfect; but I did take the time to test it and found it to work).
Give it the address of a gopher server as the first argument
(optionally including a port number, as in gopher.cit.cornell.edu:7070),
and any number of paths to be retrieved as subsequent arguments.
As implemented, the script will take care of the terminating . line,
and replace <CRLF> with <LF> as the line separator.
You can modify it to read the paths from standard input if you wish;

#!/usr/local/bin/perl
#
# Usage: $0 host[:port] path [path...]
#
# Will establish a connection to the named host (the port number is taken
# to be 70 unless specified otherwise) to retrieve each of the paths given
# on the command line. The returned data are written to standard output.
# A new connection is made for each path.
# Error messages are written to standard error.
#
$host = shift;
if ($host =~ /:(\d+)$/) {
    $port = $1;
    $host = $`;

Quote:} else {
    $port = 70;
}

if ($host =~ /^(\d+)\.(\d+)\.(\d+)\.(\d+)$/) {
    $serveraddr = pack('C4', $1, $2, $3, $4);
Quote:} else {



        die "gethostbyname($host): $!\n";
    }
    $serveraddr = $x[$[+4];
Quote:}

# Form the destination port address.
$portaddr = pack('S n a4 x8', 2, $port, $serveraddr);
$thisaddr = pack('S n a4 x8', 2, 0, "\0\0\0\0");


    unless (socket(S,2,1,6)) {
        die "socket(): $!\n";
    }
    unless (bind(S, $thisaddr)) {
        die "bind(): $!\n";
    }
    unless (connect(S, $portaddr)) {
        warn "connect(): $!\n";
        die "Could not retrieve $path; aborting...\n";
        # One may want better error handling here (retries, etc.)
    }
    select((select(S),$|=1)[$[]); # Flush on every write to the socket.
    print S $path,"\n";               # Send the path down the socket.
    while (<S>) {
        last if ($_ eq ".\r\n"); # End of data marker.
        s/^\.//;                # Strip leading period.
        s/\r\n$/\n/;            # Turn <CRLF> into <LF>.
        print; }        # Copy the data.
    close (S);                  # Done.

Quote:}

exit 0;
__END__
--

 
 
 

script for Gopher file retrieval

Post by Adrian V Maria » Mon, 09 May 1994 07:37:26




>>I need help writing a c-shell script which runs telnet interactively to
>>retrieve files from a gopher site.  Pretty basic: I want to telnet
>>gopher-site.univ.edu 70, then issue a file path such as
>>0/.dirs/INSEO/AOFNWSXX/Apts, and concatenate >> the results to a file.
>Forget about using the standard telnet command, and try the following
>perl script

If you want to reinvent the wheel, go ahead.  

Here's an expect script which spawns a telnet process.  Note that
this requires somewhat less work that rewriting telnet in perl.

#!/usr/local/bin/expect
log_user 0
spawn telnet gopher.cit.cornell.edu 70
expect "Escape character"
log_file -a output
send "0/.files/WEATHS\r"
expect eof

 
 
 

script for Gopher file retrieval

Post by Karl J. Run » Mon, 09 May 1994 14:35:30


There is also a client that comes with the standard gopher distribution
that is called "gophfilt" (for gopher-filter) that I have found useful.

It goes something like

gophfilt -p "0/path/to/the/file" -h gopher-site.univ.edu

The result is sent to stdout, so you could append to a file with >>.
There are a few other features that escape my memory at the moment...

Karl Runge

 
 
 

script for Gopher file retrieval

Post by Malcolm Murp » Tue, 10 May 1994 22:24:33


:> >I don't think you can use I/O redirection with telnet.  You need to
:> >use 'expect' to drive the telnet process.
:> You're right. Telnet does not seem to support input redirection,
:> so you would have to use something such as expect. In any case, all

Not necessarily, I seem to remember doing this kind of redirection:

( echo command1 ; sleep 5 ; echo command2 ; sleep5 ; ... ) |telnet host

but that was on a port that didn't require a username.  The idea does go
through though, I think.

--------------------------------------------------------------------------

--------------------------------------------------------------------------
"By the cold and religious we were taken in hand.  Shown how to feel good,
and told to feel bad"             --  from The Final Cut, Pink Floyd

 
 
 

1. Gopher & Gopher Server

First of all, I am unable to run gopher from my Linux machine.
Does gopher come in one of the Slackware packages? I have installed A,
AP, F, N, and D. I have no gopher.

Secondly, how can I set up a gopher server? I seem to be completely
missing all of the gopher files.

                Dan Kamins

2. ttyps not reseting

3. need help with a script to call Gopher

4. contest v0.30

5. UNIX to DOS File Retrieval

6. REPOST patch 17/38: CLIENT: move nfs_async_handle_jukebox() into ->unlink_done()

7. Copy files using filenames from text files with shell script or bash script

8. Web Mail on Linux

9. Programming file retrieval and processing

10. automate retrieval of a file via ftp by a specific date

11. WGET - multiple file retrieval

12. GOPHER PROBLEMS: Cant run script....

13. mail bound file retrieval software