Board index Unix Programmer

howto? open, write, read, close an HTTP connection?

howto? open, write, read, close an HTTP connection?

Post by Pete Wils » Sat, 14 Jun 2003 08:13:15



Somewhat OT, sorry. And also somewhat vague: I'm not quite sure of the
right question. So far, RFC2616 has been too abstract for this
question but maybe the answer's implied in there somewhere.

On my Linux server, I need to do stuff that a browser normally does,
to /be/ a browser in a limited way and for a time.

From my code in, say, cgi-bin on my server I need to:

Open a connection to some named web resource, send a GET HTTP header,
receive the the web page or error report from that web resource,
process the HTML page I've received, and then close the HTTP
connection.

Is there an analog of the socket library that'll let me do all that,
but on an HTTP connection? If so, what is the analog? If the analog is
a bunch of C code I have to write, what docs to I have to look at? Or
does the socket library have a protocol choice == HTTP (would be
great)?

Thanks!
--
Pete Wilson

 
 
 
Top

howto? open, write, read, close an HTTP connection?

Post by Barry Margoli » Sat, 14 Jun 2003 08:43:22




>Is there an analog of the socket library that'll let me do all that,
>but on an HTTP connection? If so, what is the analog?

Search for "libhttp".

--

Level(3), Woburn, MA
*** DON'T SEND TECHNICAL QUESTIONS DIRECTLY TO ME, post them to newsgroups.
Please DON'T copy followups to me -- I'll assume it wasn't posted to the group.

 
 
 
Top

howto? open, write, read, close an HTTP connection?

Post by Mohun Biswa » Sat, 14 Jun 2003 10:50:13





>>Is there an analog of the socket library that'll let me do all that,
>>but on an HTTP connection? If so, what is the analog?

> Search for "libhttp".

Or libcurl (http://curl.haxx.se/). Quite nice, utterly free license.

MB

 
 
 
Top

howto? open, write, read, close an HTTP connection?

Post by Frank Schmit » Wed, 18 Jun 2003 16:26:28



> Somewhat OT, sorry. And also somewhat vague: I'm not quite sure of the
> right question. So far, RFC2616 has been too abstract for this
> question but maybe the answer's implied in there somewhere.

> On my Linux server, I need to do stuff that a browser normally does,
> to /be/ a browser in a limited way and for a time.

> From my code in, say, cgi-bin on my server I need to:

> Open a connection to some named web resource, send a GET HTTP header,
> receive the the web page or error report from that web resource,
> process the HTML page I've received, and then close the HTTP
> connection.

If using script languages is an option, you might want to use Ruby:

require "net/http"

site = "www.slashdot.org"
h = Net::HTTP.new(site,80)
resp, data = h.get("/",nil)
# just write what we got to stdout
data.split.each {|line| puts line}

HTH & kind regards
frank

--
Frank Schmitt
4SC AG          phone: +49 89 700763-0
                e-mail: frank DOT schmitt AT 4sc DOT com

 
 
 
Top

1. howto? open, write, read, close an HTTP connection?

You should know better, Pete :).  Best practice is to crosspost and set a
followup, that way discussion doesn't get needlessly duplicated.  I've set
a followup on this message, but I don't read that group, sorry :)

Okay, so you are writing an HTTP client/HTML UA.

I'm not sure I'm following you; the socket library is at a lower level than
HTTP.  You open a TCP connection to port 80 on the server with the sockets,
and read/write at will.  If I remember correctly, when using BSD sockets,
you pick the INET family and TCP.  Any decent UNIX programming book should
include a bit about sockets.

I'm sure you are looking for code examples, the only one which springs to
mind is libwww, which was bundled with Amaya, and which I haven't looked
at.  Presumably, being worked on at W3C, it's a decent implementation.

Bear in mind that most people prefer to work with higher-level languages
when dealing with higher-level protocols like HTTP.  Whilst your code might
be ultra-efficient when written in C, the time spent implementing it might
be better used elsewhere.

Also, if you are using Linux, consider farming off the HTTP client bit to
wget or curl; at least for the prototype.  If you are worried about the
overhead of starting a new process, I believe most of curl's features are
implemented as part of 'libcurl', which you can link into your executable.

I know perl has adequate HTML parsing routines, but in case your language of
choice does not, consider running the resulting HTML document through tidy
to convert it to xhtml, and just using an xml parser on the end product
(or, of course, the standard grep/sed/awk combinations if you aren't too
fussy about it being a conformant HTML UA).

--
Jim Dabell

2. RH6 -> KDE -> Kvirc ;)

3. can a socket be closed for reading but open for writing?

4. nntpd setup

5. Connection: close HTTP header sent with every request

6. keyboard doesn't work - help!

7. HTTP connections do not close

8. gs, jpeg funny on my ET4000

9. How to open, lock, write and close a file in ksh

10. Apache 2.0 howto write my own http-header

11. Linux sockets problem (closed connections staying open?)

12. Writing a web server. Connection close problem.

13. Close FIFO and open it again to read the remaining data?


All times are UTC
Board index