Very Computer

All,

I'm working on spec'ing a new WWW proxy machine for Citicorp's
Internet gateway... with the goal of handling at least 500,000
WWW accesses per day, 20% of which should to come from the
proxy's cache.

We are currently running an HP e35 w/ 64 Mb RAM and a combination
of CERN httpd (http, gopher, ftp, wais proxy) and Netscape Proxy
(only proxies SSL at the moment).  With this proxy machine sitting
behind a Sun SS20 based firewall we currently service about 110,000
proxy accesses - 10% satisfied from a 200Mb cache.  Load average
on our proxy server is typically in the 7 - 12 range due to the
number of simultaneously active connections.

I am very interested in learning more about what you all have done
with your corporate gateways in terms of http proxies, firewall
mods, etc. in response to the growth in http demand from internal
users.  And, in particular, the performance and capacity you have
achieved with your solution.

I'll summarize responses.

-- John Brady
   Citicorp Internet Services

Quote:>All,

>I'm working on spec'ing a new WWW proxy machine for Citicorp's
>Internet gateway... with the goal of handling at least 500,000
>WWW accesses per day, 20% of which should to come from the
>proxy's cache.

>We are currently running an HP e35 w/ 64 Mb RAM and a combination
>of CERN httpd (http, gopher, ftp, wais proxy) and Netscape Proxy
>(only proxies SSL at the moment).  With this proxy machine sitting
>behind a Sun SS20 based firewall we currently service about 110,000
>proxy accesses - 10% satisfied from a 200Mb cache.  Load average
>on our proxy server is typically in the 7 - 12 range due to the
>number of simultaneously active connections.

We are running the Harvest cache on a Linux-1.3.77 server with 16 Mb
of RAM and 1 Gb. of disk space reserved to the cache. This machine is also
a generic login machine and handles (pop) mail for the users.

Load on the machine averages around 0.7. We get about 60000 accesses
a day, of which 30% can be satisfied from the cache.

We used the CERN proxy for some time, but as you can see from the
figures mentioned the Harvest cache is _far_ superiour. Also, the
CERN proxy used to hang or crash regulary - we had special perl
scripts installed to watch and test the server, and kill and restart
it if needed. All this is not nessecary with the Harvest server - it just runs.

The harvest cache also has other niceties, such as using other proxies
in a hierarchical system, bypassing other proxies if they are down, etc.

Mike.
--
  Miquel van    | Cistron Internet Services   --    Alphen aan den Rijn.

Perhaps you should increase the cache size to increase
the percentage of hits?

Quote:>I am very interested in learning more about what you all have done
>with your corporate gateways in terms of http proxies, firewall
>mods, etc. in response to the growth in http demand from internal
>users.  And, in particular, the performance and capacity you have
>achieved with your solution.

We're running an ISP here, with only 128 Kbps total bandwidth
to the Net, since it's very expensive.  We're using Harvest
to distribute the cache among variety of machines, totalling
4 GB of cache space.  All our downstream clients use their own
caches.

I don't have any stats handy right now, though.

--

: >With this proxy machine sitting behind a Sun SS20 based firewall we
: >currently service about 110,000 proxy accesses - 10% satisfied from
: >a 200Mb cache.  
:
: Perhaps you should increase the cache size to increase
: the percentage of hits?
:
: >I am very interested in learning more about what you all have done
: >with your corporate gateways in terms of http proxies, firewall
: >mods, etc. in response to the growth in http demand from internal
: >users.  And, in particular, the performance and capacity you have
: >achieved with your solution.
:
: We're running an ISP here, with only 128 Kbps total bandwidth
: to the Net, since it's very expensive.  We're using Harvest
: to distribute the cache among variety of machines, totalling
: 4 GB of cache space.  All our downstream clients use their own
: caches.

We have 19500 users sat behind 2 gateways here in Germany.

Both gateways are using the same setup to deal with the load.

We have a dual machine proxy setup due to certain departments still
using non-NIC addresses (you just have to love those guys :( ).

Anyway the first proxy that does the illegal -> NIC IP conversion is
a SS20/SS1000 (192-256MB Ram / 2GB SCSI2 / 1 SCSI Controller) running
NS-Proxy 1.12 with 175-250 processes running. We cache 1GB of the 2GB
disk. We don't let ns-gc run continually. It runs once a night with
the -update switch. The cache size grows to 2GB during the day but gets
knocked down to 1GB during the night. Works well as ns-gc *kills*
performance during the day.

The second proxy (external) that does the retrieval of the URLS is a
SGI Challenge DM (2-4 R4400 200MHz / 256MB Ram / 4x2GB Fast-Wide Disks /
4 x SCSI2 Controllers). We have setup the system on disk 1 and a 6GB
stripped filesystem on the other 3 disks running the xfs filesystem.
This thing is beastial with contstant read/writes of ~14MB/Sec which
really helps with the large cache. We run NS-Proxy 1.12 with 300
processes running. We cache 4GB of the 6GB and let it crawl upto 5.5GB
during the day and then knock it back down to 4GB in the night.

We have a 32-40% cache hit on the ~350-400K daily requests.

--
Iain Lea                                            DS 83, SBS GmbH., Germany

We're a small ISP with about 120 28.8 modems - what recomendations can
you make about the proxy cache size, machine needed, etc? I've been
testing the Netscape Proxy server and it seems very good. But how much
cache and machine will I need to actually support about 100-150 modem
users?

Any help appreciated.

-Mark

Mark Mayo