Apache 2: poor performance on Solaris

Apache 2: poor performance on Solaris

Post by joe » Wed, 10 Mar 2004 00:45:12



Hello there

I installed apache 2 on a solaris 8 64bit machine with 1GB RAM and 500
MHz CPU. I am very disappointed off the performance of the web server.
While I am downloading binary data as PDF files in my intranet zone
(100MBit) I am getting an average speed of only 4Kb per sec. (only one
request at this time) Other applications on my machine as oracle or
samba is performing well, means I could transfer a few megabytes
within seconds.

I did also a recompile with the recommended settings mentioned here
http://httpd.apache.org/docs-2.0/misc/perf-tuning.html, however no big
improvements (only around 2 Kb/sec more) so far.

So I am wondering what I did wrong? How can this be? Have anybody out
there some good ideas what I have to verify of my system settings or
what I can do to detect the real bottleneck?

thank you in advance
regards
Mark

 
 
 

Apache 2: poor performance on Solaris

Post by Bruno Saverio Delbon » Wed, 10 Mar 2004 01:10:38



> (100MBit) I am getting an average speed of only 4Kb per sec. (only one
> request at this time)

I assume you're running apache2. I did some performance benchmarking and
found that apache2 compiled with SunPRO had a bit better RPS (1900+
Requests per second) {Optimization - -xO5} from ab than that compiled
with gcc 3.3.2 (1500+ Requests per second). On another test, I enabled
SNCA (http://www.hejazi.org/solaris/snca/) and found that the server
performance literally doubled with the apache 1.3.x

If you want to enable SNCA, read the above link and add this in your
apache2's apachectl:

NCAKMODCONF=/etc/nca/ncakmod.conf
if [ -f $NCAKMODCONF ]; then
         . $NCAKMODCONF
         if [ "x$status" = "xenabled" ]; then
                 HTTPD="env LD_PRELOAD=/usr/lib/ncad_addr.so $HTTPD"
         fi
fi

<right below ARGS="help" \fi line>

As a sidenote, I installed SunONE WebServer 6.1 SP1 on the same machine
and I got 70% boost vs. the speed of apache with RPS topping 4000+. So I
suggest:

a) Try to compile with SunPRO and/or use SNCA with apache2
b) Try to use the apache 1.3.x shipped with SNCA
c) Try to use SunONE WebServer 6.1SP1 (It's free for download with no
expiration) and see performance

Get back to us on what you did :)

HTH

--
Bruno Saverio Delbono
Systems Engineer
Open-Systems Group (not-profit)            http://www.open-systems.org/
Email: Bruno.S.Delbono at {mail.ac,open-systems.org,wf0.com,lucifer.at}

 
 
 

Apache 2: poor performance on Solaris

Post by James Carlso » Wed, 10 Mar 2004 05:08:57



> While I am downloading binary data as PDF files in my intranet zone
> (100MBit) I am getting an average speed of only 4Kb per sec. (only one

That's absurdly low.  I don't think that has anything to do with
Apache or any of the application-level performance tuning bits.

I think you need to look first at the hardware interfaces.  What
you're reporting is consistent with a confused Ethernet switch that
thinks the port is "half-duplex" when the Sun system has negotiated
"full-duplex."

--

Sun Microsystems / 1 Network Drive         71.234W   Vox +1 781 442 2084
MS UBUR02-212 / Burlington MA 01803-2757   42.497N   Fax +1 781 442 1677

 
 
 

Apache 2: poor performance on Solaris

Post by joe » Wed, 10 Mar 2004 17:12:06



> That's absurdly low.  I don't think that has anything to do with
> Apache or any of the application-level performance tuning bits.

> I think you need to look first at the hardware interfaces.  What
> you're reporting is consistent with a confused Ethernet switch that
> thinks the port is "half-duplex" when the Sun system has negotiated
> "full-duplex."

You miss my point that other network applications as oracle or samba
is performing well. This means I could i.e. with Samba transfer 10MB
file within
5 seconds. Also If I am using Netscape on Solaris to download from
somewhere software I have no such performance problems as the current
installed apache. So it can't be so near at the ethernet itself.

any other suggestions ?

regards
Mark

 
 
 

Apache 2: poor performance on Solaris

Post by Michael Laajane » Wed, 10 Mar 2004 17:24:24


HI,


>>That's absurdly low.  I don't think that has anything to do with
>>Apache or any of the application-level performance tuning bits.

>>I think you need to look first at the hardware interfaces.  What
>>you're reporting is consistent with a confused Ethernet switch that
>>thinks the port is "half-duplex" when the Sun system has negotiated
>>"full-duplex."

> You miss my point that other network applications as oracle or samba
> is performing well. This means I could i.e. with Samba transfer 10MB
> file within
> 5 seconds. Also If I am using Netscape on Solaris to download from
> somewhere software I have no such performance problems as the current
> installed apache. So it can't be so near at the ethernet itself.

> any other suggestions ?

Two things you could try.

1. Try a prebuilt apache to see if you compiled it in a strange way.

2. How have you configurated http.conf? Do you have a DNS entry for the
http server and so?

/michael

 
 
 

Apache 2: poor performance on Solaris

Post by James Carlso » Wed, 10 Mar 2004 21:04:55



> You miss my point that other network applications as oracle or samba
> is performing well. This means I could i.e. with Samba transfer 10MB
> file within
> 5 seconds.

What I didn't understand was that this was on the same network -- you
seemed to go out of your way to say that apache was serving an
"intranet."  My error.

Are those other successful transfers of bulk data occurring in the
same direction?  It still could be a bad interface.

Quote:> Also If I am using Netscape on Solaris to download from
> somewhere software I have no such performance problems as the current
> installed apache. So it can't be so near at the ethernet itself.

> any other suggestions ?

Another poster suggested looking at DNS.  Apache uses reverse
resolutions to log accesses, and if your in-addr.arpa records are
missing, that'd cause some long delays.  Unless the application
measuring throughput is confused, though, those delays ought not be
counted as part of the transfer.

The next step I'd suggest would be gathering traces with snoop or
ethereal to find out what's going on.  I'm hard pressed to imagine
much at the application level that could account for performance that
low.

--

Sun Microsystems / 1 Network Drive         71.234W   Vox +1 781 442 2084
MS UBUR02-212 / Burlington MA 01803-2757   42.497N   Fax +1 781 442 1677

 
 
 

Apache 2: poor performance on Solaris

Post by Matt » Thu, 11 Mar 2004 04:28:22



> HI,


>>>That's absurdly low.  I don't think that has anything to do with
>>>Apache or any of the application-level performance tuning bits.

>>>I think you need to look first at the hardware interfaces.  What
>>>you're reporting is consistent with a confused Ethernet switch that
>>>thinks the port is "half-duplex" when the Sun system has negotiated
>>>"full-duplex."

>> You miss my point that other network applications as oracle or samba
>> is performing well. This means I could i.e. with Samba transfer 10MB
>> file within
>> 5 seconds. Also If I am using Netscape on Solaris to download from
>> somewhere software I have no such performance problems as the current
>> installed apache. So it can't be so near at the ethernet itself.

>> any other suggestions ?
> Two things you could try.

> 1. Try a prebuilt apache to see if you compiled it in a strange way.

> 2. How have you configurated http.conf? Do you have a DNS entry for the
> http server and so?

> /michael

What kind of throughput do you get with a large ftp transfer? Can
you post your httpd.conf? What flags did you use to compile Apache? Are
you using gcc or the Forte C compiler? What are you using to measure the
throughput? You might want to dump the traffic from the box that runs
Apache. This will allow you to see relative timestamps and MSS/MTU sizes.
In case the tool used to measure traffic throughput is confused (I have
seen broken tools in the past), you can calculate yourself. This will
also allow you to view window scaling issues and troubles with delayed
ACKs/packet loss/selective ACKs.
 
 
 

Apache 2: poor performance on Solaris

Post by joe » Thu, 11 Mar 2004 05:38:01




> > You miss my point that other network applications as oracle or samba
> > is performing well. This means I could i.e. with Samba transfer 10MB
> > file within
> > 5 seconds.
> Are those other successful transfers of bulk data occurring in the
> same direction?  It still could be a bad interface.

You have been right and you are my king for today! If I change the
direction of the bulk transfer (i.e. for samba) then I have a very
slow network performance and timeouts. puhhh never mind that the
interface could have a problem in only  one direction.

But now I have no idea how to solve that problem, because the network
card is on the motherboard itself and I tried 3 different switches
with no success, still the same behaviour.

So any ideas how to see if it is the driver a system setting or the
hardware itself ?

thank you all in advance
Mark

 
 
 

Apache 2: poor performance on Solaris

Post by Juha Laih » Thu, 11 Mar 2004 06:02:01






>> > You miss my point that other network applications as oracle or samba
>> > is performing well. This means I could i.e. with Samba transfer 10MB
>> > file within
>> > 5 seconds.

>> Are those other successful transfers of bulk data occurring in the
>> same direction?  It still could be a bad interface.

>You have been right and you are my king for today! If I change the
>direction of the bulk transfer (i.e. for samba) then I have a very
>slow network performance and timeouts. puhhh never mind that the
>interface could have a problem in only  one direction.

>But now I have no idea how to solve that problem, because the network
>card is on the motherboard itself and I tried 3 different switches
>with no success, still the same behaviour.

What do you see with "netstat -i"? Interface errors?
Could it be a bad cable?

Also, check the duplex mismatch issue pointed out by someone else;
so, if the switch is a maanged one, look (or have the net amdin look)
what the switch says about the interface state, i.e. is it in full or
half duplex state, and check yourself what "ndd" tells about the
interface on the machine end. Command "ndd -get /dev/XXX link_mode"
will tell the duplex setting, 0 means half-duplex, 1 full-duplex.
If you're using a non-zero interface, the ndd sequence is more
tricky -- so f.ex. your network interface is qfe2. Then you need to
do as follows:
ndd -set /dev/qfe instance 2
ndd -get /dev/qfe link_mode

Use also "ndd -get /dev/XXX \?" to see what possible parameters
your XXX-type network interface supports. If you're changing the
allowed speed/duplex parameters, the new parameters take effect
only after you set the autonegotiation parameter (not needed to
change the value, if it's good value currently, but just "complete"
the setting by writing the autoneg value, and this will make the
system to activate the other changes).
--
Wolf  a.k.a.  Juha Laiho     Espoo, Finland

         PS(+) PE Y+ PGP(+) t- 5 !X R !tv b+ !DI D G e+ h---- r+++ y++++
"...cancel my subscription to the resurrection!" (Jim Morrison)

 
 
 

Apache 2: poor performance on Solaris

Post by Thomas Deh » Thu, 11 Mar 2004 07:35:05



> So any ideas how to see if it is the driver a system setting or the
> hardware itself ?

The first thing I'd do is make sure that
I have the current patch for the driver installed.

Thomas

 
 
 

Apache 2: poor performance on Solaris

Post by Casper H.S. Di » Thu, 11 Mar 2004 18:05:23



>You have been right and you are my king for today! If I change the
>direction of the bulk transfer (i.e. for samba) then I have a very
>slow network performance and timeouts. puhhh never mind that the
>interface could have a problem in only  one direction.
>But now I have no idea how to solve that problem, because the network
>card is on the motherboard itself and I tried 3 different switches
>with no success, still the same behaviour.
>So any ideas how to see if it is the driver a system setting or the
>hardware itself ?

It looks like a half/full duplex mismatch; are you running the test
between the same to systems all the time?  Did you replace the net
"end to end"?

Did you perhaps force full duplex on the Sun or on the switch?
What is the switch saying about negotiated speeds? And what does
the Sun say?

Casper
--
Expressed in this posting are my opinions.  They are in no way related
to opinions held by my employer, Sun Microsystems.
Statements on Sun products included here are not gospel and may
be fiction rather than truth.

 
 
 

Apache 2: poor performance on Solaris

Post by James Carlso » Fri, 12 Mar 2004 21:52:43



> So any ideas how to see if it is the driver a system setting or the
> hardware itself ?

I don't think it's a hardware problem.

Where to look depends on exactly which driver this is.  One common way
is to use "kstat <drivername>" to dump the statistics, and look for
"duplex."

Since you're having trouble transmitting but not receiving, I'll bet
that it's stuck in half-duplex mode.  There are lots of hits for
'duplex' on docs.sun.com.

--

Sun Microsystems / 1 Network Drive         71.234W   Vox +1 781 442 2084
MS UBUR02-212 / Burlington MA 01803-2757   42.497N   Fax +1 781 442 1677

 
 
 

Apache 2: poor performance on Solaris

Post by Jan Rasch » Mon, 15 Mar 2004 17:13:24


Hi Juha,

Thanks for the very interesting ndd part!!!

So I see my own server seems to use only 10MBit/s at the moment

ndd -get /dev/eri link_mode
0

But unfortunately I seem not able to set anything on my Sun:
# ndd -set /dev/eri link_mode 1
operation failed: Permission denied

:-)

# ndd -get /dev/eri \?
?                             (read only)
transceiver_inuse             (read only)
link_status                   (read only)
link_speed                    (read only)
link_mode                     (read only)
ipg1                          (read and write)
ipg2                          (read and write)
use_int_xcvr                  (read and write)
pace_size                     (read and write)
adv_autoneg_cap               (read and write)
adv_100T4_cap                 (read and write)
adv_100fdx_cap                (read and write)
adv_100hdx_cap                (read and write)
adv_10fdx_cap                 (read and write)
adv_10hdx_cap                 (read and write)
autoneg_cap                   (read only)
100T4_cap                     (read only)
100fdx_cap                    (read only)
100hdx_cap                    (read only)
10fdx_cap                     (read only)
10hdx_cap                     (read only)
lp_autoneg_cap                (read only)
lp_100T4_cap                  (read only)
lp_100fdx_cap                 (read only)
lp_100hdx_cap                 (read only)
lp_10fdx_cap                  (read only)
lp_10hdx_cap                  (read only)
instance                      (read and write)
lance_mode                    (read and write)
ipg0                          (read and write)
intr_blank_time               (read and write)
intr_blank_packets            (read and write)

Thank you in advance!

Jan


> Also, check the duplex mismatch issue pointed out by someone else;
> so, if the switch is a maanged one, look (or have the net amdin look)
> what the switch says about the interface state, i.e. is it in full or
> half duplex state, and check yourself what "ndd" tells about the
> interface on the machine end. Command "ndd -get /dev/XXX link_mode"
> will tell the duplex setting, 0 means half-duplex, 1 full-duplex.
> If you're using a non-zero interface, the ndd sequence is more
> tricky -- so f.ex. your network interface is qfe2. Then you need to
> do as follows:
> ndd -set /dev/qfe instance 2
> ndd -get /dev/qfe link_mode

> Use also "ndd -get /dev/XXX \?" to see what possible parameters
> your XXX-type network interface supports. If you're changing the
> allowed speed/duplex parameters, the new parameters take effect
> only after you set the autonegotiation parameter (not needed to
> change the value, if it's good value currently, but just "complete"
> the setting by writing the autoneg value, and this will make the
> system to activate the other changes).

 
 
 

Apache 2: poor performance on Solaris

Post by Juha Laih » Mon, 15 Mar 2004 21:02:00



Quote:>Thanks for the very interesting ndd part!!!

>So I see my own server seems to use only 10MBit/s at the moment

>ndd -get /dev/eri link_mode
>0

link_mode tells the link is in half-duplex mode, but says nothing
about the speed, the speed you'd get by querying link_speed.

Quote:>But unfortunately I seem not able to set anything on my Sun:
># ndd -set /dev/eri link_mode 1
>operation failed: Permission denied

Yes, the status variables are read-only, however they're somewhat
controlled by other, writable variables.

Quote:># ndd -get /dev/eri \?
...
>adv_autoneg_cap               (read and write)
>adv_100T4_cap                 (read and write)
>adv_100fdx_cap                (read and write)
>adv_100hdx_cap                (read and write)
>adv_10fdx_cap                 (read and write)
>adv_10hdx_cap                 (read and write)

...

So, there are the ones you can set per interface. "adv" as in advertise,
"cap" as in capability.  Default is to advertise all but 100T4. The
mode selection can be restricted by setting one or more of these to
zero. Note though, that just doing f.ex.
ndd -set /dev/eri adv_100hdx_cap 0
does not take effect until setting the adv_autoneg_cap. So, even if you
were happy of the current value of adv_autoneg_cap, you must write its
value to make effective the other changes.

Before making any changes, it's good to find out what capabilities are
advertised bu the link partner (the switch, that is). These you get by
querying:

Quote:>lp_autoneg_cap                (read only)
>lp_100T4_cap                  (read only)
>lp_100fdx_cap                 (read only)
>lp_100hdx_cap                 (read only)
>lp_10fdx_cap                  (read only)
>lp_10hdx_cap                  (read only)

... the significant being lp_autoneg_cap (i.e. does the link partner have
automatic link speed/mode negotiation enabled). If this is zero, the all
the others will be zero as well (and you must find whoever configured the
switch port to tell you what settings are forced there).

One more thing; the changes made with ndd are not persistent - the values
are only kept until next reboot. To make the changes permanent, you'll
need to edit /etc/system. Google for "/etc/system adv_100fdx_cap" to find
the proper syntax (also described in some books on docs.sun.com; something
may also be in manual page for "eri").

As I wrote in my earlier message, the issue is not to force full duplex;
the issue is to make parameters equal at both ends of the cable. Most
often the automatic negotiation handles this (if either end of the link
has negotiation disabled, then negotiation must be disabled on both ends
AND both ends must be forced to equal mode). When your hardware is less
than 5 years in age, the negotiation should work just fine; earlier than
that were not implementing the specifications so well - and apparently
even the specs were not that well settled then, so for older hardware
forcing the mode might be the only properly working option - but for
anything even remotely current disabling the negotiation typically
creates more problems than solves.

With negotiation, when a cable link becomes active (a machine is powered
up, or a able is connected), both ends start sending and listening for
special negotiation packets. These packets contain simple bit array about
modes supported at the sending end, so 100fdx, 100hdx, 10fdx and 10hdx.
When a machine configured for negotiation receives this kind of packet,
it looks through its own supported and enabled modes, and picks the first
in from the list in the above order. So, I can have a machine that is
configured for negotiation and 10Mbit/s only operation, and machine
that is configured for all the available modes, and these will negotiate
the link just fine -- the faster machine will see the negotiation packet
sent by the slower machine and will just choose the best mode supported
by the other end.

In a situation where one end of link is forced to some "desired" mode
and has negotiation disabled, but the other end of the link is trying
to negotiate (and has multiple allowed modes), the "flexible" end will
go into "hunt" mode (after timing out on waiting for the negotiation
packets), and will start going through the modes in order 10hdx, 10fdx,
100hdx, 100fdx. Sounds good, going from worst to best - especially
considering that traditionally the devices that have not been able to
negotiate have been the legacy systems only capable of half-duplex
communication - so this approach has ended up in both ends of the link
in matching modes. What happens today, however, is that some people,
remembering the old problems with negotiation, force devices to 100fdx
mode. If this is done, _and_ the other end of the link is trying to
negotiate from unrestricted set of parameters, things go wrong. The
device will try (and fail) first the 10Mbit/s modes, then it will try
the 100Mbit/s half-duplex mode. With this, when the link is not congested,
communication will succeed, even when the other end is configured for
full-duplex, so the device will settle for this, and will neven even
try the full-duplex mode. As a result of this, we have a link where one
end is in 100hdx mode, the other in 100fdx mode; the devices are
communicating, but performance problems arise immediately when the
link gets any significant amount of traffic.

So, with speed mismatch you get no traffic through - so on a link
that passes any traffic you're not having a speed mismatch. A duplex
mismatch will pass traffic through, and with small amounts of traffic
will mostly go unnoticed - but with large amounts of traffic it will
be almost non-functional (much worse than f.ex. a properly configured
half-duplex link).

Summary: Only disable negotiation when you're experiencing trouble that
you can reasonably show being a consequence of problems in negotiation.
(of course, there may be cases with f.ex. stubborn network admins who
only provide switch ports with negotiation disabled -- if this happens
to be your case, then your only remedy is to force the parameters on
the machine side as well).
--
Wolf  a.k.a.  Juha Laiho     Espoo, Finland

         PS(+) PE Y+ PGP(+) t- 5 !X R !tv b+ !DI D G e+ h---- r+++ y++++
"...cancel my subscription to the resurrection!" (Jim Morrison)

 
 
 

Apache 2: poor performance on Solaris

Post by Rick Jone » Wed, 17 Mar 2004 05:24:07


FWIW, here is a small table that may be of some concise use wrt
auto-negotiation on 100BT. The table might even be small enough to fit
on a sticky note afixed to one's monitor :)

How Autoneg is supposed to work:

When both sides of the link are set to autoneg, they will "negotiate"
the duplex setting and select full duplex if both sides can do
full-duplex.

If one side is hardcoded and not using autoneg, the autoneg process
will "fail" and the side trying to autoneg is required by spec to use
half-duplex mode.

If one side is using half-duplex, and the other is using full-duplex,
sorrow and woe is the usual result.

So, the following table shows what will happen given various settings
on each side:

                 Auto       Half       Full

   Auto        Happiness   Lucky      Sorrow

   Half        Lucky       Happiness  Sorrow

   Full        Sorrow      Sorrow     Happiness

Happiness means that there is a good shot of everything going well.
Lucky means that things will likely go well, but not because you did
anything correctly :) Sorrow means that there _will_ be a duplex
mis-match.

On the side running half-duplex you will see various errors and
probably a number of late collisions. On the side running full-duplex
you will see things like FCS/CRC errors.

rick jones
--
oxymoron n, Hummer H2 with California Save Our Coasts and Oceans plates
these opinions are mine, all mine; HP might not want them anyway... :)
feel free to post, OR email to raj in cup.hp.com  but NOT BOTH...

 
 
 

1. Poor Apache performance on Solaris x86 vs Linux

Hi all,

I have a client that is comparing the performance of Apache on Solaris x86
with that of Apache on Linux.  The Apache version is 1.3.9 (I know it's old,
but I think that's irrelevent...), and they're using Solaris 7 MU4.  Dunno
which version of Linux they're running.

In all the tests, the hardware is identical (it's the same machine, dual
boot), but for some reason, Linux's performance exceeds Solaris' by 30%!

I've tried everything I can think of (tuning the TCP/IP stack, increasing
the number of Apache processes, etc.), to no avail.

mpstat shows that the CPU is idle nearly 50% of the time, although top
says that no time is being spent waiting for I/O.

When the test is running, the smtx column is around 30; is this figure high
enough to cause concern?  (It's otherwise 0 or 1.)  The theory is that there's
lots of mutexes (which is good for MP scalability), that's hurting the
uniprocessor performance (but then why is the CPU idle 50% of the time?).

Any input welcomed!

TIA,

--
Rich Teer

NT tries to do almost everything UNIX does, but fails - miserably.

The use of Windoze cripples the mind; its use should, therefore, be
regarded as a criminal offence.  (With apologies to Edsger W. Dijkstra)

If it ain't analogue, it ain't music.

Voice: +1 (250) 763-6205
WWW: www.rite-group.com

2. NTP xntpd With Windows NT

3. Poor Apache 2 performance (relative to 1)

4. Downloading

5. Poor Apache performance

6. Altos error with PLOWBUG

7. Apache 1.0.5 poor performance

8. Environments

9. Poor(?) Linux Apache performance

10. Poor HTTP based file upload performance to Apache

11. HELP: very poor apache performance....

12. Poor Python/Perl network performance under Solaris

13. Poor NFS performance between Linux and Solaris