Performance issues using ksh for cgi scripts

Performance issues using ksh for cgi scripts

Post by Bernd.Schemme » Wed, 06 Apr 2005 05:42:07



Hi,

I've written a cgi script in ksh that needs some performance tuning.

The main work in the script is done in the loop below. This loop needs
about 90% of the execution time so I want to make it a little bit
faster.

Environment: apache 1.3.27 on Solaris 9, ksh is the ksh from Solaris

Any hints are welcome.

regards

Bernd

   while read __PATCHNO __PATCHREV __PATCHDATE __PATCH_REC __PATCH_SEC
__PATCH_OBSOLETE __PATCH_WITHDRAWN \
              __PATCH_OS __PATCH_ARC __PATCH_PKGS __PATCH_DESC ; do

     if [ "${PATCH_ARC}"x != ""x ] ; then
       [[ ! ${__PATCH_ARC} == *${PATCH_ARC}* ]] && continue
     fi

     if [ "${PATCH_OS}"x != ""x ] ; then
       [[ ! ${__PATCH_OS} == *${PATCH_OS}* ]] && continue
     fi

     [[ ${RECOMMENDED_PATCH} = "R" && ! ${__PATCH_REC} = "R" ]] && continue
     [[ ${RECOMMENDED_PATCH} = "r" &&   ${__PATCH_REC} = "R" ]] && continue

     [[ ${SECURITY_PATCH} = "S" && ! ${__PATCH_SEC} = "S" ]] && continue
     [[ ${SECURITY_PATCH} = "s" &&   ${__PATCH_SEC} = "S" ]] && continue

     [[ ${OBSOLETE_PATCH} = "O" && ! ${__PATCH_OBSOLETE} = "O" ]] &&
continue
     [[ ${OBSOLETE_PATCH} = "o" &&   ${__PATCH_OBSOLETE} = "O" ]] &&
continue

     [[ ${WITHDRAWN_PATCH} = "B"  && ! ${__PATCH_WITHDRAWN} == *B ]] &&
continue
     [[ ${WITHDRAWN_PATCH} = "b"  &&   ${__PATCH_WITHDRAWN} == *B ]] &&
continue

# echo "${__PATCH_OS} ${__PATCH_REC} ${__PATCH_SEC} ${__PATCH_OBSOLETE}
${__PATCH_WITHDRAWN}"

#    if [[ ${SEARCHSTRING} != "" ]] ; then
#      for CURWORD in ${SEARCHSTRING} ; do
#        [[ ! "${__PATCHNO} ${__PATCHREV} ${__PATCHDATE} ${__PATCH_REC}
${__PATCH_SEC} ${__PATCH_OBSOLETE} ${__PATCH_WITHDRAWN} ${__PATCH_OS}
${__PATCH_ARC} ${__PATCH_PKGS} ${__PATCH_DESC}" ==  *${CURWORD}* ]] &&
continue
#      done
#    fi

     if [[ ${PATCH_MUST_EXIST_LOCAL} != "" ]] ; then
       if [ -d ${LOCAL_PATCH_DIRECTORY}/${__PATCHNO}-${__PATCHREV} ] ; then
         __PATCH_EXISTS_ON_LOCAL_FS="Y"
         [[ ${PATCH_MUST_EXIST_LOCAL} = "N" ]] && continue
       else
         __PATCH_EXISTS_ON_LOCAL_FS="NO"
         [[ ${PATCH_MUST_EXIST_LOCAL} = "Y" ]] && continue
       fi
     fi

     LIST_OF_PATCHES="${LIST_OF_PATCHES} ${__PATCHNO}-${__PATCHREV}"
     (( NO_OF_PATCHES=NO_OF_PATCHES+1 ))

     oIFS="${IFS}"
     IFS=";"
     set -- ${__PATCH_ARC}
     if [ "$1"x != ""x ] ; then
       CUR_ARCS=$1
       shift
     fi

     while [ "$1"x != ""x ] ; do
       CUR_ARCS="${CUR_ARCS}<br>$1"
       shift
     done
     IFS="${oIFS}"

     oIFS="${IFS}"
     IFS=";"
     set -- ${__PATCH_PKGS}
     if [ "$1"x != ""x ] ; then
       CUR_PKGS="$1"
       shift
     fi

     while [ "$1"x != ""x ] ; do
       CUR_PKGS="${CUR_PKGS}<br>$1"
       shift
     done
     IFS="${oIFS}"

     cat <<EOT

<tr>
<td style="vertical-align: top;">
<a
href="http://patches.sun.com/all_unsigned/${__PATCHNO}-${__PATCHREV}.zip">${__PATCHNO}-${__PATCHREV}</a>
<a
href="http://sunsolve.sun.com/pub-cgi/pdownload.pl?target=${__PATCHNO}-${__PATCHREV}&method=h">http</a>
<a
href="http://sunsolve.sun.com/pub-cgi/pdownload.pl?target=${__PATCHNO}-${__PATCHREV}&method=f">ftp</a>
<br>
signed:
<br>
<a
href="http://sunsolve.sun.com/pub-cgi/pdownload.pl?target=${__PATCHNO}-${__PATCHREV}&method=hs">http</a>
<a
href="http://sunsolve.sun.com/pub-cgi/pdownload.pl?target=${__PATCHNO}-${__PATCHREV}&method=fs">ftp</a>
</td>
<td style="vertical-align: top;">${__PATCHREV}<br>
</td>
<td style="vertical-align: top;">${__PATCHDATE}<br>
</td>

<td style="vertical-align: top;">${__PATCH_EXISTS_ON_LOCAL_FS}<br>
</td>

<td style="vertical-align: top;">${__PATCH_REC}<br>
</td>
<td style="vertical-align: top;">${__PATCH_SEC}<br>
</td>
<td style="vertical-align: top;">${__PATCH_OBSOLETE}<br>
</td>
<td style="vertical-align: top;">${__PATCH_WITHDRAWN}<br>
</td>
<td style="vertical-align: top;">${__PATCH_OS}<br>
</td>
<td style="vertical-align: top; width: 50px;">${CUR_ARCS}<br>
</td>
<td style="vertical-align: top; width: 50px;">${CUR_PKGS}<br>
</td>
<td style="vertical-align: top; width: 50px;"><a target=patchdesc
href="http://patches.sun.com/all_unsigned/${__PATCHNO}-${__PATCHREV}.README">${__PATCH_DESC}</a><br>
</td>
</tr>

EOT

# <td style="vertical-align: top;"><a
href="http://patches.sun.com/all_unsigned/${__PATCHNO}-${__PATCHREV}.zip">${__PATCHNO}-${__PATCHREV}</a><br>

   done <${TMPFILE2}

--
Bernd Schemmer                         http://home.arcor.de/bnsmb/index.html

 
 
 

Performance issues using ksh for cgi scripts

Post by Kevin Collin » Wed, 06 Apr 2005 08:10:40



> Hi,

> I've written a cgi script in ksh that needs some performance tuning.

> The main work in the script is done in the loop below. This loop needs
> about 90% of the execution time so I want to make it a little bit
> faster.

> Environment: apache 1.3.27 on Solaris 9, ksh is the ksh from Solaris

> Any hints are welcome.

> regards

> Bernd

>    while read __PATCHNO __PATCHREV __PATCHDATE __PATCH_REC __PATCH_SEC
> __PATCH_OBSOLETE __PATCH_WITHDRAWN \
>               __PATCH_OS __PATCH_ARC __PATCH_PKGS __PATCH_DESC ; do

>      if [ "${PATCH_ARC}"x != ""x ] ; then
>        [[ ! ${__PATCH_ARC} == *${PATCH_ARC}* ]] && continue
>      fi

>      if [ "${PATCH_OS}"x != ""x ] ; then
>        [[ ! ${__PATCH_OS} == *${PATCH_OS}* ]] && continue
>      fi

Why are you making combined tests and using Bourne and Korn test types? Those
could be:

      [[ ! -z ${PATCH_ARC} && ! ${__PATCH_ARC} == *${PATCH_ARC}* ]] && continue

      [[ ! -z ${PATCH_OS} && ! ${__PATCH_OS} == *${PATCH_OS}* ]] && continue

- Show quoted text -

Quote:>      [[ ${RECOMMENDED_PATCH} = "R" && ! ${__PATCH_REC} = "R" ]] && continue
>      [[ ${RECOMMENDED_PATCH} = "r" &&   ${__PATCH_REC} = "R" ]] && continue

>      [[ ${SECURITY_PATCH} = "S" && ! ${__PATCH_SEC} = "S" ]] && continue
>      [[ ${SECURITY_PATCH} = "s" &&   ${__PATCH_SEC} = "S" ]] && continue

>      [[ ${OBSOLETE_PATCH} = "O" && ! ${__PATCH_OBSOLETE} = "O" ]] &&
> continue
>      [[ ${OBSOLETE_PATCH} = "o" &&   ${__PATCH_OBSOLETE} = "O" ]] &&
> continue

>      [[ ${WITHDRAWN_PATCH} = "B"  && ! ${__PATCH_WITHDRAWN} == *B ]] &&
> continue
>      [[ ${WITHDRAWN_PATCH} = "b"  &&   ${__PATCH_WITHDRAWN} == *B ]] &&
> continue

> # echo "${__PATCH_OS} ${__PATCH_REC} ${__PATCH_SEC} ${__PATCH_OBSOLETE}
> ${__PATCH_WITHDRAWN}"

> #    if [[ ${SEARCHSTRING} != "" ]] ; then
> #      for CURWORD in ${SEARCHSTRING} ; do
> #        [[ ! "${__PATCHNO} ${__PATCHREV} ${__PATCHDATE} ${__PATCH_REC}
> ${__PATCH_SEC} ${__PATCH_OBSOLETE} ${__PATCH_WITHDRAWN} ${__PATCH_OS}
> ${__PATCH_ARC} ${__PATCH_PKGS} ${__PATCH_DESC}" ==  *${CURWORD}* ]] &&
> continue
> #      done
> #    fi

Again, mixing Korn '[[ ... ]]' and Bourne '[ ... ]' styles for no apparent
reason...

- Show quoted text -

Quote:>      if [[ ${PATCH_MUST_EXIST_LOCAL} != "" ]] ; then
>        if [ -d ${LOCAL_PATCH_DIRECTORY}/${__PATCHNO}-${__PATCHREV} ] ; then
>          __PATCH_EXISTS_ON_LOCAL_FS="Y"
>          [[ ${PATCH_MUST_EXIST_LOCAL} = "N" ]] && continue
>        else
>          __PATCH_EXISTS_ON_LOCAL_FS="NO"
>          [[ ${PATCH_MUST_EXIST_LOCAL} = "Y" ]] && continue
>        fi
>      fi

>      LIST_OF_PATCHES="${LIST_OF_PATCHES} ${__PATCHNO}-${__PATCHREV}"
>      (( NO_OF_PATCHES=NO_OF_PATCHES+1 ))

>      oIFS="${IFS}"
>      IFS=";"
>      set -- ${__PATCH_ARC}

Use ksh semantics here:

Quote:>      if [ "$1"x != ""x ] ; then

       if [[ $# > 0 ]]; then

Quote:>        CUR_ARCS=$1
>        shift
>      fi

And here:

Quote:>      while [ "$1"x != ""x ] ; do

       while [[ $# > 0 ]] ; do

Quote:>        CUR_ARCS="${CUR_ARCS}<br>$1"
>        shift
>      done
>      IFS="${oIFS}"

>      oIFS="${IFS}"
>      IFS=";"
>      set -- ${__PATCH_PKGS}
>      if [ "$1"x != ""x ] ; then
>        CUR_PKGS="$1"
>        shift
>      fi

Same...

- Show quoted text -

Quote:>      while [ "$1"x != ""x ] ; do
>        CUR_PKGS="${CUR_PKGS}<br>$1"
>        shift
>      done
>      IFS="${oIFS}"

>      cat <<EOT

><tr>
><td style="vertical-align: top;">
><a
> href="http://patches.sun.com/all_unsigned/${__PATCHNO}-${__PATCHREV}.zip">${__PATCHNO}-${__PATCHREV}</a>
><a
> href="http://sunsolve.sun.com/pub-cgi/pdownload.pl?target=${__PATCHNO}-${__PATCHREV}&method=h">http</a>
><a
> href="http://sunsolve.sun.com/pub-cgi/pdownload.pl?target=${__PATCHNO}-${__PATCHREV}&method=f">ftp</a>
><br>
> signed:
><br>
><a
> href="http://sunsolve.sun.com/pub-cgi/pdownload.pl?target=${__PATCHNO}-${__PATCHREV}&method=hs">http</a>
><a
> href="http://sunsolve.sun.com/pub-cgi/pdownload.pl?target=${__PATCHNO}-${__PATCHREV}&method=fs">ftp</a>
></td>
><td style="vertical-align: top;">${__PATCHREV}<br>
></td>
><td style="vertical-align: top;">${__PATCHDATE}<br>
></td>

><td style="vertical-align: top;">${__PATCH_EXISTS_ON_LOCAL_FS}<br>
></td>

><td style="vertical-align: top;">${__PATCH_REC}<br>
></td>
><td style="vertical-align: top;">${__PATCH_SEC}<br>
></td>
><td style="vertical-align: top;">${__PATCH_OBSOLETE}<br>
></td>
><td style="vertical-align: top;">${__PATCH_WITHDRAWN}<br>
></td>
><td style="vertical-align: top;">${__PATCH_OS}<br>
></td>
><td style="vertical-align: top; width: 50px;">${CUR_ARCS}<br>
></td>
><td style="vertical-align: top; width: 50px;">${CUR_PKGS}<br>
></td>
><td style="vertical-align: top; width: 50px;"><a target=patchdesc
> href="http://patches.sun.com/all_unsigned/${__PATCHNO}-${__PATCHREV}.README">${__PATCH_DESC}</a><br>
></td>
></tr>

> EOT

> # <td style="vertical-align: top;"><a
> href="http://patches.sun.com/all_unsigned/${__PATCHNO}-${__PATCHREV}.zip">${__PATCHNO}-${__PATCHREV}</a><br>

>    done <${TMPFILE2}

Nothing here looks too inefficient at first look. Everything is really
happening in the shell memory, so I can't make the standard "do less external
commands" recommendation :) Are you sure this is where the script is slow? How
big is the file you read in from (# of lines)? How long are the variables you
are looping on (the 3 while loops with changed IFS)?

What does the rest of your script look like?

Kevin

--
Unix Guy Consulting, LLC
Unix and Linux Automation, Shell, Perl and CGI scripting
http://www.unix-guy.com

 
 
 

Performance issues using ksh for cgi scripts

Post by Willia » Thu, 07 Apr 2005 00:14:48



Quote:

> Why are you making combined tests and using Bourne and Korn test types?
Those
> could be:

>   [[ ! -z ${PATCH_ARC} && ! ${__PATCH_ARC} == *${PATCH_ARC}* ]] &&

continue

Is there anything wrong with using -n instead of ! -z?

And, since case is often faster than a test, you could probably use
something like this (it assumes that __PATCH_ARC can never be "X", if it
could, you'd want to select a different default value in
${PATCH_ARC:-X}).

    case "$__PATCH_ARC" in
        *${PATCH_ARC:-X}* ) ;;
        *) continue ;;
    esac

-Wm

 
 
 

Performance issues using ksh for cgi scripts

Post by Bernd.Schemme » Thu, 07 Apr 2005 04:23:05


Thanks for the hints.

  > Why are you making combined tests and using Bourne and Korn test types?

Never thought about using Bourne or Korn test types - if it works I use it.

Quote:> Nothing here looks too inefficient at first look. Everything is really
> happening in the shell memory, so I can't make the standard "do less external
> commands" recommendation :) Are you sure this is where the script is slow?

Yes, I inserted some date commands to get the time used in different
parts of the script.

Quote:> big is the file you read in from (# of lines)?

Depends on the user selection from 3 to about 6000 lines.

Anyway I found the command that used the most of the time in the script.
It was the line

      LIST_OF_PATCHES="${LIST_OF_PATCHES} ${__PATCHNO}-${__PATCHREV}"

After I replaced this construct with

      echo  "${__PATCHNO}-${__PATCHREV}">>"${TMPFILE3}"

it runs much faster now.

Quote:> What does the rest of your script look like?

http://home.arcor.de/bnsmb/public/htdocs/vxpref.html

Thanks

Bernd

--
Bernd Schemmer                         http://home.arcor.de/bnsmb/index.html

 
 
 

Performance issues using ksh for cgi scripts

Post by Bernd.Schemme » Thu, 07 Apr 2005 04:24:17



> And, since case is often faster than a test, you could probably use
> something like this (it assumes that __PATCH_ARC can never be "X", if it
> could, you'd want to select a different default value in
> ${PATCH_ARC:-X}).

>     case "$__PATCH_ARC" in
>         *${PATCH_ARC:-X}* ) ;;
>         *) continue ;;
>     esac

Thanks, but it seems not to work: The script always continues with the
last case.

regards

Bernd

Quote:> -Wm

--
Bernd Schemmer                         http://home.arcor.de/bnsmb/index.html
 
 
 

Performance issues using ksh for cgi scripts

Post by Kevin Collin » Thu, 07 Apr 2005 08:57:22




> Thanks for the hints.

>  > Why are you making combined tests and using Bourne and Korn test types?

> Never thought about using Bourne or Korn test types - if it works I use it.

>> Nothing here looks too inefficient at first look. Everything is really
>> happening in the shell memory, so I can't make the standard "do less external
>> commands" recommendation :) Are you sure this is where the script is slow?

> Yes, I inserted some date commands to get the time used in different
> parts of the script.

>> big is the file you read in from (# of lines)?

> Depends on the user selection from 3 to about 6000 lines.

> Anyway I found the command that used the most of the time in the script.
> It was the line

>       LIST_OF_PATCHES="${LIST_OF_PATCHES} ${__PATCHNO}-${__PATCHREV}"

> After I replaced this construct with

>       echo  "${__PATCHNO}-${__PATCHREV}">>"${TMPFILE3}"

> it runs much faster now.

Strange, technically it should be slower - you are doing file I/O on every loop
iteration instead of just adjusting a variable string in memory. Is your system
memory constrained?

Quote:>> What does the rest of your script look like?

> http://home.arcor.de/bnsmb/public/htdocs/vxpref.html

Kevin

--
Unix Guy Consulting, LLC
Unix and Linux Automation, Shell, Perl and CGI scripting
http://www.unix-guy.com

 
 
 

Performance issues using ksh for cgi scripts

Post by Ed Morto » Thu, 07 Apr 2005 09:24:16




<snip>
>>Anyway I found the command that used the most of the time in the script.
>>It was the line

>>      LIST_OF_PATCHES="${LIST_OF_PATCHES} ${__PATCHNO}-${__PATCHREV}"

>>After I replaced this construct with

>>      echo  "${__PATCHNO}-${__PATCHREV}">>"${TMPFILE3}"

>>it runs much faster now.

> Strange, technically it should be slower - you are doing file I/O on every loop
> iteration instead of just adjusting a variable string in memory. Is your system
> memory constrained?

I don't know about shell, but that would be the expected result in awk
where string concatenation is very expensive as it requires shuffling
all the existing strings around in memory. Perhaps this shell's doing
the same thing...

        Ed.

 
 
 

Performance issues using ksh for cgi scripts

Post by Icarus Sparr » Thu, 07 Apr 2005 12:07:05



> Anyway I found the command that used the most of the time in the script.
> It was the line

>       LIST_OF_PATCHES="${LIST_OF_PATCHES} ${__PATCHNO}-${__PATCHREV}"

> After I replaced this construct with

>       echo  "${__PATCHNO}-${__PATCHREV}">>"${TMPFILE3}"

> it runs much faster now.

>> What does the rest of your script look like?

> http://home.arcor.de/bnsmb/public/htdocs/vxpref.html

You might try putting
        exec 3>"${TMPFILE3}"
before your while loop, and
        exec 3>&-
after it, and change your echo to
        echo  "${__PATCHNO}-${__PATCHREV}">&3

and see if it makes any difference to the speed. Depending on the shell
implementation, it can reduce the number of system calls a lot.

 
 
 

Performance issues using ksh for cgi scripts

Post by Willia » Thu, 07 Apr 2005 23:59:13




> > And, since case is often faster than a test, you could probably use
> > something like this (it assumes that __PATCH_ARC can never be "X", if it
> > could, you'd want to select a different default value in
> > ${PATCH_ARC:-X}).

> >     case "$__PATCH_ARC" in
> >         *${PATCH_ARC:-X}* ) ;;
> >         *) continue ;;
> >     esac

> Thanks, but it seems not to work: The script always continues with the
> last case.

Seems to work for me:

#!/bin/sh
__PATCH_ARC='theTestValue'
for PATCH_ARC in something1 something2 TestValue something3
do
   case "$__PATCH_ARC" in
      *${PATCH_ARC:-X}* ) ;;
      *) echo "Skipping $PATCH_ARC" ; continue ;;
  esac
echo "Bingo!"
done

Quote:> test.sh

Skipping something1
Skipping something2
Bingo!
Skipping something3

-Wm

 
 
 

1. Apache, cgi forms, cgi programs - issues - restart resolves issue??

This is a very strange occurance.

Running Apache 1.3.26 or other versions (tried several)
Sun Ultra 2 w/1536 meg
Solaris 8

Same issue on other machines running similar configs

We are running several web servers on this box.
Maybe about 25 total. We also use a shopping cart program
written in cgi. Our email form scripts are written in cgi.
Ok, here is the weird and strange happenings we are running into.

Error log reports malformed headers when a form is submitted
while this is happening.

Every so often the shopping cart program will not allow
you to "view the next 25 items". Clicking that button just brings them
back to the shopping carts home page.
When this happens with the cgi forms, submitting the form
returns an error as if their is a field missing and there are not any
missing or empty fields.

Now if I "./apachectl restart", both of the above seems to work fine.
Then a couple hours, sometimes a day, sometimes a few days or even a week later
it happens again. We never know unless a client calls about the form
not working.

I'm thinking there may be some sort of process that gets killed.
Any ideas on what that process would be?
Any way to monitor the system to find out what process that would
be or what is happening to it?

I am lost with this issue.
Any help is greatly appreciated.

Thanks

2. ncd explora 700 xterminal and rh8 ?

3. Using .cgi/.pl to enable CGI script in Apache

4. dvd playing and redion code?

5. ksh script security issues

6. Second NIC.

7. ksh and awk scripts to HPUX - any issues

8. urgent -- AIX 4.33 files

9. How to issue a "return" in a ksh script?

10. Apache & ksh cgi scripts...

11. ksh cgi script

12. ksh - Are there semaphores in ksh? Concurrency issues...

13. Performance issue when using FreeBSD as a gateway