contest benchmark v0.60

contest benchmark v0.60

Post by Con Koliva » Wed, 15 Jan 2003 01:30:14



Contest is a system responsiveness benchmark for kernel development looking at
fair scheduling.

It can be found here:
http://contest.kolivas.net

Aggelos Economopoulos has completed a massive rewrite of the contest benchmark
in c to remove all BASH scripts and this will form the codebase for further
versions of contest.

Changes since the last release:
Complete change in the codebase, consolidation into one executable.
FreeBSD compatibility
Different format of the logs etc.
Addition of cacherun and dbench_loads
Rewrite of process_load by Rene Herman
More detailed instructions
Man Page
Lots of internal changes

This version will give different results to previous versions of contest so to
upgrade please delete all the .log files in your testbed directory.

Known Bugs:
Contest will still run each kernel compilation if the load fails to initialise
(eg dbench doesnt start) so please ensure all the necessary load applications
are in your PATH
Other minor annoyances

Aggelos is the main coder now but I am still the official maintainer so please
direct all bug reports, comments, patches etc to me. Further releases are
planned in the near future with minor changes, but results obtained from contest
should be compatible with this version.

Con
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in

More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

 
 
 

contest benchmark v0.60

Post by Con Koliva » Thu, 16 Jan 2003 05:20:08


-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Ok some mildly annoying bugs have already shown up in this release.

I've put up a contest-0.61pre on the contest website that addresses the one
which ruins the results and has some of the changes going into 0.61

http://contest.kolivas.net

Major problem is that you need to run make oldconfig and make dep manually
before running contest successfully the first time. This will be corrected by
the time 0.61 comes out (probably by weekend).

Con
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.0 (GNU/Linux)

iD8DBQE+JNKiF6dfvkL3i1gRAssdAJ0TRiDhbQUP3lcgTV86/iqS6SfdfgCeNI1d
t73yXAn48I+8j+w/l2ieDT0=
=TqgD
-----END PGP SIGNATURE-----
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in

More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

 
 
 

contest benchmark v0.60

Post by Ian Wienan » Thu, 16 Jan 2003 06:20:12



> Ok some mildly annoying bugs have already shown up in this release.

> I've put up a contest-0.61pre on the contest website that addresses the one
> which ruins the results and has some of the changes going into 0.61

Con/Aggelos,

What was the motivation for re-writing in C?

I've done some hacking on the old version here, and so I realise that
such a big shell script was getting a little out of hand, but surely
perl or python is a better option for this application?

If it's going to stay in C maybe we could integrate profiling support
from /proc/profile, bypassing readprofile?  One of the guys here
recently wanted to get profiling information from his program, and it
would have been really good to have a library that could reset, start,
pause and return in some format the profiling data.  If you think your
users might be interested in profile outputs I can write something
maybe we can both use.

-i

http://www.gelato.unsw.edu.au
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in

More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

 
 
 

contest benchmark v0.60

Post by Con Koliva » Thu, 16 Jan 2003 06:50:05


-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1



> > Ok some mildly annoying bugs have already shown up in this release.

> > I've put up a contest-0.61pre on the contest website that addresses the
> > one which ruins the results and has some of the changes going into 0.61

> Con/Aggelos,

> What was the motivation for re-writing in C?

I believe Aggelos was looking at freebsd compatibility and c was his most
comfortable language.  Since programming such a thing in c was way beyond my
capabilities and Aggelos had already taken on the task I was very happy to
use this as the new codebase. The previous BASH one was getting flakier the
more I tried to add to it. Also there were subtle things happening in the
BASH version that made for much more variation in results than this version.
So it's as much coincidence that Aggelos was writing a new version and I was
looking for someone to write one that this codebase was chosen for it.

Quote:> I've done some hacking on the old version here, and so I realise that
> such a big shell script was getting a little out of hand, but surely
> perl or python is a better option for this application?

Possibly but clearly c has no major limitations once the hard part (the
wrapper for the other applications) has been done.

Quote:> If it's going to stay in C maybe we could integrate profiling support
> from /proc/profile, bypassing readprofile?  One of the guys here
> recently wanted to get profiling information from his program, and it
> would have been really good to have a library that could reset, start,
> pause and return in some format the profiling data.  If you think your
> users might be interested in profile outputs I can write something
> maybe we can both use.

I don't know. Anything that may taint the results is bad but I have no idea
how many confounding variables this really introduces. I need feedback from
the developers that look at these results to know whether this is worthwhile
or not. While I'm the said maintainer, contest has grown into a beast of it's
own.

Con
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.0 (GNU/Linux)

iD8DBQE+JOYHF6dfvkL3i1gRAnY3AJ42aiDcMm1oMhT4fU1c+BhPKzgUGQCfQdu7
KzNQ4kT3OFvOY4dgHPlfAaM=
=so41
-----END PGP SIGNATURE-----
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in

More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

 
 
 

contest benchmark v0.60

Post by Ian Wienan » Thu, 16 Jan 2003 07:30:11



> more I tried to add to it. Also there were subtle things happening in the
> BASH version that made for much more variation in results than this version.

I agree, it had begun to outgrow it's bash roots (e.g. i think it was
the ratio section with variables named a b c d e which drove me nuts
when I couldn't figure out which one was giving me a zero value).

Quote:> Possibly but clearly c has no major limitations once the hard part (the
> wrapper for the other applications) has been done.

I would disagree.  For example, playing tricks with strings,
pipes/redirects and files in C is a complete pain, compared with
perl/python.  Most of the things I changed with the orignal bash
version were in this subset, and it allowed me to customise it for
what we were doing quickly and easily.

-i

http://www.gelato.unsw.edu.au
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in

More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

 
 
 

contest benchmark v0.60

Post by ?γγελο? Οικονομ?πουλο » Thu, 16 Jan 2003 14:10:09




> > Ok some mildly annoying bugs have already shown up in this release.

> > I've put up a contest-0.61pre on the contest website that addresses the
> > one which ruins the results and has some of the changes going into 0.61

> Con/Aggelos,

> What was the motivation for re-writing in C?

I gave up trying to make the script work in my desktop, which runs freebsd
(too many little differences in the utilities used).

Quote:> I've done some hacking on the old version here, and so I realise that
> such a big shell script was getting a little out of hand, but surely
> perl or python is a better option for this application?

The script also had a few bugs that were actually artifacts of using bash
(e.g. you had to use killall instead of just killing your children).

Quote:> If it's going to stay in C maybe we could integrate profiling support
> from /proc/profile, bypassing readprofile?  One of the guys here
> recently wanted to get profiling information from his program, and it
> would have been really good to have a library that could reset, start,
> pause and return in some format the profiling data.  If you think your
> users might be interested in profile outputs I can write something
> maybe we can both use.

Feel free to do so. I won't be able to help because
a) I'm not interested the feature :)
b) I don't have the time, I'm just helping Con squash most of the bugs in my
code.

--
Follow each decision as closely as possible with its associated action.
            - The Elements of Programming Style (Kernighan & Plaugher)

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in

More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

 
 
 

contest benchmark v0.60

Post by Cliff Whit » Thu, 16 Jan 2003 19:10:17



> > Ok some mildly annoying bugs have already shown up in this release.

> > I've put up a contest-0.61pre on the contest website that addresses the one
> > which ruins the results and has some of the changes going into 0.61

> Con/Aggelos,

> What was the motivation for re-writing in C?

> I've done some hacking on the old version here, and so I realise that
> such a big shell script was getting a little out of hand, but surely
> perl or python is a better option for this application?

> If it's going to stay in C maybe we could integrate profiling support
> from /proc/profile, bypassing readprofile?  One of the guys here
> recently wanted to get profiling information from his program, and it
> would have been really good to have a library that could reset, start,
> pause and return in some format the profiling data.  If you think your
> users might be interested in profile outputs I can write something
> maybe we can both use.

I'd be very interested in learning more about getting profile support in things
Might come in very handy for AIM7.
cliffw
OSDL

> -i

> http://www.gelato.unsw.edu.au

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in

More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/
 
 
 

contest benchmark v0.60

Post by Cliff Whit » Thu, 16 Jan 2003 21:40:08



> > more I tried to add to it. Also there were subtle things happening in the
> > BASH version that made for much more variation in results than this version.

> I agree, it had begun to outgrow it's bash roots (e.g. i think it was
> the ratio section with variables named a b c d e which drove me nuts
> when I couldn't figure out which one was giving me a zero value).

> > Possibly but clearly c has no major limitations once the hard part (the
> > wrapper for the other applications) has been done.

> I would disagree.  For example, playing tricks with strings,
> pipes/redirects and files in C is a complete pain, compared with
> perl/python.  Most of the things I changed with the orignal bash
> version were in this subset, and it allowed me to customise it for
> what we were doing quickly and easily.

Good point.
One other note, the C version should have a much smaller memory footprint then
Perl or Python - less likely to perturb test results.
Speaking for myself, i think this is such a small pile of code, it might be fun
to have a parallel Perl/Python version. (it would be a good excuse for me to
learn some Python)
My personal choice would be C to generate the data and Perl to pretty-print
it.

 I need to get into the code more for STP integration, so i'll have more
opinons
soon :)
cliffw

> -i

> http://www.gelato.unsw.edu.au
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in

> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in

More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/
 
 
 

1. contest v0.61 benchmark

Contest is a system responsiveness benchmark for kernel development looking at
fair scheduling.

It can be found here:
http://contest.kolivas.org

Changes since the last release:
Lots of little bugfixes mainly
Increased resolution of read and io loads
Added io_other load
Corrected dbench_load to dbench num_cpus*16
More detailed instructions
Man Page update

This version will give different results to previous versions of contest so to
upgrade please delete all the .log files in your testbed directory.

Known Bugs:
Contest will still run each kernel compilation if the load fails to initialise
(eg dbench doesnt start) and so please ensure all the necessary load
applications are in your PATH.
io_other needs the -o parameter to know where to dump it's io load or else it
will be no load

If your load times are the same as your no_load times then the load has failed
to initialise.

Con
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in

More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

2. RTFM? *WHAT* FM?

3. Contest v0.51 benchmark

4. /etc/hw: SCO OS 5.0.4 only?

5. ANNOUNCE: xldlas v0.60 (Stats) is now available

6. Token-Ring problems (anyone)

7. desc.c v0.60 -- print i386 CPU descriptor tables

8. multiple ethernet cards

9. Gaim v0.60 is out!

10. DOSEm v0.60.4.3 won't compile

11. Contest v0.51 on STP

12. contest v0.61 on OSDL STP

13. contest v0.30