Nightly regression runs against current bk tree

Nightly regression runs against current bk tree

Post by Mark Peloqui » Fri, 30 May 2003 22:30:27



Our team would like to assist the community in quickly identifying
patches that provide
performance improvements or regressions in the 2.5 kernel tree. The way
to do this
will be to run a nightly regression test suite against the current bk
tree, and then compare
the results against the previous night's results, showing the
differences. Additionally,
also comparing against the 2.5 point release.

We have dedicated a machine and thrown together some scripts that will grab
and build the latest kernel files, execute the regression suite,
collecting (hopefully)
enough system state information to allow meaningful analysis of any peculiar
results encountered.

Here are links to the current regression results obtained:

2.5.70 vs 2.5.70-bk1:
http://www.ibm.com/developerworks/oss/linuxperf/regression/2.5.70-bk1...

2.5.70 vs 2.5.70-bk2:
http://www.ibm.com/developerworks/oss/linuxperf/regression/2.5.70-bk2...
2.5.70-bk1 vs 2.5.70-bk2
http://www.ibm.com/developerworks/oss/linuxperf/regression/2.5.70-bk2...

The regression suite executes in about 7.5 hours currently. We would
like to keep
the execution time below 12 hours, so when a problem is encountered, we
will have
time to recover without falling behind on the daily snapshots. We have
attempted
to strike a balance between test execution time and test coverage. Work
is still
ongoing in the area to provide the best balance and maintain repeatability.

Currently the regression suite operates on the 2.5 kernel bk tree. We do
plan on
adding another machine that will perform similiar regression comparisons
for the
-mm and -mjb patches.

Please bear in mind this is work in progress and there might be a few
rough edges.
However, with your input, we feel it can provide a useful function.
Please do not
hesitate to provide feedback or suggestions on improvements including
content
and presentation.

Mark Peloquin
IBM Linux performance team

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in

More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

 
 
 

Nightly regression runs against current bk tree

Post by Andi Klee » Fri, 30 May 2003 23:20:24



> We have dedicated a machine and thrown together some scripts that will grab
> and build the latest kernel files, execute the regression suite,
> collecting (hopefully)
> enough system state information to allow meaningful analysis of any peculiar
> results encountered.

How about doing a LTP run too with some difference file for new FAILs/BROKs ?
That's not strictly a benchmark, but would help catching regressions
quickly.

I notice your benchmark mix is very IO heavy, it would be nice to test other
aspects of the system too. Perhaps lmbench and reaim compute workload?

It would be nice if we had a new linux-testresults list where such
updates could be posted regularly. I don't think it belong on l-k
because it would be too noisy. Perhaps such a list could be added to
vger. David, what do you think?

-Andi
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in

More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

 
 
 

Nightly regression runs against current bk tree

Post by David S. Mille » Fri, 30 May 2003 23:30:23



   Date: 29 May 2003 23:11:17 +0200

   David, what do you think?

Would it have a single poster?
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in

More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

 
 
 

Nightly regression runs against current bk tree

Post by Andi Klee » Fri, 30 May 2003 23:40:10




>    Date: 29 May 2003 23:11:17 +0200

>    David, what do you think?

> Would it have a single poster?

OSDL, Mark's IBM team and possible LTP ?

I assume there will be more once the list exists; automated regression
tests seem to be currently in fashion.

-Andi
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in

More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

 
 
 

Nightly regression runs against current bk tree

Post by Randy.Dunla » Fri, 30 May 2003 23:50:05





| >    Date: 29 May 2003 23:11:17 +0200
| >    
| >    David, what do you think?
| >    
| > Would it have a single poster?
|
| OSDL, Mark's IBM team and possible LTP ?
|
| I assume there will be more once the list exists; automated regression
| tests seem to be currently in fashion.

If DaveM doesn't want to do it, I think that we can do it.
(I say without checking.... :)

--
~Randy
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in

More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

 
 
 

Nightly regression runs against current bk tree

Post by David S. Mille » Sat, 31 May 2003 00:00:08



   Date: Thu, 29 May 2003 14:38:20 -0700



   | > Would it have a single poster?
   |
   | OSDL, Mark's IBM team and possible LTP ?
   |
   | I assume there will be more once the list exists; automated regression
   | tests seem to be currently in fashion.

   If DaveM doesn't want to do it, I think that we can do it.
   (I say without checking.... :)

Please do :-)

The issue is that I'm easier about adding a new list if I can
restrict the poster list.

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in

More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

 
 
 

Nightly regression runs against current bk tree

Post by Craig Thoma » Sat, 31 May 2003 00:00:16





> >    Date: 29 May 2003 23:11:17 +0200

> >    David, what do you think?

> > Would it have a single poster?

> OSDL, Mark's IBM team and possible LTP ?

> I assume there will be more once the list exists; automated regression
> tests seem to be currently in fashion.

> -Andi
> -

OSDL has a linux stabilization web page where several tests are run
automatically when a new kernel is built.  It currently runs Linus'
kernel as well as the -mm series.  We do run LTP, I/O tests, memory
tests, reaim, and database tests as part of an automated regression
run. Some of you are familiar with the web page, but for those who are
not, it is located here: http://www.osdl.org/projects/linstab/  

In addition, there are links to other sites, most notably IBM's LTC
test results.

We have just completed a physical move to a new office and we believe
we have all of our systems working again, so test results for the
latest kernels are a bit behind.  We hope to have completed runs for
all tests by the weekend.  Note, we are experiencing some test failures
but we suspect it is due to the move and not the kernels at the moment.

--
Craig Thomas

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in

More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

 
 
 

Nightly regression runs against current bk tree

Post by Andi Klee » Sat, 31 May 2003 00:10:09


Quote:> OSDL has a linux stabilization web page where several tests are run

[...] Would you be willing to change your scripts to report
any new results to this new list?

-Andi
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in

More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

 
 
 

Nightly regression runs against current bk tree

Post by Craig Thoma » Sat, 31 May 2003 00:10:10




>    Date: Thu, 29 May 2003 14:38:20 -0700



>    | > Would it have a single poster?
>    |
>    | OSDL, Mark's IBM team and possible LTP ?
>    |
>    | I assume there will be more once the list exists; automated regression
>    | tests seem to be currently in fashion.

>    If DaveM doesn't want to do it, I think that we can do it.
>    (I say without checking.... :)

> Please do :-)

> The issue is that I'm easier about adding a new list if I can
> restrict the poster list.

OSDL has a mail list that is used to discuss the stability of the linux
kernel.  This would be a perfect list to use for posting test results.

OSDL.  To subscribe: http://www.osdl.org/mailman/listinfo/linstab

--
Craig Thomas

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in

More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

 
 
 

Nightly regression runs against current bk tree

Post by Natha » Sat, 31 May 2003 00:10:11



> It would be nice if we had a new linux-testresults list where such
> updates could be posted regularly. I don't think it belong on l-k
> because it would be too noisy. Perhaps such a list could be added to
> vger. David, what do you think?

The OSDL has a serious amount of automated testing we could point the
results of to a separate list if it is created.

Right now we avoid pointing that sort of thing to l-k because it would
drive people nuts.  On average we complete 40+ tests a day.

With all the testing efforts going on, a central list to post and
analyze results would be good.  People interested in helping out could
easily work with testers to look for trends and help with root cause
analysis.

When results are found to contain significant data, we can always notify l-k.

-Nathan Dabney
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in

More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

 
 
 

Nightly regression runs against current bk tree

Post by Andi Klee » Sat, 31 May 2003 00:10:11


> OSDL has a mail list that is used to discuss the stability of the linux
> kernel.  This would be a perfect list to use for posting test results.

> OSDL.  To subscribe: http://www.osdl.org/mailman/listinfo/linstab

That's fairly obscure (Nobody knew of it before). Perhaps a well publicized
list on vger would be better.

-Andi
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in

More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

 
 
 

Nightly regression runs against current bk tree

Post by Cliff Whit » Sat, 31 May 2003 00:20:18


Quote:> > OSDL has a linux stabilization web page where several tests are run

> [...] Would you be willing to change your scripts to report
> any new results to this new list?

I would be very interested in this if it leads to more people *looking* at the
tests results.

Automating this stuff is the easy part - getting intelligence out of the
results is harder.
The more eyeballs we can get to look, the easier this gets.

If a new list, or better use of an old list will help, i'll change whatever is
necessary.
If people like the Web, but don't like our paper layout, i'll change that too.

cliffw

As

> -Andi
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in

> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in

More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/
 
 
 

Nightly regression runs against current bk tree

Post by Natha » Sat, 31 May 2003 00:30:15



> > OSDL has a linux stabilization web page where several tests are run

> [...] Would you be willing to change your scripts to report
> any new results to this new list?

The linux stabilization web page uses results from the STP runs I
mentioned (40+ per day).  The STP emails results summaries after test
runs so we could easily redirect the results to this new list.

-Nathan Dabney
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in

More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

 
 
 

Nightly regression runs against current bk tree

Post by Cliff Whit » Sat, 31 May 2003 00:30:19


> > OSDL has a mail list that is used to discuss the stability of the linux
> > kernel.  This would be a perfect list to use for posting test results.

> > OSDL.  To subscribe: http://www.osdl.org/mailman/listinfo/linstab

> That's fairly obscure (Nobody knew of it before). Perhaps a well publicized
> list on vger would be better.

Perhaps - though we can publicize any new list.
We're content to leave the decision to DaveM and the list team -  
if they don't want the extra work, we're always glad to help.
cliffw
OSDL

> -Andi
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in

> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in

More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/
 
 
 

Nightly regression runs against current bk tree

Post by Mark Peloqui » Sat, 31 May 2003 01:00:15




>>We have dedicated a machine and thrown together some scripts that will grab
>>and build the latest kernel files, execute the regression suite,
>>collecting (hopefully)
>>enough system state information to allow meaningful analysis of any peculiar
>>results encountered.

>How about doing a LTP run too with some difference file for new FAILs/BROKs ?
>That's not strictly a benchmark, but would help catching regressions
>quickly.

I'm under the impression that LTP and other test efforts seemed to focus
more on functional evaluation, which is fine.  We are trying to focus
purely on the performance differences seen from day to day.

Quote:

>I notice your benchmark mix is very IO heavy, it would be nice to test other
>aspects of the system too. Perhaps lmbench and reaim compute workload?

Your correct. We're just getting started with this effort and we used
this mix to get things going. Once ppl are happy with the presentation
of data, we planned to add more tests to provide a more balanced mix.
But since you asked, we have added lmbench to our -bk3 regression run. :)

Mark

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in

More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

 
 
 

1. Nightly regression runs against current bk tree

Sorry I didn't see this sooner, I'm unsubscribed for the moment until my
email provider can get exim/procmail talking nicely.

LTP has had a mailing list for a long time that is explicitly for the
purpose of posting results.  It's currently underutilized so I'd love to
see more results getting posted there again.  Please consider using that
one for posting results of all types (LTP and non-ltp)


Thanks,
Paul Larson

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in

More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

2. S3 ViRGE/VX problem

3. Nightly regression run results

4. Help with SMC Ultra card

5. bad: schedule() with irqs disabled! - current bk tree

6. NIC with several RJ45

7. fix UP links - current bk tree

8. ISAPNP for sound card

9. PANIC caused by dequeue_signal() in current Linus BK tree

10. pfn-Functionset out of order for sparc64 in current Bk tree?

11. IDE from current bk tree, UDMA and two channels...

12. fix current BK tree compilation with devfs enabled

13. BUG: Current 2.5-BK tree dies on boot!