OS/Software Release Testing in Linux??

OS/Software Release Testing in Linux??

Post by Shyhtsun » Wed, 16 Nov 1994 03:45:03



Hi,

Since I could not find the answer in FAQ list, I post the following question
here:

      I noticed that a new version of kernel (in patch file format)
      was released within a few days. Or maybe a release of some
      packages was out every month. (For example, patch62.gz or
      pcmcia.2.3.gz.)

      I am wondering (maybe this is a dumb question) whether people gave
      "enough" tests to the software before the release. In other words,
      is there anything people were/are/doing for software quality assurance?

      If the answer is "yes", what is the current testing mechanism?
      If the answer is "no", should we provide one? If not, why?

Any information will be highly appreciated. BTW, I am interested in
developing tools that will automatically perform "regression tests", i.e.,
we will need minimal man power in this task.

-Felix

--
-----------------

450 Computer Science Building, Columbia University, NYC, NY 10027

 
 
 

OS/Software Release Testing in Linux??

Post by Daniel Barl » Wed, 16 Nov 1994 19:03:06


Please do not crosspost so widely.  If you must crosspost between linux
groups, please reset the followups to appear in only one.

Yes I know I'm guilty too, but I'd rather see n threads from this
overall than n per group.  Followup to .development, despite it not
being in the original.



Quote:>      I noticed that a new version of kernel (in patch file format)
>      was released within a few days. Or maybe a release of some
>      packages was out every month. (For example, patch62.gz or
>      pcmcia.2.3.gz.)

>      I am wondering (maybe this is a dumb question) whether people gave
>      "enough" tests to the software before the release. In other words,
>      is there anything people were/are/doing for software quality assurance?

How much is enough?  When it comes right down to it, the example
'patch62.gz' is officially alpha/beta software, and people probably
*ought* not to try it on a critical system or without backing up.  The
other point is that many developers only have access to one or few
machines, so some packages are released in the expectation that other
people will test them further.

AFAIK there is no official QA process, just the simultaneous
installation on thousands of computers around the world, followed by
the frantic news postings if/when something goes wrong.  Picks up most
problems :-)

Quote:>      If the answer is "no", should we provide one? If not, why?

I've thought about this on & off, but don't really have the time.  The
basic reason that there is not much attention given to testing is that
it's not much fun c/w writing new code.  (Also that the code is
generally pretty bug free.  This is *not* a linux flame, I'm running
1.1.61 and very happy with it)

Quote:>Any information will be highly appreciated. BTW, I am interested in
>developing tools that will automatically perform "regression tests", i.e.,
>we will need minimal man power in this task.

Automated regression testing would be nice, I admit.  I think however
that before we write test cases we'd need to define what the right
answer was.  That means (a) a lot of work, and (b) careful perusal of
all the relevant standards.

Another thing to think about: testing theory says that it is more
important to test old functionality than new (people get irritated if
things stop working, but not so annoyed if the projected new feature
isn't there) and more important to test for normal inputs than
boundary cases.  The effect of n thousand people simultaneously
installing and continuing to use their systems as normal fits those
criteria exactly :-)

Actually what I think would be a good idea is an automatic bug report
program, which given an EIP would look up the appropriate function &
offset, include the kernel version, package up excerpts from relevant
config files and post the whole bundle to a mailing list somewhere.  I
suspect that the KERNEL channel is not really built for that kind of
thing, though.

Daniel

 
 
 

OS/Software Release Testing in Linux??

Post by Jouni A Koson » Fri, 18 Nov 1994 10:13:20



>       I noticed that a new version of kernel (in patch file format)
>       was released within a few days. Or maybe a release of some
>       packages was out every month. (For example, patch62.gz or
>       pcmcia.2.3.gz.)

>       I am wondering (maybe this is a dumb question) whether people gave
>       "enough" tests to the software before the release. In other words,
>       is there anything people were/are/doing for software quality assurance?

>       If the answer is "yes", what is the current testing mechanism?
>       If the answer is "no", should we provide one? If not, why?

As we are talking about the 1.1.* (read ALPHA) kernels, I'd say
Yes - we are the current testing mechanism. That's the whole point
of making alpha releases in the first place.

Quote:> -Felix

Jouni Kosonen