TDD without pay-off

TDD without pay-off

Post by Gerold Keefe » Mon, 30 Jun 2003 05:50:10



hello *,

for an interesting experiment on TDD or test-first, have a look at
http://www.ipd.uka.de/KarHPFn/papers/ease02.pdf .
you will read at the bottom:

"Test-first pays off only slightly in terms of increased reliability.
In fact, there were five programs developed with test-first with a
reliability over 96% compared to one program in the control group.
But this result is blurred by the large variance of the data-points.
Concentrating on the program versions after the implementation-phase,
the result just turns around. The test-first group has signifcantly
less reliable programs than the control group. So far, we do not know,
if this effect is caused by a false sense of security, less importance
of the acceptance-test for the test-first group, or if it is quite
simply a result of too little testing."

the study implies that it is at least unclear whether test-first is
benificial,
but certainly the whole XP consulting crowd sells it - on good trust.

regards,

gerold

 
 
 

TDD without pay-off

Post by John Rot » Mon, 30 Jun 2003 09:34:11



Quote:> hello *,

> for an interesting experiment on TDD or test-first, have a look at
> http://www.ipd.uka.de/KarHPFn/papers/ease02.pdf .
> you will read at the bottom:

> "Test-first pays off only slightly in terms of increased reliability.
> In fact, there were five programs developed with test-first with a
> reliability over 96% compared to one program in the control group.
> But this result is blurred by the large variance of the data-points.
> Concentrating on the program versions after the implementation-phase,
> the result just turns around. The test-first group has signifcantly
> less reliable programs than the control group. So far, we do not know,
> if this effect is caused by a false sense of security, less importance
> of the acceptance-test for the test-first group, or if it is quite
> simply a result of too little testing."

> the study implies that it is at least unclear whether test-first is
> benificial,
> but certainly the whole XP consulting crowd sells it - on good trust.

> regards,

> gerold

The researcher says specifically:

{begin quote]
"One of the challenges
of studying test-first is its embedding within XP. This
embedding makes it difficult to show the effects of test-first
without being blurred by other practices such as pair programming
or a simple design. A solution to this problem would be an
experiment in which XP is applied twice: with test-first and
without test-first. But this kind of experiment is too difficult
and too expensive. To solve this problem, test-first was
extracted from XP and evaluated on its own.

[end quote]

I believe this is the same researcher as the pair programming
study you brought up earlier. She seems to have a plan to
study the components in isolation, and then put them together
later.

I'm quite looking forward to how she does that, given her
disclaimer quoted above. It should be interesting.

I'm still waiting for a credible academic study of test
driven development, with all practices in place and
well practiced before starting out.

I am not, however, holding my breath.

John Roth

- Show quoted text -


 
 
 

TDD without pay-off

Post by Phli » Mon, 30 Jun 2003 13:24:46


Quote:> I am not, however, holding my breath.

How about this: Every time Gerold, searching the 'net for evidence against
XP, happens to finds evidence pro-XP, he /also/ posts a link to it here.

DHYB.

--
  Phlip

 
 
 

TDD without pay-off

Post by Gerold Keefe » Mon, 30 Jun 2003 18:58:55


how about this:

not too long ago you titled me
"the guy how made comp.software.extreme-programming useless."
in a yahoo group.
my "evidence pro-XP" seems not to convince everyone and the
reason for this is that facts are immune to propaganda.

just try again phlip. you certainly make it by iteration 8593.

regards,

gerold


> > I am not, however, holding my breath.

> How about this: Every time Gerold, searching the 'net for evidence against
> XP, happens to finds evidence pro-XP, he /also/ posts a link to it here.

> DHYB.

> --
>   Phlip

 
 
 

TDD without pay-off

Post by Gerold Keefe » Mon, 30 Jun 2003 19:07:14



> Here's a radical idea...if you don't like developing test first, then don't.
> Don't add any tests at all to your code.  Nobody is forcing you.

from a theoretical point of few your radical (although not really radical)
idea is right. in practice, however, mis-information influences even those
who know the truth.

i will give you example:
after the presentation of my paper "extreme
programming considered harmful for reliable software development"
at the CONQUEST conference last yeast somebody approached me
and reported me about a project where people *where forced* to do
pair programming, dispite their distaste,  based on the assumption that
"hype makes it right".

hype does not make it right!

best regards,

gerold

 
 
 

TDD without pay-off

Post by Gerold Keefe » Mon, 30 Jun 2003 19:12:33



> I believe this is the same researcher as the pair programming
> study you brought up earlier. She seems to have a plan to
> study the components in isolation, and then put them together
> later.

a research principle called "bottom-up analysis" is based upon the
idea that you start the investigation with parts of a problem.
this principle had some success rate in the past.

Quote:> I'm quite looking forward to how she does that, given her
> disclaimer quoted above. It should be interesting.

> I'm still waiting for a credible academic study of test
> driven development, with all practices in place and
> well practiced before starting out.

i am still waiting for a credible project with all practices in place
and well pracitced until the successful end of a project.

Quote:> I am not, however, holding my breath.

> John Roth

regards,

gerold

 
 
 

TDD without pay-off

Post by David Lightston » Mon, 30 Jun 2003 21:05:39





> > hello *,

> > for an interesting experiment on TDD or test-first, have a look at
> > http://www.ipd.uka.de/KarHPFn/papers/ease02.pdf .
> > you will read at the bottom:

> > "Test-first pays off only slightly in terms of increased reliability.
> > In fact, there were five programs developed with test-first with a
> > reliability over 96% compared to one program in the control group.
> > But this result is blurred by the large variance of the data-points.
> > Concentrating on the program versions after the implementation-phase,
> > the result just turns around. The test-first group has signifcantly
> > less reliable programs than the control group. So far, we do not know,
> > if this effect is caused by a false sense of security, less importance
> > of the acceptance-test for the test-first group, or if it is quite
> > simply a result of too little testing."

> > the study implies that it is at least unclear whether test-first is
> > benificial,
> > but certainly the whole XP consulting crowd sells it - on good trust.

> > regards,

> > gerold

> The researcher says specifically:

> {begin quote]
> "One of the challenges
> of studying test-first is its embedding within XP. This
> embedding makes it difficult to show the effects of test-first
> without being blurred by other practices such as pair programming
> or a simple design. A solution to this problem would be an
> experiment in which XP is applied twice: with test-first and
> without test-first. But this kind of experiment is too difficult
> and too expensive. To solve this problem, test-first was
> extracted from XP and evaluated on its own.

> [end quote]

> I believe this is the same researcher as the pair programming
> study you brought up earlier. She seems to have a plan to
> study the components in isolation, and then put them together
> later.

> I'm quite looking forward to how she does that, given her
> disclaimer quoted above. It should be interesting.

> I'm still waiting for a credible academic study of test
> driven development, with all practices in place and
> well practiced before starting out.

> I am not, however, holding my breath.

I find it rather amusing,  practices that alledgedly serves to create
testable software, are difficult to test

- Show quoted text -

Quote:

> John Roth

 
 
 

TDD without pay-off

Post by Michael Feather » Mon, 30 Jun 2003 21:11:06




> > I believe this is the same researcher as the pair programming
> > study you brought up earlier. She seems to have a plan to
> > study the components in isolation, and then put them together
> > later.

> a research principle called "bottom-up analysis" is based upon the
> idea that you start the investigation with parts of a problem.
> this principle had some success rate in the past.

It's a decent approach, but if you aren't careful you end up measuring
something that never occurs.

Frankly, I like the standards of their research.  They were careful not to
draw conclusions based on it.  At first I was looking at their experiment
design and thinking, "boy, how will they be able to say anything at all?"
In a case like that, it makes you wonder whether it should be published.
But, on the other hand, there is no downside really.  It'll spur people to
do more research.  Sure, some people will misundertand it and misrepresent
the results, but most of those people post the URL too, so no harm done. :-)

Quote:> i am still waiting for a credible project with all practices in place
> and well pracitced until the successful end of a project.

Well, that's tricky, because project ending doesn't mean much.  First of
all, it is infrequent.  Successful projects just go on and on.  They deliver
and get new requests for features, etc.  That's been the case with all of
the teams I've worked with that started to fire on all cylinders.

Michael Feathers
www.objectmentor.com

 
 
 

TDD without pay-off

Post by Michael Feather » Mon, 30 Jun 2003 21:24:59







> > > hello *,

> > > for an interesting experiment on TDD or test-first, have a look at
> > > http://www.ipd.uka.de/KarHPFn/papers/ease02.pdf .
> > > you will read at the bottom:

> > > "Test-first pays off only slightly in terms of increased reliability.
> > > In fact, there were five programs developed with test-first with a
> > > reliability over 96% compared to one program in the control group.
> > > But this result is blurred by the large variance of the data-points.
> > > Concentrating on the program versions after the implementation-phase,
> > > the result just turns around. The test-first group has signifcantly
> > > less reliable programs than the control group. So far, we do not know,
> > > if this effect is caused by a false sense of security, less importance
> > > of the acceptance-test for the test-first group, or if it is quite
> > > simply a result of too little testing."

> > > the study implies that it is at least unclear whether test-first is
> > > benificial,
> > > but certainly the whole XP consulting crowd sells it - on good trust.

> > > regards,

> > > gerold

> > The researcher says specifically:

> > {begin quote]
> > "One of the challenges
> > of studying test-first is its embedding within XP. This
> > embedding makes it difficult to show the effects of test-first
> > without being blurred by other practices such as pair programming
> > or a simple design. A solution to this problem would be an
> > experiment in which XP is applied twice: with test-first and
> > without test-first. But this kind of experiment is too difficult
> > and too expensive. To solve this problem, test-first was
> > extracted from XP and evaluated on its own.

> > [end quote]

> > I believe this is the same researcher as the pair programming
> > study you brought up earlier. She seems to have a plan to
> > study the components in isolation, and then put them together
> > later.

> > I'm quite looking forward to how she does that, given her
> > disclaimer quoted above. It should be interesting.

> > I'm still waiting for a credible academic study of test
> > driven development, with all practices in place and
> > well practiced before starting out.

> > I am not, however, holding my breath.

> I find it rather amusing,  practices that alledgedly serves to create
> testable software, are difficult to test

Well, it is a step forward.  Testing only the things that are easy to test
and hard to do, doesn't do anyone much good.

Michael Feathers
www.objectmentor.com

 
 
 

TDD without pay-off

Post by David Lightston » Mon, 30 Jun 2003 23:15:20









> > > > hello *,

> > > > for an interesting experiment on TDD or test-first, have a look at
> > > > http://www.ipd.uka.de/KarHPFn/papers/ease02.pdf .
> > > > you will read at the bottom:

> > > > "Test-first pays off only slightly in terms of increased
reliability.
> > > > In fact, there were five programs developed with test-first with a
> > > > reliability over 96% compared to one program in the control group.
> > > > But this result is blurred by the large variance of the data-points.
> > > > Concentrating on the program versions after the

implementation-phase,

- Show quoted text -

Quote:> > > > the result just turns around. The test-first group has signifcantly
> > > > less reliable programs than the control group. So far, we do not
know,
> > > > if this effect is caused by a false sense of security, less
importance
> > > > of the acceptance-test for the test-first group, or if it is quite
> > > > simply a result of too little testing."

> > > > the study implies that it is at least unclear whether test-first is
> > > > benificial,
> > > > but certainly the whole XP consulting crowd sells it - on good
trust.

> > > > regards,

> > > > gerold

> > > The researcher says specifically:

> > > {begin quote]
> > > "One of the challenges
> > > of studying test-first is its embedding within XP. This
> > > embedding makes it difficult to show the effects of test-first
> > > without being blurred by other practices such as pair programming
> > > or a simple design. A solution to this problem would be an
> > > experiment in which XP is applied twice: with test-first and
> > > without test-first. But this kind of experiment is too difficult
> > > and too expensive. To solve this problem, test-first was
> > > extracted from XP and evaluated on its own.

> > > [end quote]

> > > I believe this is the same researcher as the pair programming
> > > study you brought up earlier. She seems to have a plan to
> > > study the components in isolation, and then put them together
> > > later.

> > > I'm quite looking forward to how she does that, given her
> > > disclaimer quoted above. It should be interesting.

> > > I'm still waiting for a credible academic study of test
> > > driven development, with all practices in place and
> > > well practiced before starting out.

> > > I am not, however, holding my breath.

> > I find it rather amusing,  practices that alledgedly serves to create
> > testable software, are difficult to test

> Well, it is a step forward.  Testing only the things that are easy to test
> and hard to do, doesn't do anyone much good.

A better step would be to make the practices testable

- Show quoted text -

Quote:

> Michael Feathers
> www.objectmentor.com

 
 
 

TDD without pay-off

Post by Michael Feather » Tue, 01 Jul 2003 00:33:02






message





> > > > > hello *,

> > > > > for an interesting experiment on TDD or test-first, have a look at
> > > > > http://www.ipd.uka.de/KarHPFn/papers/ease02.pdf .
> > > > > you will read at the bottom:

> > > > > "Test-first pays off only slightly in terms of increased
> reliability.
> > > > > In fact, there were five programs developed with test-first with a
> > > > > reliability over 96% compared to one program in the control group.
> > > > > But this result is blurred by the large variance of the
data-points.
> > > > > Concentrating on the program versions after the
> implementation-phase,
> > > > > the result just turns around. The test-first group has
signifcantly
> > > > > less reliable programs than the control group. So far, we do not
> know,
> > > > > if this effect is caused by a false sense of security, less
> importance
> > > > > of the acceptance-test for the test-first group, or if it is quite
> > > > > simply a result of too little testing."

> > > > > the study implies that it is at least unclear whether test-first
is
> > > > > benificial,
> > > > > but certainly the whole XP consulting crowd sells it - on good
> trust.

> > > > > regards,

> > > > > gerold

> > > > The researcher says specifically:

> > > > {begin quote]
> > > > "One of the challenges
> > > > of studying test-first is its embedding within XP. This
> > > > embedding makes it difficult to show the effects of test-first
> > > > without being blurred by other practices such as pair programming
> > > > or a simple design. A solution to this problem would be an
> > > > experiment in which XP is applied twice: with test-first and
> > > > without test-first. But this kind of experiment is too difficult
> > > > and too expensive. To solve this problem, test-first was
> > > > extracted from XP and evaluated on its own.

> > > > [end quote]

> > > > I believe this is the same researcher as the pair programming
> > > > study you brought up earlier. She seems to have a plan to
> > > > study the components in isolation, and then put them together
> > > > later.

> > > > I'm quite looking forward to how she does that, given her
> > > > disclaimer quoted above. It should be interesting.

> > > > I'm still waiting for a credible academic study of test
> > > > driven development, with all practices in place and
> > > > well practiced before starting out.

> > > > I am not, however, holding my breath.

> > > I find it rather amusing,  practices that alledgedly serves to create
> > > testable software, are difficult to test

> > Well, it is a step forward.  Testing only the things that are easy to
test
> > and hard to do, doesn't do anyone much good.

> A better step would be to make the practices testable

A better step would be to try them and form your own conclusions.

Development methods are notoriously difficult to test.  Like everything else
in an economy, there is a large opportunity cost for doing that testing, so
generally people weigh risks, apply judgement and try things.

I don't know anyone offhand who only uses practices that they've found good
experimental results for in academic journals.  In fact, I suspect you might
even use some practices that don't have that basis.  Maybe I am wrong.

Michael Feathers
www.objectmentor.com

 
 
 

TDD without pay-off

Post by David Lightston » Tue, 01 Jul 2003 00:57:32








> message





> > > > > > hello *,

> > > > > > for an interesting experiment on TDD or test-first, have a look
at
> > > > > > http://www.ipd.uka.de/KarHPFn/papers/ease02.pdf .
> > > > > > you will read at the bottom:

> > > > > > "Test-first pays off only slightly in terms of increased
> > reliability.
> > > > > > In fact, there were five programs developed with test-first with
a
> > > > > > reliability over 96% compared to one program in the control
group.
> > > > > > But this result is blurred by the large variance of the
> data-points.
> > > > > > Concentrating on the program versions after the
> > implementation-phase,
> > > > > > the result just turns around. The test-first group has
> signifcantly
> > > > > > less reliable programs than the control group. So far, we do not
> > know,
> > > > > > if this effect is caused by a false sense of security, less
> > importance
> > > > > > of the acceptance-test for the test-first group, or if it is
quite
> > > > > > simply a result of too little testing."

> > > > > > the study implies that it is at least unclear whether test-first
> is
> > > > > > benificial,
> > > > > > but certainly the whole XP consulting crowd sells it - on good
> > trust.

> > > > > > regards,

> > > > > > gerold

> > > > > The researcher says specifically:

> > > > > {begin quote]
> > > > > "One of the challenges
> > > > > of studying test-first is its embedding within XP. This
> > > > > embedding makes it difficult to show the effects of test-first
> > > > > without being blurred by other practices such as pair programming
> > > > > or a simple design. A solution to this problem would be an
> > > > > experiment in which XP is applied twice: with test-first and
> > > > > without test-first. But this kind of experiment is too difficult
> > > > > and too expensive. To solve this problem, test-first was
> > > > > extracted from XP and evaluated on its own.

> > > > > [end quote]

> > > > > I believe this is the same researcher as the pair programming
> > > > > study you brought up earlier. She seems to have a plan to
> > > > > study the components in isolation, and then put them together
> > > > > later.

> > > > > I'm quite looking forward to how she does that, given her
> > > > > disclaimer quoted above. It should be interesting.

> > > > > I'm still waiting for a credible academic study of test
> > > > > driven development, with all practices in place and
> > > > > well practiced before starting out.

> > > > > I am not, however, holding my breath.

> > > > I find it rather amusing,  practices that alledgedly serves to
create
> > > > testable software, are difficult to test

> > > Well, it is a step forward.  Testing only the things that are easy to
> test
> > > and hard to do, doesn't do anyone much good.

> > A better step would be to make the practices testable

> A better step would be to try them and form your own conclusions.

> Development methods are notoriously difficult to test.

That is why they are called Methods, rather than Methodologies.

Quote:> Like everything else
> in an economy, there is a large opportunity cost for doing that testing,
so
> generally people weigh risks, apply judgement and try things.

That is what people do. It is indeed a very individual type of thing. I am
often surprised by the frequency with this the success of an individual is
used to justify the applicability of the results to others.

Quote:

> I don't know anyone offhand who only uses practices that they've found
good
> experimental results for in academic journals. In fact, I suspect you
might
> even use some practices that don't have that basis.  Maybe I am wrong.

I can assure you that you are correct about my using practices which lack
experimental validity evidence. When you see me advocating the universal
adoption of those practices, do let me know

- Show quoted text -

Quote:

> Michael Feathers
> www.objectmentor.com

 
 
 

TDD without pay-off

Post by John Rot » Tue, 01 Jul 2003 02:23:06








> > > hello *,

> > > for an interesting experiment on TDD or test-first, have a look at
> > > http://www.ipd.uka.de/KarHPFn/papers/ease02.pdf .
> > > you will read at the bottom:

> > > "Test-first pays off only slightly in terms of increased
reliability.
> > > In fact, there were five programs developed with test-first with a
> > > reliability over 96% compared to one program in the control group.
> > > But this result is blurred by the large variance of the
data-points.
> > > Concentrating on the program versions after the

implementation-phase,

- Show quoted text -

Quote:> > > the result just turns around. The test-first group has
signifcantly
> > > less reliable programs than the control group. So far, we do not
know,
> > > if this effect is caused by a false sense of security, less
importance
> > > of the acceptance-test for the test-first group, or if it is quite
> > > simply a result of too little testing."

> > > the study implies that it is at least unclear whether test-first
is
> > > benificial,
> > > but certainly the whole XP consulting crowd sells it - on good
trust.

> > > regards,

> > > gerold

> > The researcher says specifically:

> > {begin quote]
> > "One of the challenges
> > of studying test-first is its embedding within XP. This
> > embedding makes it difficult to show the effects of test-first
> > without being blurred by other practices such as pair programming
> > or a simple design. A solution to this problem would be an
> > experiment in which XP is applied twice: with test-first and
> > without test-first. But this kind of experiment is too difficult
> > and too expensive. To solve this problem, test-first was
> > extracted from XP and evaluated on its own.

> > [end quote]

> > I believe this is the same researcher as the pair programming
> > study you brought up earlier. She seems to have a plan to
> > study the components in isolation, and then put them together
> > later.

> > I'm quite looking forward to how she does that, given her
> > disclaimer quoted above. It should be interesting.

> > I'm still waiting for a credible academic study of test
> > driven development, with all practices in place and
> > well practiced before starting out.

> > I am not, however, holding my breath.

> I find it rather amusing,  practices that alledgedly serve to create
> testable software, are difficult to test

I wouldn't say difficult. I'd say expensive. In the referenced
experiments, the experimenter specifically mentioned the
expense factor for doing a complete test with all practices
in place. It's not trivial, and it's way beyond the budget
that most academic departments have availible for research.

To speculatively quantify an experiment: let's assume that
it goes for ten four-hour work sessions, with ten developers
in each group, spread over two weeks. Each developer
is required to have at least one year production
experiance with the technologies involved. At 50USD per
hour, that starts out at $40,000 just to pay the subjects.
You also have to set up the experimental facility, create
and validate the problem and acceptance tests, and
evaluate the results. I'd be surprised if such an experiment
would come in at much under 100,000USD.

Properly designed, such an experiment would be
quite convincing, which the current spate of experiments
are not.

Where's John Veris verTipton when you need him?

John Roth

- Show quoted text -

Quote:

> > John Roth

 
 
 

1. Feature Driven Development - Evidence for pay-off

Is there any hard evidence for the pay-offs which can be achieved with
feature driven development. I know the weh-site
www.featuredrivendevelopment.com and read
the articles from R.Palmer, etc. But I am looking for additional case
studies, success stories etc.

cheers,
thomas

2. PLI-80 applications guide

3. tdd without main?

4. video CD/MPEG1 encoding speeds PB G3/500 vs desktops eg new dual G4/800 wanted

5. Pay OFF your debts NOW! EASILY!

6. Fieldbus protocols

7. Pay Off Christmas Debts

8. db22 from devcon 5 doesn't work with warp

9. Does OOP, C++ really pay off?

10. Acquiring hotfixes without having to pay for them.

11. Power off without exiting

12. Recording wave without taking "off hook"

13. profiler and off topic - off topic posts.