TDD without pay-off

TDD without pay-off

Post by David Lightston » Tue, 01 Jul 2003 04:33:30










> > > > hello *,

> > > > for an interesting experiment on TDD or test-first, have a look at
> > > > http://www.ipd.uka.de/KarHPFn/papers/ease02.pdf .
> > > > you will read at the bottom:

> > > > "Test-first pays off only slightly in terms of increased
> reliability.
> > > > In fact, there were five programs developed with test-first with a
> > > > reliability over 96% compared to one program in the control group.
> > > > But this result is blurred by the large variance of the
> data-points.
> > > > Concentrating on the program versions after the
> implementation-phase,
> > > > the result just turns around. The test-first group has
> signifcantly
> > > > less reliable programs than the control group. So far, we do not
> know,
> > > > if this effect is caused by a false sense of security, less
> importance
> > > > of the acceptance-test for the test-first group, or if it is quite
> > > > simply a result of too little testing."

> > > > the study implies that it is at least unclear whether test-first
> is
> > > > benificial,
> > > > but certainly the whole XP consulting crowd sells it - on good
> trust.

> > > > regards,

> > > > gerold

> > > The researcher says specifically:

> > > {begin quote]
> > > "One of the challenges
> > > of studying test-first is its embedding within XP. This
> > > embedding makes it difficult to show the effects of test-first
> > > without being blurred by other practices such as pair programming
> > > or a simple design. A solution to this problem would be an
> > > experiment in which XP is applied twice: with test-first and
> > > without test-first. But this kind of experiment is too difficult
> > > and too expensive. To solve this problem, test-first was
> > > extracted from XP and evaluated on its own.

> > > [end quote]

> > > I believe this is the same researcher as the pair programming
> > > study you brought up earlier. She seems to have a plan to
> > > study the components in isolation, and then put them together
> > > later.

> > > I'm quite looking forward to how she does that, given her
> > > disclaimer quoted above. It should be interesting.

> > > I'm still waiting for a credible academic study of test
> > > driven development, with all practices in place and
> > > well practiced before starting out.

> > > I am not, however, holding my breath.

> > I find it rather amusing,  practices that alledgedly serve to create
> > testable software, are difficult to test

> I wouldn't say difficult. I'd say expensive. In the referenced
> experiments, the experimenter specifically mentioned the
> expense factor for doing a complete test with all practices
> in place. It's not trivial, and it's way beyond the budget
> that most academic departments have availible for research.

You may call it expensive if you wish, I definitely ment Difficult. Clearly
the developers did not consider its formulation as an example of  TDD

- Show quoted text -

Quote:

> > > John Roth

 
 
 

TDD without pay-off

Post by Greg Bac » Tue, 01 Jul 2003 06:02:36






:
: > A better step would be to make the practices testable
:
: A better step would be to try them and form your own conclusions.
:
: Development methods are notoriously difficult to test.  Like
: everything else in an economy, there is a large opportunity cost for
: doing that testing, so generally people weigh risks, apply judgement
: and try things.

Austrian school economists reject what they call mathematization of
economics -- e.g., the construction of  economic models -- because they
claim the resulting formulae are insufficient oversimplifications and,
thus, poor analytical tools.  For more, see "Why Austrian Economics
Matters" by Lew Rockwell (http://www.mises.org/why_ae.asp).

In *The Ultimate Foundation of Economic Science*, Ludwig von Mises wrote

    . . . no scientific method can succeed in determining
    how definite external events, liable to a description
    by the methods of the natural sciences, produce within
    the human mind definite ideas, value judgments, and
    volitions.  In this sense the individual that cannot
    be dissolved into components is both the starting point
    and the ultimate given of all endeavors to deal with
    human action.

    http://www.mises.org/ufofes/ch5~5.asp

In other words, some scholars would say the effectiveness of software
methods and processes are *impossible* to test in general, which means
that any given method won't work for everyone.

: [...]

I have no problem admitting the appeal of TDD and other XP practices
comes from my subjective value judgments such as FINALLY having a
backscratcher to reach those pesky itches of developing for years with
no feedback, integration nightmares, terrible testing, schedules plucked
from the air, etc.

Greg
--
These days it's slightly different.  You give the Web site your privacy
and your spam-vulnerable email address ... it gives you a cookie.  What
is it good for?  Don't worry about it.  Good user, have a cookie.
    -- Internet Oracularities 1308-01

 
 
 

TDD without pay-off

Post by Michael Feather » Tue, 01 Jul 2003 01:20:11






message




> > message





> > > > > > > hello *,

> > > > > > > for an interesting experiment on TDD or test-first, have a
look
> at
> > > > > > > http://www.ipd.uka.de/KarHPFn/papers/ease02.pdf .
> > > > > > > you will read at the bottom:

> > > > > > > "Test-first pays off only slightly in terms of increased
> > > reliability.
> > > > > > > In fact, there were five programs developed with test-first
with
> a
> > > > > > > reliability over 96% compared to one program in the control
> group.
> > > > > > > But this result is blurred by the large variance of the
> > data-points.
> > > > > > > Concentrating on the program versions after the
> > > implementation-phase,
> > > > > > > the result just turns around. The test-first group has
> > signifcantly
> > > > > > > less reliable programs than the control group. So far, we do
not
> > > know,
> > > > > > > if this effect is caused by a false sense of security, less
> > > importance
> > > > > > > of the acceptance-test for the test-first group, or if it is
> quite
> > > > > > > simply a result of too little testing."

> > > > > > > the study implies that it is at least unclear whether
test-first
> > is
> > > > > > > benificial,
> > > > > > > but certainly the whole XP consulting crowd sells it - on good
> > > trust.

> > > > > > > regards,

> > > > > > > gerold

> > > > > > The researcher says specifically:

> > > > > > {begin quote]
> > > > > > "One of the challenges
> > > > > > of studying test-first is its embedding within XP. This
> > > > > > embedding makes it difficult to show the effects of test-first
> > > > > > without being blurred by other practices such as pair
programming
> > > > > > or a simple design. A solution to this problem would be an
> > > > > > experiment in which XP is applied twice: with test-first and
> > > > > > without test-first. But this kind of experiment is too difficult
> > > > > > and too expensive. To solve this problem, test-first was
> > > > > > extracted from XP and evaluated on its own.

> > > > > > [end quote]

> > > > > > I believe this is the same researcher as the pair programming
> > > > > > study you brought up earlier. She seems to have a plan to
> > > > > > study the components in isolation, and then put them together
> > > > > > later.

> > > > > > I'm quite looking forward to how she does that, given her
> > > > > > disclaimer quoted above. It should be interesting.

> > > > > > I'm still waiting for a credible academic study of test
> > > > > > driven development, with all practices in place and
> > > > > > well practiced before starting out.

> > > > > > I am not, however, holding my breath.

> > > > > I find it rather amusing,  practices that alledgedly serves to
> create
> > > > > testable software, are difficult to test

> > > > Well, it is a step forward.  Testing only the things that are easy
to
> > test
> > > > and hard to do, doesn't do anyone much good.

> > > A better step would be to make the practices testable

> > A better step would be to try them and form your own conclusions.

> > Development methods are notoriously difficult to test.

> That is why they are called Methods, rather than Methodologies.

> > Like everything else
> > in an economy, there is a large opportunity cost for doing that testing,
> so
> > generally people weigh risks, apply judgement and try things.

> That is what people do. It is indeed a very individual type of thing. I am
> often surprised by the frequency with this the success of an individual is
> used to justify the applicability of the results to others.

I don't know of any cases off-hand.  Usually, someone tries something and if
works for them, they mention it to other people and they see if it works for
them, so it is a matter of a group predicting applicability based on their
experiences.

Quote:> > I don't know anyone offhand who only uses practices that they've found
> good
> > experimental results for in academic journals. In fact, I suspect you
> might
> > even use some practices that don't have that basis.  Maybe I am wrong.

> I can assure you that you are correct about my using practices which lack
> experimental validity evidence. When you see me advocating the universal
> adoption of those practices, do let me know

Actually, would you let me know?  If you feel confident about some practice
that I haven't tried.  Let me know.  I'd like to try it out.

Michael Feathers
www.objectmentor.com

 
 
 

TDD without pay-off

Post by Michael Feather » Tue, 01 Jul 2003 08:00:48






message

> :
> : > A better step would be to make the practices testable
> :
> : A better step would be to try them and form your own conclusions.
> :
> : Development methods are notoriously difficult to test.  Like
> : everything else in an economy, there is a large opportunity cost for
> : doing that testing, so generally people weigh risks, apply judgement
> : and try things.

> Austrian school economists reject what they call mathematization of
> economics -- e.g., the construction of  economic models -- because they
> claim the resulting formulae are insufficient oversimplifications and,
> thus, poor analytical tools.  For more, see "Why Austrian Economics
> Matters" by Lew Rockwell (http://www.mises.org/why_ae.asp).

> In *The Ultimate Foundation of Economic Science*, Ludwig von Mises wrote

>     . . . no scientific method can succeed in determining
>     how definite external events, liable to a description
>     by the methods of the natural sciences, produce within
>     the human mind definite ideas, value judgments, and
>     volitions.  In this sense the individual that cannot
>     be dissolved into components is both the starting point
>     and the ultimate given of all endeavors to deal with
>     human action.

>     http://www.mises.org/ufofes/ch5~5.asp

> In other words, some scholars would say the effectiveness of software
> methods and processes are *impossible* to test in general, which means
> that any given method won't work for everyone.

> : [...]

> I have no problem admitting the appeal of TDD and other XP practices
> comes from my subjective value judgments such as FINALLY having a
> backscratcher to reach those pesky itches of developing for years with
> no feedback, integration nightmares, terrible testing, schedules plucked
> from the air, etc.

Greg, thanks for the references.  Memories of my libertarian youth came
rushing back when I read Mises' name.

Michael Feathers
www.objectmentor.com

 
 
 

TDD without pay-off

Post by Ilja Preu » Tue, 01 Jul 2003 22:51:35




>> somebody approached me
>> and reported me about a project where people *where forced* to do
>> pair programming, dispite their distaste,  based on the assumption
>> that "hype makes it right".

> That sounds more like a management/people issue than an extreme
> programming problem.

Specifically, all the established XP proponents I know of explicitely state
that a team needs to *choose* and *own* its process to be effective.
 
 
 

TDD without pay-off

Post by Greg Bac » Wed, 02 Jul 2003 00:24:07




: Greg, thanks for the references.  Memories of my libertarian youth
: came rushing back when I read Mises' name.

It could be worse.  At least your own publications don't criticize
you. :-)  See, for example

    http://www.gold-eagle.com/greenspan041998.html

Greg
--
Freedom of men under government is . . . a liberty to follow my own will
in all things, when the rule prescribes not, and not to be subject to the
inconstant, unknown, arbitrary will of another man.
    -- John Locke

 
 
 

TDD without pay-off

Post by Michael Feather » Wed, 02 Jul 2003 01:16:12





> : Greg, thanks for the references.  Memories of my libertarian youth
> : came rushing back when I read Mises' name.

> It could be worse.  At least your own publications don't criticize
> you. :-)  See, for example

>     http://www.gold-eagle.com/greenspan041998.html

Haha!  But he has learned.  IMHO, Greenspan is the best president we've
recently had in this country :-)

Michael