Multi-user Test Scripts

Multi-user Test Scripts

Post by Don Magi » Fri, 03 Jul 1998 04:00:00



Your point is well taken, and my comment 'for the most part' I believe
covers the issue you describe.

Typically, I would utilize a regression test to understand the timing
information as it relates to the workstation specific information, and
other regression issues.  

The scripting of a load test should also accomodate (and a good capture
facility accomodates this) the timing of events that were experienced
during the initial capture and allow flexibility to accomodate other
'unknown' events such as waiting for a file close or a flag to be set, etc.

Your comments and the original request do involve features of a well
designed load test tool, and in my opinion this requires more than just a
record playback tool either for load testing or regression testing.
Compuware's QALoad can handle these issues very nicely.

Finally, Compuware's Xpediter/SQL has full debugging support for Stored
procedures including breakpoints, variable watches, stepping and much more
for stored procedures which would often accomodate these types of
activities on the server side.  QA required for similar events on the
client side can be handled by QARun, or in some cases by QALoad.

--
Kind Regards,
Don Magie

(My opinion is personal, my employer is Compuware)

 
 
 

Multi-user Test Scripts

Post by Jim Haze » Sat, 04 Jul 1998 04:00:00


Don & Joel,
        I keep seeing the term "capture" in the explanation for doing
multiple machine and concurrency testing.  If you are "capturing" or
"recording" your actions (using keyboard/mouse capture/playback) and
using them to run the tests I think you need to rethink your strategy
(correct me if I am wrong).  If you are using just capture/playback
then you are not getting the full benefit of the tool and are wasting
your time trying to automate these types of tests.
        Script development is the only way to really do multiple machine
and concurrency testing.  This is pure programming, which means building
in wait loops and other conditional checks into the test.  This will
help eliminate the latency and timing errors of the script running
against the application under test, this is especially true in Client/
Server type applications (like an accounting program running on SQL
Server and Windows).
        On an OS/2 project we had to test out a branch automation
system that could have up to 20 clients running against an OS/2 server
and DB2/2.  By using a programmatic approach I was able to run
multiple tests on multiple machines at the same time.  I had Master/
Executive machine running all the Clients/Agents.  For concurrency tests
I could grab control of as many machines as I wanted and send the same
actions to all of them and time out the latency for each machine to
grab a particular record or range of records.  This could also be
setup to do a performance test for transactions per hour with the agent
machines performing a transaction at timed intervals, we could see how
much of a load it would handle before it started dropping them.
        Enough bragging, look at some of the latest articles at STLabs
website under the Testers Network.  Also, look at the Softbridge web
site and its white papers on how Automated Test Facility (ATF) has
done this job.  Segue also has white papers on this type of tesing.
        Good luck.

        Jim Hazen
        Veteran of the Automation war's


> Your point is well taken, and my comment 'for the most part' I believe
> covers the issue you describe.

> Typically, I would utilize a regression test to understand the timing
> information as it relates to the workstation specific information, and
> other regression issues.

> The scripting of a load test should also accomodate (and a good capture
> facility accomodates this) the timing of events that were experienced
> during the initial capture and allow flexibility to accomodate other
> 'unknown' events such as waiting for a file close or a flag to be set, etc.

> Your comments and the original request do involve features of a well
> designed load test tool, and in my opinion this requires more than just a
> record playback tool either for load testing or regression testing.
> Compuware's QALoad can handle these issues very nicely.

> Finally, Compuware's Xpediter/SQL has full debugging support for Stored
> procedures including breakpoints, variable watches, stepping and much more
> for stored procedures which would often accomodate these types of
> activities on the server side.  QA required for similar events on the
> client side can be handled by QARun, or in some cases by QALoad.

> --
> Kind Regards,
> Don Magie

> (My opinion is personal, my employer is Compuware)


 
 
 

Multi-user Test Scripts

Post by AnimaLvr9 » Mon, 06 Jul 1998 04:00:00


Jim:

You have described what a well designed automated load testing tool will
do...and Compuware's QALoad will do this!  Additionally, Compuware offers
monitoring tools that will allow you to correlate response times with such
performance metrics as memory utilization, cpu, database performance,
network performance...  On top of QALoad's robust testing features, the
reporting and graphing is outstanding!


>Don & Joel,
> I keep seeing the term "capture" in the explanation for doing
>multiple machine and concurrency testing.  If you are "capturing" or
>"recording" your actions (using keyboard/mouse capture/playback) and
>using them to run the tests I think you need to rethink your strategy
>(correct me if I am wrong).  If you are using just capture/playback
>then you are not getting the full benefit of the tool and are wasting
>your time trying to automate these types of tests.
> Script development is the only way to really do multiple machine
>and concurrency testing.  This is pure programming, which means building
>in wait loops and other conditional checks into the test.  This will
>help eliminate the latency and timing errors of the script running
>against the application under test, this is especially true in Client/
>Server type applications (like an accounting program running on SQL
>Server and Windows).
> On an OS/2 project we had to test out a branch automation
>system that could have up to 20 clients running against an OS/2 server
>and DB2/2.  By using a programmatic approach I was able to run
>multiple tests on multiple machines at the same time.  I had Master/
>Executive machine running all the Clients/Agents.  For concurrency tests
>I could grab control of as many machines as I wanted and send the same
>actions to all of them and time out the latency for each machine to
>grab a particular record or range of records.  This could also be
>setup to do a performance test for transactions per hour with the agent
>machines performing a transaction at timed intervals, we could see how
>much of a load it would handle before it started dropping them.
> Enough bragging, look at some of the latest articles at STLabs
>website under the Testers Network.  Also, look at the Softbridge web
>site and its white papers on how Automated Test Facility (ATF) has
>done this job.  Segue also has white papers on this type of tesing.
> Good luck.

> Jim Hazen
> Veteran of the Automation war's


>> Your point is well taken, and my comment 'for the most part' I believe
>> covers the issue you describe.

>> Typically, I would utilize a regression test to understand the timing
>> information as it relates to the workstation specific information, and
>> other regression issues.

>> The scripting of a load test should also accomodate (and a good capture
>> facility accomodates this) the timing of events that were experienced
>> during the initial capture and allow flexibility to accomodate other
>> 'unknown' events such as waiting for a file close or a flag to be set,
etc.

>> Your comments and the original request do involve features of a well
>> designed load test tool, and in my opinion this requires more than just a
>> record playback tool either for load testing or regression testing.
>> Compuware's QALoad can handle these issues very nicely.

>> Finally, Compuware's Xpediter/SQL has full debugging support for Stored
>> procedures including breakpoints, variable watches, stepping and much
more
>> for stored procedures which would often accomodate these types of
>> activities on the server side.  QA required for similar events on the
>> client side can be handled by QARun, or in some cases by QALoad.

>> --
>> Kind Regards,
>> Don Magie

>> (My opinion is personal, my employer is Compuware)

 
 
 

Multi-user Test Scripts

Post by Jim Haze » Tue, 07 Jul 1998 04:00:00


AnimalLvr99,
        Yes, I know.  I was using ATF 5 years ago to do a lot of that
type of testing on an OS/2 Client/Server project.  I didn't have a
"Load" testing tool so I built it myself (with input from other users
on the same types of projects).  
        I had to string together other tools (dB performance monitor's,
LAN traffic sniffers, Lan Management Utilities, etc.) to accomplish the
same type of job that the newer Load testing tools now have.  Feel
lucky, because you get alot of different functionality in the tools
today in comparison to 5 or more years ago.  
        My point being made here is do not rely on the Capture/Playback
functionality.  Automation has to be treated as a development type
project unto itself.  There is way too much marketing hype by the
Sales & Marketing Wonks out there.  I have seen two previous automation
projects fail because upper management was convinced that capture/play-
back was the way to go.  If done right the testing libraries are
easier to maintain, more robust, and have a higher ROI.
        Automation cannot do everything, but if it is designed and
employed properly it can do a alot.  It just grates on me when I
see and hear "Capture/Playback" or "Record & Run".  It is throwaway
work, and not what automation tools (any of them) were designed for.
        Pardon me if I am coming down * you, but this is one
area of QA/Testing that I take very seriously and I just hate to see
other people in this line of work having to put up with or make some
of the same mistakes I have made.  Eleven plus years in the PC QA/Test
arena (I remember DOS 2.11 and the XT) does kinda warp ya.

        Regards,
                Jim


> Jim:

> You have described what a well designed automated load testing tool will
> do...and Compuware's QALoad will do this!  Additionally, Compuware offers
> monitoring tools that will allow you to correlate response times with such
> performance metrics as memory utilization, cpu, database performance,
> network performance...  On top of QALoad's robust testing features, the
> reporting and graphing is outstanding!


> >Don & Joel,
> > I keep seeing the term "capture" in the explanation for doing
> >multiple machine and concurrency testing.  If you are "capturing" or
> >"recording" your actions (using keyboard/mouse capture/playback) and
> >using them to run the tests I think you need to rethink your strategy
> >(correct me if I am wrong).  If you are using just capture/playback
> >then you are not getting the full benefit of the tool and are wasting
> >your time trying to automate these types of tests.
> > Script development is the only way to really do multiple machine
> >and concurrency testing.  This is pure programming, which means building
> >in wait loops and other conditional checks into the test.  This will
> >help eliminate the latency and timing errors of the script running
> >against the application under test, this is especially true in Client/
> >Server type applications (like an accounting program running on SQL
> >Server and Windows).
> > On an OS/2 project we had to test out a branch automation
> >system that could have up to 20 clients running against an OS/2 server
> >and DB2/2.  By using a programmatic approach I was able to run
> >multiple tests on multiple machines at the same time.  I had Master/
> >Executive machine running all the Clients/Agents.  For concurrency tests
> >I could grab control of as many machines as I wanted and send the same
> >actions to all of them and time out the latency for each machine to
> >grab a particular record or range of records.  This could also be
> >setup to do a performance test for transactions per hour with the agent
> >machines performing a transaction at timed intervals, we could see how
> >much of a load it would handle before it started dropping them.
> > Enough bragging, look at some of the latest articles at STLabs
> >website under the Testers Network.  Also, look at the Softbridge web
> >site and its white papers on how Automated Test Facility (ATF) has
> >done this job.  Segue also has white papers on this type of tesing.
> > Good luck.

> > Jim Hazen
> > Veteran of the Automation war's


> >> Your point is well taken, and my comment 'for the most part' I believe
> >> covers the issue you describe.

> >> Typically, I would utilize a regression test to understand the timing
> >> information as it relates to the workstation specific information, and
> >> other regression issues.

> >> The scripting of a load test should also accomodate (and a good capture
> >> facility accomodates this) the timing of events that were experienced
> >> during the initial capture and allow flexibility to accomodate other
> >> 'unknown' events such as waiting for a file close or a flag to be set,
> etc.

> >> Your comments and the original request do involve features of a well
> >> designed load test tool, and in my opinion this requires more than just a
> >> record playback tool either for load testing or regression testing.
> >> Compuware's QALoad can handle these issues very nicely.

> >> Finally, Compuware's Xpediter/SQL has full debugging support for Stored
> >> procedures including breakpoints, variable watches, stepping and much
> more
> >> for stored procedures which would often accomodate these types of
> >> activities on the server side.  QA required for similar events on the
> >> client side can be handled by QARun, or in some cases by QALoad.

> >> --
> >> Kind Regards,
> >> Don Magie

> >> (My opinion is personal, my employer is Compuware)

 
 
 

Multi-user Test Scripts

Post by Don Magi » Thu, 09 Jul 1998 04:00:00


The purpose of a middleware capture facility (see Compuware's EasyScript
modules) is to perform the majority of this work for you.  

Just like in the capture/playback philosophy of the regression tools, it is
not 100% (unless you are ONLY requesting data from a server/system).  You
typically need to insert variables in place of data items being sent to the
server/system, you need to modify timing checkpoints and you may want to
complicate your scripts in any number of other ways to accomplish your own
personal/business goals.

All of that being said, QALoad which I use almost daily will perform 70-80%
of that work for me in the supported environments (except WinSock, only
about 50-60% there).  Therefore I can have transaction scripted and my
first data generated for a medium complexity environment in about 3-5
hours.

With all that said, you can simply script the bejeepers out of your system,
but you may miss calls which are being made.  An example of this came at a
customer site where a rather complex middleware was being used.  The logon
to the Oracle database seemed simple enough, but there where hundreds
(literally) of middleware calls being made to accomplish that simple feat.

If I hard code an Oracle logon, am I behaving just like the application?
No.   . . . Like everything, no single approach responds to all problems.
Scripting line by line may be appropriate at times, so may a capture
facility.  Neither is the total solution and neither invalidates the other
as a possible approach.
--
Kind Regards,
Don Magie

(My employer is Compuware; My opinion is personal)

(My opinion is personal, my employer is Compuware)



> Don & Joel,
>    I keep seeing the term "capture" in the explanation for doing
> multiple machine and concurrency testing.  If you are "capturing" or
> "recording" your actions (using keyboard/mouse capture/playback) and
> using them to run the tests I think you need to rethink your strategy
> (correct me if I am wrong).  If you are using just capture/playback
> then you are not getting the full benefit of the tool and are wasting
> your time trying to automate these types of tests.
>    Script development is the only way to really do multiple machine
> and concurrency testing.  This is pure programming, which means building
> in wait loops and other conditional checks into the test.  This will
> help eliminate the latency and timing errors of the script running
> against the application under test, this is especially true in Client/
> Server type applications (like an accounting program running on SQL
> Server and Windows).
>    On an OS/2 project we had to test out a branch automation
> system that could have up to 20 clients running against an OS/2 server
> and DB2/2.  By using a programmatic approach I was able to run
> multiple tests on multiple machines at the same time.  I had Master/
> Executive machine running all the Clients/Agents.  For concurrency tests
> I could grab control of as many machines as I wanted and send the same
> actions to all of them and time out the latency for each machine to
> grab a particular record or range of records.  This could also be
> setup to do a performance test for transactions per hour with the agent
> machines performing a transaction at timed intervals, we could see how
> much of a load it would handle before it started dropping them.
>    Enough bragging, look at some of the latest articles at STLabs
> website under the Testers Network.  Also, look at the Softbridge web
> site and its white papers on how Automated Test Facility (ATF) has
> done this job.  Segue also has white papers on this type of tesing.
>    Good luck.

>    Jim Hazen
>    Veteran of the Automation war's


> > Your point is well taken, and my comment 'for the most part' I believe
> > covers the issue you describe.

> > Typically, I would utilize a regression test to understand the timing
> > information as it relates to the workstation specific information, and
> > other regression issues.

> > The scripting of a load test should also accomodate (and a good capture
> > facility accomodates this) the timing of events that were experienced
> > during the initial capture and allow flexibility to accomodate other
> > 'unknown' events such as waiting for a file close or a flag to be set,
etc.

> > Your comments and the original request do involve features of a well
> > designed load test tool, and in my opinion this requires more than just
a
> > record playback tool either for load testing or regression testing.
> > Compuware's QALoad can handle these issues very nicely.

> > Finally, Compuware's Xpediter/SQL has full debugging support for Stored
> > procedures including breakpoints, variable watches, stepping and much
more
> > for stored procedures which would often accomodate these types of
> > activities on the server side.  QA required for similar events on the
> > client side can be handled by QARun, or in some cases by QALoad.

> > --
> > Kind Regards,
> > Don Magie

> > (My opinion is personal, my employer is Compuware)

 
 
 

1. Beta Testing Opportunity (multi-user cordless phone)

Cygnion is planning to beta test the latest revision of CyberGenie, and is
seeking interested beta testers of all experience levels to participate in
field testing next month.

CyberGenie is a professional cordless phone system (micro PBX).  It features
a USB connection to your PC to provide unified messaging, voice activated
commands, and supports up to 2 lines and 10 handsets.

For details on the product, please see www.cygnion.com

If you are interested in being a beta tester, please visit apply.cygnion.com
to complete a beta application.

Brent Mosbrook
Product Manager
Cygnion Corp.

2. All this Stacker stuff

3. multi-user test tools for Windows

4. Watchdogs

5. ? Network and Multi-User Testing in VisualTest

6. friend template class

7. multi-user and multi-level database access

8. VPN Connect through firewall failing??????

9. Looking for multi platform to test my script.

10. PPTP VPN +XBOX = Playing HALO multi-user over the internet

11. Multi-user card for PC

12. ROM monitor error --- Solved..Now boot in Multi-user mode.