Al raises some very interesting points, many of which I have struggled with.
I have been "benchmarking" RDBMS's for Solbourne Computer
for the past 8 months or so.
Quote:>> What I would like is to get a hold
>>of a benchmark which would let us test the performance
>>of our system.
Standard benchmarks exist, TPC-a,b,c - but...
1) They do not accurately model anything but transactions
2) They are very complicated to run
3) They are not public domain
These are the only standard benchmarks I know about. Please let me know
what else is out there.
Quote:>>We would like the benchmark to model some realistic
>>user interactions with a database. This is something
>>else I am looking for.
In order to create the benchmark, the Database Application must be known.
What is realistic- well what really happens? This is a flaw with the
TPC benchmarks - The results show max TPS's - but it is not a real application.
Quote:>>Where can I find these kind of statistics? That is
>>things like, how many queries compared to updates (i.e.
>>SQL inserts, deletes and update combined), how long are
>>transactions around for? How big is the data which is
>>queried? How big is the data which is updated?
Aye, theres the rub.
OK - So what did I do? I narrowed my reseach down to a particular application:
ORACLE FINANCIALS ( Oracle Corp's integrated fianacial package, all TM's and
other ownership identifiers respectfully ommitted)
I used a remote terminal emulation package ( Neil Nelson Inc. RTE ). The RTE
allowed keystroke capture ( important because the application was screen based),
and the ability to edit the resultant scipts. I created several scripts to
emulate several types of users. Further editing of three scripts allowed me
to add unique variables ( based on user number) so that multiple invocations
of the same script were possible.
The RTE supported (psuedo)random numbers and wait times, as well as variable
typing speeds.
Since I was testing an application, I did not have to deal with individual
ratios of selects, updates, inserts, commits etc. The application handled
all of that.
The development of an inhouse benchmark - defining the data, scheama, actions -
then testing on several databases and several machines would be very usefull as
an academic (no slash on schools - I went to a couple myself) exercise. But, testing
of the actual application is really where its at.
I can pass along more info.
----------------------
jonathan schnee
303-678-4528
by