multithread processing resultset

multithread processing resultset

Post by dragon bal » Sun, 23 Jun 2002 13:49:43



Hi all,

    I am using jdbc to access database but suffer from performance problem.
Well, I think I can't boost the performance by modifying the code since the
resultset is quite large and maybe it is the JdbcDriver problem.

    So I am now thinking of using multithread to traverse the resultset and
do the processing. Say thread 1 do processing on 1 to 100 records. Thread 2
go to 101 to 200 (maybe by going thru a cloned resultset). Can this boost
the performance ? Or is there other ways that I may use multithread? Since
my server has 4 cpu. So it is quite advantageous if I could use multithread.
Is there anything I need to be aware of ? Is there any article or example
source on that ?

    It seems that I seldom heard ppl using this. So I am not sure if this
work .

Thanks .

Perseus

 
 
 

multithread processing resultset

Post by Alex » Sun, 23 Jun 2002 14:11:28


My suggestion would be test relative time of 3 steps:

-1- database execution (from sending request till getting first row)
-2-  network speed (getting next row of set of rows)
-3- speed of java side row consuption
(from receiving row to next rs.next())

Multi-threading way can be faster if 2 > 1+3
and there is no dependance of row consumption from result of
consuption of previous rows.

In general, I see increase of code complexity that can be justify only
by serious speed gain.

AlexV.


Quote:> Hi all,

>     I am using jdbc to access database but suffer from performance
problem.
> Well, I think I can't boost the performance by modifying the code since
the
> resultset is quite large and maybe it is the JdbcDriver problem.

>     So I am now thinking of using multithread to traverse the resultset
and
> do the processing. Say thread 1 do processing on 1 to 100 records. Thread
2
> go to 101 to 200 (maybe by going thru a cloned resultset). Can this boost
> the performance ? Or is there other ways that I may use multithread? Since
> my server has 4 cpu. So it is quite advantageous if I could use
multithread.
> Is there anything I need to be aware of ? Is there any article or example
> source on that ?

>     It seems that I seldom heard ppl using this. So I am not sure if this
> work .

> Thanks .

> Perseus


 
 
 

multithread processing resultset

Post by dragon bal » Sun, 23 Jun 2002 15:06:06


Quote:> -1- database execution (from sending request till getting first row)
> -2-  network speed (getting next row of set of rows)

Do you mean that the jdbc driver will go to the database to retrieve the row
I need whenever I call rslt.next() ?
if yes, my db is a remote one and I have thousands of records, is there any
common way to resolve this performance issue ?
 
 
 

multithread processing resultset

Post by Alex » Sun, 23 Jun 2002 21:49:39



Quote:> > -1- database execution (from sending request till getting first row)
> > -2-  network speed (getting next row of set of rows)

> Do you mean that the jdbc driver will go to the database to retrieve the
row
> I need whenever I call rslt.next() ?
> if yes, my db is a remote one and I have thousands of records, is there
any
> common way to resolve this performance issue ?

In most cases driver transfer rows by tens. There is specific
parameter (setFetchSize()). For example, upon first rs.next()
Oracle driver will get first 10 or 20 records. Unless it is "very"
remote server, and unless columns are huge,
network transfer time is not [relatively] big.

My main point is that it must be test in your real
environment.

AlexV

 
 
 

1. VB+DSO to process cubes multithread okay?

I'm having trouble processing 300+ cubes within our daily batch window.

I'm using VB+DSO to process these 300+ cubes and it takes almost 1 minute to
process each cube.  So it takes almost 5 hours each night which does not fit
into our tight batch window.

Since my server is a 4-CPU one, I'm considering to re-write the VB program
to use multiple threads calling DSO to process the cubes.  Is DSO
thread-safe?  Has anyone tried this before?  Could anyone share some code
fragments to help solve this problem?

Thanks in advance.

Alex Wong

2. INF4.1 -1 ERROR ?? problem

3. ResultSets and Tuple processing???

4. OLE Automomation error -2147483640

5. ResultSet processing slowly

6. How do I connect?

7. ResultSet processing

8. LEFT JOIN - ODBC Call failed - Error 3146 - (#107)

9. Processing Multiple Resultsets using RDO 2.0

10. Can't process multiple result sets when first resultset is empty

11. How do we know if a process is user process or system process

12. user process,server process,application process

13. ResultSet with few rows invisible to ResultSet.next()