BDE and large databases (>2Gb)

BDE and large databases (>2Gb)

Post by Jarvis N. Bra » Sun, 31 Dec 1899 09:00:00



Is there any known problem with using BDE to access or write databases
past 2Gb. I know there is a problem with individual tables of that
size but we have a system which creates databases which can be in
excess of that size by writing the data into multiple dated tables.

However we have discovered a problem! When the total of the data in
the data directory reaches almost, but not quite, 2Gb there appears to
be a chance that it will not be possible to write further data to the
database due to 'insufficient disk space' errors even though there is
more than sufficient disk space. There is no problem with having a
database larger than this as we can copy one of the older tables to
force the directory to contain >2Gb and then restart everything and
the database continues to grow normally, it's just getting past that
limit.

More curiously it does not appear to be dependent on the open tables
(our system rarely has more than one or two tables open at a time,
perhaps 200 or 300Mb at most) and does not appear to depend on the
directory in which the data resides (we have seen the problem when the
size of d:\data + d:\data\temp reached 2Gb).

Has anyone out there encountered anything similar? Can this be
explained? Can it be solved? At present our only solution is to wait
until we reach 2Gb and the system stalls at which point we copy a
table to force the data past 2Gb and restart. This is all implemented
using Delphi3, BDE 5.01 (or BDE 4.51) and Paradox 7.

Thanks in advance for any suggestions.

Jarvis N. Brand


 
 
 

BDE and large databases (>2Gb)

Post by Jesper Stougaard Knudse » Sun, 31 Dec 1899 09:00:00


Don't you think a 2Gb database based on Paradox table might become a bit
slow?

You must consider changing to at "real" database system e.g. like
Interbase which comes with Delphi C/S or Oracle og MS SQL Server 6.5 (7)

Gr.

JSK


> Is there any known problem with using BDE to access or write databases
> past 2Gb. I know there is a problem with individual tables of that
> size but we have a system which creates databases which can be in
> excess of that size by writing the data into multiple dated tables.

> However we have discovered a problem! When the total of the data in
> the data directory reaches almost, but not quite, 2Gb there appears to
> be a chance that it will not be possible to write further data to the
> database due to 'insufficient disk space' errors even though there is
> more than sufficient disk space. There is no problem with having a
> database larger than this as we can copy one of the older tables to
> force the directory to contain >2Gb and then restart everything and
> the database continues to grow normally, it's just getting past that
> limit.

> More curiously it does not appear to be dependent on the open tables
> (our system rarely has more than one or two tables open at a time,
> perhaps 200 or 300Mb at most) and does not appear to depend on the
> directory in which the data resides (we have seen the problem when the
> size of d:\data + d:\data\temp reached 2Gb).

> Has anyone out there encountered anything similar? Can this be
> explained? Can it be solved? At present our only solution is to wait
> until we reach 2Gb and the system stalls at which point we copy a
> table to force the data past 2Gb and restart. This is all implemented
> using Delphi3, BDE 5.01 (or BDE 4.51) and Paradox 7.

> Thanks in advance for any suggestions.

> Jarvis N. Brand




 
 
 

BDE and large databases (>2Gb)

Post by Jarvis N. Bra » Sun, 31 Dec 1899 09:00:00


On Fri, 15 Jan 1999 08:18:50 +0100, Jesper Stougaard Knudsen


>Don't you think a 2Gb database based on Paradox table might become a bit
>slow?

From what we have found Paradox is reasonably efficient and fast
handling databses of this size as long as the design is such that it
reduces as far as possible the search space. From our tests we have
found that an indexed search on databases of several hundred Mb can be
performed in sub 5 seconds. Text searches across a asingle table
(again perhaps 2-300 Mb) run in typically under 1 minute. Not
astounding, but for the application we have and functionality we
provide the customer regards it as "the dog's bollocks" (which, for
anyone who hasn't heard the expression, is, implausible as it sounds,
a compliment). Just to give some indication of its performance the
figues are not affected by the total database size and we have this
running with systms of up to 10Gb of active data at present and, with
our current installed systems we anticipate being able to handle up to
1/2 Tb of data (any bottleneck will arise from the physical access to
the optical juke box on which we hold historic data)

The thing to avoid with paradox and BDE is any single table reaching
an excessive size. We record data in separate tables based on the
received date of the data which keeps individual tables below 300Mb.
Our problem arise from the fact that BDE / Paradox appears to be aware
of the total size of the data even though the tables making up that
2Gb of data are not open at the time tha the problem arises.

Quote:>You must consider changing to at "real" database system e.g. like
>Interbase which comes with Delphi C/S or Oracle og MS SQL Server 6.5 (7)

I argued for this when we implemented the system but ultimately as we
archive all data data on to an optical juke box the paradox table
format simplified that (it's easier to split a paraodx table than to
split a database on an SQL server).

As the system is implemented at present our only identified problem is
the total size of all tables in the accessed drive but we can see no
rational reason for such behaviour.

Jarvis N. Brand