disk performance using FSI on D3/NT is slower than VME

disk performance using FSI on D3/NT is slower than VME

Post by Craig Tai » Fri, 15 Sep 2000 15:28:48



When running large selects and working tight loops updating records in the
FSI I got a 30-40% decrease in performance as apposed to the exact same
process updating a VME account.

This is the opposite to what I would of expected.

Unfortunately I must use the FSI cause the VME max size is just under 2Gb.

Disk IO went up considerably more and this is the crux of the problem. The
drives are stripped and defragmented regularly so this isn't the issue. I
have multiple indexes on the large files. There are no overflow frames as a
regular resize is performed for maximum optimisation.

If anyone out there can enlighten me as to why I would appreciate it.

Thanks in advance

 
 
 

disk performance using FSI on D3/NT is slower than VME

Post by K. Powi » Fri, 15 Sep 2000 04:00:00




Quote:>When running large selects and working tight loops updating records in the
>FSI I got a 30-40% decrease in performance as apposed to the exact same
>process updating a VME account.

>This is the opposite to what I would of expected.

>Unfortunately I must use the FSI cause the VME max size is just under 2Gb.

>Disk IO went up considerably more and this is the crux of the problem. The
>drives are stripped and defragmented regularly so this isn't the issue. I
>have multiple indexes on the large files. There are no overflow frames as a
>regular resize is performed for maximum optimisation.

>If anyone out there can enlighten me as to why I would appreciate it.

>Thanks in advance

I don't have any quick answers for you, but I can tell you that the
stability of your system and the safety of your data is increased by
keeping your accounts in the FSI as apposed to in the VME.

If you VME ever goes "belly-up" and you have all your data accounts
there, you're looking at a full-blown restore.  Always a
nerve-wracking experience.

The only accounts in our VMEs are the realatively static base system
accounts DM, etc.  This means our VME's only need to be large enough
to accomodate user workspace and spooler entries -- usually no larger
than 250MB.   All user accounts are in the FSI.  When the VME needs to
be restored, we do so from a small pseudo tape save, originally
created with:

Select MDS with *a1 eq "D"
Save (FDT.

When restoring, It only takes a minute before the system is
up-and-running again.  The only big downside is the loss of spooler
entries, but this happens in any type of restore.

Good luck with your disk i/o issues.

Kevin Powick
Trident Information Systems
kkp(at)tridentinfosys(dot)com

 
 
 

disk performance using FSI on D3/NT is slower than VME

Post by dalelb.. » Fri, 15 Sep 2000 04:00:00


Craig,

Not being a D3/NT dude I'll give my two cents worth anyways...

Slower performance from files stored in the FSI is what I would
expect!  Remember, Pick was optomized to use the VME, and extra
processes need to be in the loop to extract the data from the FSI.

I could be wrong on this point, and if I am I'm sure I'll here about it.

Best regards,

Dale

P.S.  Thanks for your input on the de*.



Quote:> When running large selects and working tight loops updating records
in the
> FSI I got a 30-40% decrease in performance as apposed to the exact
same
> process updating a VME account.

> This is the opposite to what I would of expected.

> Unfortunately I must use the FSI cause the VME max size is just under
2Gb.

> Disk IO went up considerably more and this is the crux of the
problem. The
> drives are stripped and defragmented regularly so this isn't the
issue. I
> have multiple indexes on the large files. There are no overflow
frames as a
> regular resize is performed for maximum optimisation.

> If anyone out there can enlighten me as to why I would appreciate it.

> Thanks in advance

Sent via Deja.com http://www.veryComputer.com/
Before you buy.
 
 
 

disk performance using FSI on D3/NT is slower than VME

Post by BobJ » Fri, 15 Sep 2000 04:00:00


Another guess - but just that, a guess.  The minimum write block size in the
FSI is probably much larger than the average item size in some of the files.
This could lead to a lot more seek time and read and write time.
BobJ

> Craig,

> Not being a D3/NT dude I'll give my two cents worth anyways...

> Slower performance from files stored in the FSI is what I would
> expect!  Remember, Pick was optomized to use the VME, and extra
> processes need to be in the loop to extract the data from the FSI.

> I could be wrong on this point, and if I am I'm sure I'll here about it.

> Best regards,

> Dale

> P.S.  Thanks for your input on the de*.



> > When running large selects and working tight loops updating records
> in the
> > FSI I got a 30-40% decrease in performance as apposed to the exact
> same
> > process updating a VME account.

> > This is the opposite to what I would of expected.

> > Unfortunately I must use the FSI cause the VME max size is just under
> 2Gb.

> > Disk IO went up considerably more and this is the crux of the
> problem. The
> > drives are stripped and defragmented regularly so this isn't the
> issue. I
> > have multiple indexes on the large files. There are no overflow
> frames as a
> > regular resize is performed for maximum optimisation.

> > If anyone out there can enlighten me as to why I would appreciate it.

> > Thanks in advance

> Sent via Deja.com http://www.veryComputer.com/
> Before you buy.

 
 
 

disk performance using FSI on D3/NT is slower than VME

Post by es » Fri, 15 Sep 2000 04:00:00


Craig,

All our tests indicate that the FSI is faster than the VME file system,
so I think there is another explanation:

1. Make sure you use FlashBASIC (option O) to compile you application.
Use the (F) option as well to take advantage of the floating point chip.

2. FSI is more sensitive to file sizing than the VME. You mentioned you
keep your files properly sized. It is worth using the Windowsd NT
Performance Monitor to check this:
- Open the NT performance Monitor (Start>Programs>Admnin
Tools>Performance Monitor)
- Edit > Add to Chart
- Object: Select D3 File System
- Counter: Select Ovf Add/s, Ovf Reads/, Reads/s, Writes/s
- Add
- Done

(Please see the D3 Books online for what these are)

Rerun your test. You should see hardly any Ovf Reads. If you see a ratio
of ( Ovf Reads / (Reads+Writes) ) greater than 0, there is some
overflow.

Also, the Reads/s and Writes/s will give you an indication of item reads
and writes rates you can sustain on your system. We routinely see 3,000
to 5,000 item reads or writes per second on a good machine (Quad P2
200Mhz, 512 Meg, RAID 1, Ultrawide SCSI, mixture or sequential and
random accesses).

3. If items are large (greater than 3K), they will all be in the
Overflow space of the file, and performance will not be as good. The
solution is to create these files with a larger frame size. Only the D3
File Manager can create files with non default frame size.

Hope this helps.

PS. In an answer to this post, somebody mentioned that the accesses were
optimized for the VME. Actually, we tried to do the opposite... The FSI
internal structure was designed to avoid all the group shifting, string
scanning,... that used to take place. It takes more disk space, but it
is faster, simpler and more robust.

---


> When running large selects and working tight loops updating records in the
> FSI I got a 30-40% decrease in performance as apposed to the exact same
> process updating a VME account.

> This is the opposite to what I would of expected.

> Unfortunately I must use the FSI cause the VME max size is just under 2Gb.

> Disk IO went up considerably more and this is the crux of the problem. The
> drives are stripped and defragmented regularly so this isn't the issue. I
> have multiple indexes on the large files. There are no overflow frames as a
> regular resize is performed for maximum optimisation.

> If anyone out there can enlighten me as to why I would appreciate it.

> Thanks in advance

 
 
 

disk performance using FSI on D3/NT is slower than VME

Post by dalelb.. » Fri, 15 Sep 2000 04:00:00


Well,  It looks like Bob and I need to learn from the old Kung Fu
show....

Grasshopper, expect the unexpected!

Dale



> Craig,

> All our tests indicate that the FSI is faster than the VME file
system,
> so I think there is another explanation:

> 1. Make sure you use FlashBASIC (option O) to compile you application.
> Use the (F) option as well to take advantage of the floating point
chip.

> 2. FSI is more sensitive to file sizing than the VME. You mentioned
you
> keep your files properly sized. It is worth using the Windowsd NT
> Performance Monitor to check this:
> - Open the NT performance Monitor (Start>Programs>Admnin
> Tools>Performance Monitor)
> - Edit > Add to Chart
> - Object: Select D3 File System
> - Counter: Select Ovf Add/s, Ovf Reads/, Reads/s, Writes/s
> - Add
> - Done

> (Please see the D3 Books online for what these are)

> Rerun your test. You should see hardly any Ovf Reads. If you see a
ratio
> of ( Ovf Reads / (Reads+Writes) ) greater than 0, there is some
> overflow.

> Also, the Reads/s and Writes/s will give you an indication of item
reads
> and writes rates you can sustain on your system. We routinely see
3,000
> to 5,000 item reads or writes per second on a good machine (Quad P2
> 200Mhz, 512 Meg, RAID 1, Ultrawide SCSI, mixture or sequential and
> random accesses).

> 3. If items are large (greater than 3K), they will all be in the
> Overflow space of the file, and performance will not be as good. The
> solution is to create these files with a larger frame size. Only the
D3
> File Manager can create files with non default frame size.

> Hope this helps.

> PS. In an answer to this post, somebody mentioned that the accesses
were
> optimized for the VME. Actually, we tried to do the opposite... The
FSI
> internal structure was designed to avoid all the group shifting,
string
> scanning,... that used to take place. It takes more disk space, but it
> is faster, simpler and more robust.

> ---


> > When running large selects and working tight loops updating records
in the
> > FSI I got a 30-40% decrease in performance as apposed to the exact
same
> > process updating a VME account.

> > This is the opposite to what I would of expected.

> > Unfortunately I must use the FSI cause the VME max size is just
under 2Gb.

> > Disk IO went up considerably more and this is the crux of the
problem. The
> > drives are stripped and defragmented regularly so this isn't the
issue. I
> > have multiple indexes on the large files. There are no overflow
frames as a
> > regular resize is performed for maximum optimisation.

> > If anyone out there can enlighten me as to why I would appreciate
it.

> > Thanks in advance

Sent via Deja.com http://www.deja.com/
Before you buy.
 
 
 

disk performance using FSI on D3/NT is slower than VME

Post by Craig Tai » Sat, 16 Sep 2000 01:58:53


I agree the FSI has many advantages over the VME especially your point on
data restoration. I`ll keep trying some things to see if I can get better
performance through the FSI. I might see if its quicker running the
programmes from a VME account using qpointers to them from the FSI and
whatever else I can think of.

Craig Tait
Tai-Tech Ltd

Quote:> I don't have any quick answers for you, but I can tell you that the
> stability of your system and the safety of your data is increased by
> keeping your accounts in the FSI as apposed to in the VME.

> If you VME ever goes "belly-up" and you have all your data accounts
> there, you're looking at a full-blown restore.  Always a
> nerve-wracking experience.

> The only accounts in our VMEs are the realatively static base system
> accounts DM, etc.  This means our VME's only need to be large enough
> to accomodate user workspace and spooler entries -- usually no larger
> than 250MB.   All user accounts are in the FSI.  When the VME needs to
> be restored, we do so from a small pseudo tape save, originally
> created with:

> Select MDS with *a1 eq "D"
> Save (FDT.

> When restoring, It only takes a minute before the system is
> up-and-running again.  The only big downside is the loss of spooler
> entries, but this happens in any type of restore.

> Good luck with your disk i/o issues.

> Kevin Powick
> Trident Information Systems
> kkp(at)tridentinfosys(dot)com

 
 
 

disk performance using FSI on D3/NT is slower than VME

Post by Craig Tai » Sat, 16 Sep 2000 12:55:10


Thats interesting indeed maybe it is the average item size that effects the
speed. All

programmes are compile with (o4f I determined the decrease in performance by
viewing the

average read & write operations per second on the drive by using the
physical disk

counter object. There must be an issue with the pick performance counter
under windows 2000 cause I can't use them. They are in the performance list
but are greyed out.

I generated some stats that show the avg read & write operation in the VME
is faster in this case. Interesting results cause you will notice that the
FSI has a higher max read/write counter yet the average performance is less.

physically looking at the record counters I would notice in the FSI that the
records would write at a steady rate for about 1500 records then slow for
about 300-400 then pickup again and go through this cycle. The VME record
counters sustained a steady rate from A to Z.

Since I can`t see the PICK counters here are the disk read/write byte
counters.

FSI Stats
===============
Write Bytes per second
Max=2,823,239
Avg=681,534

Read Bytes per second
Max=1,243,559
Avg=147,000

CPU % (Dual pIII-500)
Max=90
Avg=66

peek memory=26%
peek page file=15%

Total Proccessing Time=41:05

VME stats
===============
Write Bytes per second
Max=2,188,932
Avg=806,534

Read Bytes per second * cant see read ops
Max=1,238,456
Avg=290,000

CPU % (Dual pIII-500)
Max=95%
Avg=78%

> Craig,

> All our tests indicate that the FSI is faster than the VME file system,
> so I think there is another explanation:

> 1. Make sure you use FlashBASIC (option O) to compile you application.
> Use the (F) option as well to take advantage of the floating point chip.

> 2. FSI is more sensitive to file sizing than the VME. You mentioned you
> keep your files properly sized. It is worth using the Windowsd NT
> Performance Monitor to check this:
> - Open the NT performance Monitor (Start>Programs>Admnin
> Tools>Performance Monitor)
> - Edit > Add to Chart
> - Object: Select D3 File System
> - Counter: Select Ovf Add/s, Ovf Reads/, Reads/s, Writes/s
> - Add
> - Done

> (Please see the D3 Books online for what these are)

> Rerun your test. You should see hardly any Ovf Reads. If you see a ratio
> of ( Ovf Reads / (Reads+Writes) ) greater than 0, there is some
> overflow.

> Also, the Reads/s and Writes/s will give you an indication of item reads
> and writes rates you can sustain on your system. We routinely see 3,000
> to 5,000 item reads or writes per second on a good machine (Quad P2
> 200Mhz, 512 Meg, RAID 1, Ultrawide SCSI, mixture or sequential and
> random accesses).

> 3. If items are large (greater than 3K), they will all be in the
> Overflow space of the file, and performance will not be as good. The
> solution is to create these files with a larger frame size. Only the D3
> File Manager can create files with non default frame size.

> Hope this helps.

> PS. In an answer to this post, somebody mentioned that the accesses were
> optimized for the VME. Actually, we tried to do the opposite... The FSI
> internal structure was designed to avoid all the group shifting, string
> scanning,... that used to take place. It takes more disk space, but it
> is faster, simpler and more robust.

> ---


> > When running large selects and working tight loops updating records in
the
> > FSI I got a 30-40% decrease in performance as apposed to the exact same
> > process updating a VME account.

> > This is the opposite to what I would of expected.

> > Unfortunately I must use the FSI cause the VME max size is just under
2Gb.

> > Disk IO went up considerably more and this is the crux of the problem.
The
> > drives are stripped and defragmented regularly so this isn't the issue.
I
> > have multiple indexes on the large files. There are no overflow frames
as a
> > regular resize is performed for maximum optimisation.

> > If anyone out there can enlighten me as to why I would appreciate it.

> > Thanks in advance

 
 
 

1. D3 FSI to VME Restore

We want to move some production accounts (small ones) back from the FSI to
the VME.

Will the file-save (a and deleting the accounts with the file manager do the
trick?

This is because NT-resize problems - we use the old f-resize and works like
a a charm on AIX/SCO D3

Thanks Nick

2. ODBC Manager in the Registry

3. D3 NT How to get D3 to see files in the fsi

4. Using AS400 Client Access as a Source

5. Restore d3/NT fsi account on D3/SCO

6. US-IL-Chicago Sybase DBA Position - Financial Trading Firm $$$

7. D3/NT VME space problems

8. VBScript modify record?

9. D3 ODBC and VME deaths on NT

10. D3/NT VME startup problem

11. D3/NT vme restores

12. Starting D3 VME service in NT from desktop

13. be carefull resizing vme d3/nt