2 gigs from fm3 to fm 5.5 to web

2 gigs from fm3 to fm 5.5 to web

Post by ben freedm » Fri, 03 Aug 2001 19:00:56



We are considering taking on a project involving a database
constructed in FileMaker 3, and moving it onto the Web.

The database is very large: 2 Gb of material (text and images) in
about 5,000 records.  The client was motivated to do something about
it because there has been some kind of degradation lately.  Older
records are sometimes corrupt, and new records will not record
correctly.

I'm not very familiar with FileMaker, but I gather it integrates well
with the Web.  I've seen a few sites (mentioned here and at the fm
site) that work very nicely.  None of these seem to involve quite this
volume of material though.  We're wondering whether to stay with fm or
go some other route.

Is this client stretching fm3 beyond its limits, or does it sound more
like a hardware problem? Can fm5.5 handle this quantity of material?
(I see nothing about such limits on the website or anywhere in the
trial version of fmpro I've downloaded.) Are there likely to be
problems skipping a version (moving from ver. 3 to 5.5)? This would be
a pretty low traffic site, but possibly more than ten users at a time.
 They will not be manipulating the data, though.  The idea is for
people to have access to the information on the Web (they currently
have to go to the client's office!), but not to enter any or alter it.
 Will we definitely need the Unlimited edition?

One attraction of fm is that it seems relatively easy to learn (we do
have expertise in SQL, ASP and related).  But I'm not sure it's
intended for this kind of use.

Any enlightenment most appreciated.

Ben

 
 
 

2 gigs from fm3 to fm 5.5 to web

Post by John Weinshe » Fri, 03 Aug 2001 22:20:03


2 Gig per file is FMP's limit, and the problems you describe may well be due
to that limit being approached. The application needs some room to maneuver
within the limit, and it is thus not a good idea to get too close. I have
heard (it is a rare event I have never seen) that it is an ugly thing to
behold when the limit is breached.

It is almost certain that the large file size, particularly with so few
records, is a result of the images being stored directly within the files.
The more usual method is to store only the path to the images and store the
images themselves elsewhere. That "elsewhere" may be software especially
designed for image cataloguing, such as Cumulus. Again, to move images to
the web expeditiously, it's quite likely their size will need to be reduced,
independently of how they are stored-- either by compression, conversion to
another protocol, and/or size reduction.

I believe there are some good plugins (try Troi, but others here may know
better) for grabbing the file path, so that you do not need to go back and
manually re-target the images.5.5 has improved that capability, but a plugin
seems more promising.

It's possible some file size reduction might also be achieved by breaking
the file into two or more parts, but my guess is that the text, unless you
are storing encyclopedias, is not taking up much room.

You should not encounter any problems skipping revs, and 5K records is no
problem, and a good argument for trying to solve the size problem and
staying within Filemaker, as it seems unlikely an enterprise app is needed
here, and FMP is so perfect in every other way for what you describe. You
will need Unlimited to serve out the pages.

Whatever you do, do it quickly, before the 2 gig limit gets any closer. As
an interim measure, you can export half the records to a clone, but that is
not a good file structure at all.

--
John Weinshel
Datagrace
Vashon Island, WA
(206) 463-1634
Associate Member, Filemaker Solutions Alliance


Quote:> We are considering taking on a project involving a database
> constructed in FileMaker 3, and moving it onto the Web.

> The database is very large: 2 Gb of material (text and images) in
> about 5,000 records.  The client was motivated to do something about
> it because there has been some kind of degradation lately.  Older
> records are sometimes corrupt, and new records will not record
> correctly.

> I'm not very familiar with FileMaker, but I gather it integrates well
> with the Web.  I've seen a few sites (mentioned here and at the fm
> site) that work very nicely.  None of these seem to involve quite this
> volume of material though.  We're wondering whether to stay with fm or
> go some other route.

> Is this client stretching fm3 beyond its limits, or does it sound more
> like a hardware problem? Can fm5.5 handle this quantity of material?
> (I see nothing about such limits on the website or anywhere in the
> trial version of fmpro I've downloaded.) Are there likely to be
> problems skipping a version (moving from ver. 3 to 5.5)? This would be
> a pretty low traffic site, but possibly more than ten users at a time.
>  They will not be manipulating the data, though.  The idea is for
> people to have access to the information on the Web (they currently
> have to go to the client's office!), but not to enter any or alter it.
>  Will we definitely need the Unlimited edition?

> One attraction of fm is that it seems relatively easy to learn (we do
> have expertise in SQL, ASP and related).  But I'm not sure it's
> intended for this kind of use.

> Any enlightenment most appreciated.

> Ben


 
 
 

2 gigs from fm3 to fm 5.5 to web

Post by Tim Boot » Sat, 04 Aug 2001 08:03:45



> We are considering taking on a project involving a database
> constructed in FileMaker 3, and moving it onto the Web.

> The database is very large: 2 Gb of material (text and images) in
> about 5,000 records.  The client was motivated to do something about
> it because there has been some kind of degradation lately.  Older
> records are sometimes corrupt, and new records will not record
> correctly.

Holy cow!! 2Gb is the absolute maximum, go no further, top end
limit of FileMaker - but I've never seen anyone get close to it.
Anecdotally, I don't think even the International Movie Database
is that size... And this is probably causing the 'degradation'.

Quote:

> I'm not very familiar with FileMaker, but I gather it integrates well
> with the Web.  I've seen a few sites (mentioned here and at the fm
> site) that work very nicely.  None of these seem to involve quite this
> volume of material though.  We're wondering whether to stay with fm or
> go some other route.

Well, first thing is I would get all of the stored images out
of the database and just have references to them. This will probably
drop the database back to under 100 Mb (I can't think of any
reason aprt from storing images in container fields that
would cause a database to be so big...)

Quote:

> Is this client stretching fm3 beyond its limits, or does it sound more
> like a hardware problem? Can fm5.5 handle this quantity of material?
> (I see nothing about such limits on the website or anywhere in the
> trial version of fmpro I've downloaded.) Are there likely to be
> problems skipping a version (moving from ver. 3 to 5.5)? This would be
> a pretty low traffic site, but possibly more than ten users at a time.
>  They will not be manipulating the data, though.  The idea is for
> people to have access to the information on the Web (they currently
> have to go to the client's office!), but not to enter any or alter it.
>  Will we definitely need the Unlimited edition?

Yes. The 10 user limit is a rolling 12 hour limitation. On the
sites I run, it would take about 2 minutes to exceed that :-)

Quote:

> One attraction of fm is that it seems relatively easy to learn (we do
> have expertise in SQL, ASP and related).  But I'm not sure it's
> intended for this kind of use.

Well, I'm running 23 systems across 3 machines taking 50,000 hits
a day with very few problems. But I do not keep images in the
databases (or even on the same machine actually)...

Cheers

Webko

 
 
 

2 gigs from fm3 to fm 5.5 to web

Post by Ruben van den Boogaar » Sat, 04 Aug 2001 16:52:47


Hi Ben,

As pointed out by John and Tim, it is much better to get the pictures out of the database and only reference to them, but the problem is how?
extracting 5000 pictures manually is a hard job.

I would suggest to use plug-ins to do the conversion,

Use the 'Exporter 1.1.1' plugin www.oo7.net to export the pictures,
the plug-in also allows to convert the pictures if you like, and create
thumbnails, if you like which you can import into the database again. (those are much smaller and otherwise you would have 2 picture files per record.)

Maybe use this plug-in in combination with the Troi File plug-in
www.troi.com to import the references back into the database, make also a separate textfield where you can store the filepath, if you have to change the position of the pictures to the database (other harddisk?)
you can use this field to update the references.

Good Luck!

Ruben



> > We are considering taking on a project involving a database
> > constructed in FileMaker 3, and moving it onto the Web.

> > The database is very large: 2 Gb of material (text and images) in
> > about 5,000 records.  The client was motivated to do something about
> > it because there has been some kind of degradation lately.  Older
> > records are sometimes corrupt, and new records will not record
> > correctly.

> Holy cow!! 2Gb is the absolute maximum, go no further, top end
> limit of FileMaker - but I've never seen anyone get close to it.
> Anecdotally, I don't think even the International Movie Database
> is that size... And this is probably causing the 'degradation'.

> > I'm not very familiar with FileMaker, but I gather it integrates well
> > with the Web.  I've seen a few sites (mentioned here and at the fm
> > site) that work very nicely.  None of these seem to involve quite this
> > volume of material though.  We're wondering whether to stay with fm or
> > go some other route.

> Well, first thing is I would get all of the stored images out
> of the database and just have references to them. This will probably
> drop the database back to under 100 Mb (I can't think of any
> reason aprt from storing images in container fields that
> would cause a database to be so big...)

> > Is this client stretching fm3 beyond its limits, or does it sound more
> > like a hardware problem? Can fm5.5 handle this quantity of material?
> > (I see nothing about such limits on the website or anywhere in the
> > trial version of fmpro I've downloaded.) Are there likely to be
> > problems skipping a version (moving from ver. 3 to 5.5)? This would be
> > a pretty low traffic site, but possibly more than ten users at a time.
> >  They will not be manipulating the data, though.  The idea is for
> > people to have access to the information on the Web (they currently
> > have to go to the client's office!), but not to enter any or alter it.
> >  Will we definitely need the Unlimited edition?

> Yes. The 10 user limit is a rolling 12 hour limitation. On the
> sites I run, it would take about 2 minutes to exceed that :-)

> > One attraction of fm is that it seems relatively easy to learn (we do
> > have expertise in SQL, ASP and related).  But I'm not sure it's
> > intended for this kind of use.

> Well, I'm running 23 systems across 3 machines taking 50,000 hits
> a day with very few problems. But I do not keep images in the
> databases (or even on the same machine actually)...

> Cheers

> Webko

 
 
 

2 gigs from fm3 to fm 5.5 to web

Post by ben freedm » Sun, 05 Aug 2001 07:14:54



Quote:> Hi Ben,

> As pointed out by John and Tim, it is much better to get the pictures out of the database and only reference to them, but the problem is how?
> extracting 5000 pictures manually is a hard job.

Yes, indeed.   Those images are definitely the issue.  

Quote:> I would suggest to use plug-ins to do the conversion,

> Use the 'Exporter 1.1.1' plugin www.oo7.net to export the pictures,
> the plug-in also allows to convert the pictures if you like, and create
> thumbnails, if you like which you can import into the database again. (those are much smaller and otherwise you would have 2 picture files per record.)

This sounds like the ticket.  I've just been to their site and it
looks like this is just what we need.

Of course, what hadn't occured to me initially (I'm not the one who
has been in contact with the client so far) was that these images will
not, generally, have been compressed in any way.  When the system was
set up three years ago, they weren't thinking about the Web (nor that
the thing would get so big) -- it's all just sitting on the one
computer, to be accessed locally.  So, yes, the images are going to
have to be extracted from where they are, and then
compressed/converted so that people with dial-up connections don't
have to wait half an hour for each picture.

The client is now mulling over the implications of all this.  We
should know next week what they want to do.

Quote:

> Maybe use this plug-in in combination with the Troi File plug-in
> www.troi.com to import the references back into the database, make also a separate textfield where you can store the filepath, if you have to change the position of the pictures to the database (other harddisk?)
> you can use this field to update the references.

> Good Luck!

> Ruben

Thanks very much.  And thank you all for some excellent information!
I feel much more like I have my bearings now.

Ben

 
 
 

2 gigs from fm3 to fm 5.5 to web

Post by Jeff R. Harmo » Tue, 28 Aug 2001 12:32:39




> Holy cow!! 2Gb is the absolute maximum, go no further, top end
> limit of FileMaker - but I've never seen anyone get close to it.

It's a common numeric boundary (and is also the maximum file size on a
Mac HFS partition).  If you're storing very large graphics within a
FileMaker database you should use on of the many file editor plugins
(e.g., Troi File) to link to them instead.  Not only will your database
shrink to a trivial size it's performance will improve by many, many
orders of magnitude.

JR

 
 
 

2 gigs from fm3 to fm 5.5 to web

Post by Tim Boot » Tue, 28 Aug 2001 16:15:40





> > Holy cow!! 2Gb is the absolute maximum, go no further, top end
> > limit of FileMaker - but I've never seen anyone get close to it.

> It's a common numeric boundary (and is also the maximum file size on a
> Mac HFS partition).  If you're storing very large graphics within a
> FileMaker database you should use on of the many file editor plugins
> (e.g., Troi File) to link to them instead.  Not only will your database
> shrink to a trivial size it's performance will improve by many, many
> orders of magnitude.

Agreed - see my original post on the topic...

Webko

 
 
 

1. Workaround for FM 5.5, 6 crashes with web companion

I reported the problems, as have many others, that FM 5.5 and 6 crashes
very frequently under MacOS X when using the web companion, for instance
with Lasso or other web serving solutions. I have looked for a solution
here and other places, but haven't found any. So I thought maybe others
would find the following tip on how to make it as rock solid as under
MacOS 9 useful. The workaround is just that - not a solution. But it
might work for you.
So, the idea is simply this: Run FM, with the databases you need, under
Classic. You can open the info panel on your database files, and mark
them as "Open in Classic". If you have FM 5.5 for MacOS 9 installed, the
files will open in FM 5.5 classic (starting Classic first, if needed).
Lasso and your webserver still runs native in MacOS X, of course, and
Lasso at least have no problems communication with the web companion in
FM even if FM runs under Classic. If you cleanse your Classic System
Folder of any unneccesary extensions etc, and make sure FM is the only
App running under Classic, the whole system can be very stable and
almost as responsive as a fully native setup. My setup is now rock
stable, with one execption: If I Quit the classic FM, sometimes it
refuses to start again, and it seems Classic has frozen, forcing me to
force-quit Classic. But this happens infrequently, and is not a big
problem since FM as good as never crashes under Classic.
Hopes this can help some of you frustrated souls, as it did for me.

Lars

2. Crosstab Queries in SQL Server? Newbie needs help!

3. FM 5.5 crashes when web companion checked

4. please anybody help me?

5. Can FM Server 5.5 do Web Publishing?

6. Quick question

7. FM 5.5 and Web

8. FM 5.5 and Instant Web Publishing

9. FM 5.5 - Web Companion Issue :- (

10. FM 5.5 - Web Companion problem

11. Problem with FM 5.5 Instant Web Publishing

12. FM Pro 5.5 Username on Web