Progress, Proquiet and Snapshots

Progress, Proquiet and Snapshots

Post by Toni » Thu, 03 Jul 2003 21:56:30



Hi people,

I need some help to design a Disaster Recovery solution.

The idea is to make a replication server and each one hour, for
example, update content from one to other.

I've thought on Progress, Proquiet and Snapshots on Solaris (with
fssnap). Can it work ? What is your experience on them ?

I've suggest Fathom HA but for money reasons, it cannot be possible,
at this moment.

The other idea is to use rsync command to send files in secure mode,
at this case, Progress files, database, etc. Do you have implement it
?

I wait your suggestions.

Thanks,

Toni

PS: I'm a non-experiened on Progress database

 
 
 

Progress, Proquiet and Snapshots

Post by Kevi » Fri, 04 Jul 2003 00:57:26




Quote:> Hi people,

> I need some help to design a Disaster Recovery solution.

> The idea is to make a replication server and each one hour, for
> example, update content from one to other.

> I've thought on Progress, Proquiet and Snapshots on Solaris (with
> fssnap). Can it work ? What is your experience on them ?

> I've suggest Fathom HA but for money reasons, it cannot be possible,
> at this moment.

> The other idea is to use rsync command to send files in secure mode,
> at this case, Progress files, database, etc. Do you have implement it
> ?

> I wait your suggestions.

> Thanks,

> Toni

> PS: I'm a non-experiened on Progress database

The easiest method is log based replication.. Backup your DB, enable AI.
Restore the backup on the replication server. Write a script on the
production machine that checks every x minutes for the oldest full file..
If no full file exists, force AI to switch and grab the full file and
copy it to the replication server. Make sure the file arrived okay, and
mark it as empty on the production machine. Have another job on the
replication server that checks for the existance of the file, and applies
it to the restored db..

Keep in mind that a 50% license fee is required from Progress for the
replication machine.

I guess Proquiet and Snapshots on Soalris would work fine, but there will
always be a level of interruption, which you may not want.

Kevin

 
 
 

Progress, Proquiet and Snapshots

Post by Rob Lyon » Fri, 04 Jul 2003 04:50:09


We currently use this method with a 9.1D system but it has problems.
1. Due to a bug in progress, you can never truncate the bi files on the DR
machine (there is a knowledge base article on this but I forget the number),
if you do it may cause the AI files to stop applying (but it may not!).
2. Progress are still investigating another possible bug where the DR system
is not re using BI space (this is how we found out about #1).
3. When generating DB backups to initially set up or rebuild the databases
on the DR machine, remeber to use the -norecover option, otherwise the DR
system may not apply the AI files (but again, it may!).

When we implemented this (about18 months ago) Progress reccomended this
method, but due to the amount of problems reported they no longer do.

Another tip, make sure your method of transfering the AI files is solid. We
have a problem whereby the Live server will re-send an AI file if it hasn't
been processed from the last transfer (incase it never got there), if the AI
file is in the process of being applied on the DR machine,  Progress really
gets confused and starts reporting Disk Full errors. A rebuild of the DR DB
is then required.

I wouldn't reccomend it for mission critical systems if you are
inexperienced. On the other hand it is an excelent way to learn (the hard
way!) how AI works.

Oh and another thing, make sure you start AI on your live server BEFORE you
back up the DB's. Starting AI updates the DB internal date and time stamps
so they are then out of sync with the backups. (Another one we learnt the
hard way)!

Cheers,
Rob.




> > Hi people,

> > I need some help to design a Disaster Recovery solution.

> > The idea is to make a replication server and each one hour, for
> > example, update content from one to other.

> > I've thought on Progress, Proquiet and Snapshots on Solaris (with
> > fssnap). Can it work ? What is your experience on them ?

> > I've suggest Fathom HA but for money reasons, it cannot be possible,
> > at this moment.

> > The other idea is to use rsync command to send files in secure mode,
> > at this case, Progress files, database, etc. Do you have implement it
> > ?

> > I wait your suggestions.

> > Thanks,

> > Toni

> > PS: I'm a non-experiened on Progress database

> The easiest method is log based replication.. Backup your DB, enable AI.
> Restore the backup on the replication server. Write a script on the
> production machine that checks every x minutes for the oldest full file..
> If no full file exists, force AI to switch and grab the full file and
> copy it to the replication server. Make sure the file arrived okay, and
> mark it as empty on the production machine. Have another job on the
> replication server that checks for the existance of the file, and applies
> it to the restored db..

> Keep in mind that a 50% license fee is required from Progress for the
> replication machine.

> I guess Proquiet and Snapshots on Soalris would work fine, but there will
> always be a level of interruption, which you may not want.

> Kevin

 
 
 

Progress, Proquiet and Snapshots

Post by Kevi » Fri, 04 Jul 2003 05:28:53




Quote:> We currently use this method with a 9.1D system but it has problems.
> 1. Due to a bug in progress, you can never truncate the bi files on
> the DR machine (there is a knowledge base article on this but I forget
> the number), if you do it may cause the AI files to stop applying (but
> it may not!). 2. Progress are still investigating another possible bug
> where the DR system is not re using BI space (this is how we found out
> about #1). 3. When generating DB backups to initially set up or
> rebuild the databases on the DR machine, remeber to use the -norecover
> option, otherwise the DR system may not apply the AI files (but again,
> it may!).

> When we implemented this (about18 months ago) Progress reccomended
> this method, but due to the amount of problems reported they no longer
> do.

> Another tip, make sure your method of transfering the AI files is
> solid. We have a problem whereby the Live server will re-send an AI
> file if it hasn't been processed from the last transfer (incase it
> never got there), if the AI file is in the process of being applied on
> the DR machine,  Progress really gets confused and starts reporting
> Disk Full errors. A rebuild of the DR DB is then required.

> I wouldn't reccomend it for mission critical systems if you are
> inexperienced. On the other hand it is an excelent way to learn (the
> hard way!) how AI works.

> Oh and another thing, make sure you start AI on your live server
> BEFORE you back up the DB's. Starting AI updates the DB internal date
> and time stamps so they are then out of sync with the backups.
> (Another one we learnt the hard way)!

> Cheers,
> Rob.

I'm not sure about the bugs you mention, but I will check the bug
database tomorrow morning.. Do you have a bug number??

Yes, truncating the BI can be an issue, but as Knowledge Center article
P26615 points out, there are reasons for the BI growth. P3622 also has
some points to note.

Kevin

 
 
 

Progress, Proquiet and Snapshots

Post by Kevi » Fri, 04 Jul 2003 20:22:56




Quote:> We currently use this method with a 9.1D system but it has problems.
> 1. Due to a bug in progress, you can never truncate the bi files on
> the DR machine (there is a knowledge base article on this but I forget
> the number), if you do it may cause the AI files to stop applying (but
> it may not!). 2. Progress are still investigating another possible bug
> where the DR system is not re using BI space (this is how we found out
> about #1). 3. When generating DB backups to initially set up or
> rebuild the databases on the DR machine, remeber to use the -norecover
> option, otherwise the DR system may not apply the AI files (but again,
> it may!).

> When we implemented this (about18 months ago) Progress reccomended
> this method, but due to the amount of problems reported they no longer
> do.

> Another tip, make sure your method of transfering the AI files is
> solid. We have a problem whereby the Live server will re-send an AI
> file if it hasn't been processed from the last transfer (incase it
> never got there), if the AI file is in the process of being applied on
> the DR machine,  Progress really gets confused and starts reporting
> Disk Full errors. A rebuild of the DR DB is then required.

> I wouldn't reccomend it for mission critical systems if you are
> inexperienced. On the other hand it is an excelent way to learn (the
> hard way!) how AI works.

> Oh and another thing, make sure you start AI on your live server
> BEFORE you back up the DB's. Starting AI updates the DB internal date
> and time stamps so they are then out of sync with the backups.
> (Another one we learnt the hard way)!

> Cheers,
> Rob.

There is a bug #20030327-035 which will be fixed in 9.1D07 to do with
inconsistent in-flight information in AI files and truncating the bi
file, but it would appear that it is very sporadic..

Bug #20030122-011 was logged for 9.1D05 and was fixed in 9.1D06.
Apparantley this only manifests itself if the target BI cluster size is
larger than the source cluster size.

Hope this is of some use.

Kevin

 
 
 

Progress, Proquiet and Snapshots

Post by Kevi » Fri, 04 Jul 2003 22:24:37





>> We currently use this method with a 9.1D system but it has problems.
>> 1. Due to a bug in progress, you can never truncate the bi files on
>> the DR machine (there is a knowledge base article on this but I forget
>> the number), if you do it may cause the AI files to stop applying (but
>> it may not!). 2. Progress are still investigating another possible bug
>> where the DR system is not re using BI space (this is how we found out
>> about #1). 3. When generating DB backups to initially set up or
>> rebuild the databases on the DR machine, remeber to use the -norecover
>> option, otherwise the DR system may not apply the AI files (but again,
>> it may!).

>> When we implemented this (about18 months ago) Progress reccomended
>> this method, but due to the amount of problems reported they no longer
>> do.

>> Another tip, make sure your method of transfering the AI files is
>> solid. We have a problem whereby the Live server will re-send an AI
>> file if it hasn't been processed from the last transfer (incase it
>> never got there), if the AI file is in the process of being applied on
>> the DR machine,  Progress really gets confused and starts reporting
>> Disk Full errors. A rebuild of the DR DB is then required.

>> I wouldn't reccomend it for mission critical systems if you are
>> inexperienced. On the other hand it is an excelent way to learn (the
>> hard way!) how AI works.

>> Oh and another thing, make sure you start AI on your live server
>> BEFORE you back up the DB's. Starting AI updates the DB internal date
>> and time stamps so they are then out of sync with the backups.
>> (Another one we learnt the hard way)!

>> Cheers,
>> Rob.

> There is a bug #20030327-035 which will be fixed in 9.1D07 to do with
> inconsistent in-flight information in AI files and truncating the bi
> file, but it would appear that it is very sporadic..

> Bug #20030122-011 was logged for 9.1D05 and was fixed in 9.1D06.
> Apparantley this only manifests itself if the target BI cluster size is
> larger than the source cluster size.

Just to be a bit more clear, this was to do with Roll Forward not re-
using BI space..

- Show quoted text -

Quote:

> Hope this is of some use.

> Kevin

 
 
 

Progress, Proquiet and Snapshots

Post by pro » Fri, 04 Jul 2003 22:33:23




Quote:> We currently use this method with a 9.1D system but it has problems.
> 1. Due to a bug in progress, you can never truncate the bi files on
> the DR machine (there is a knowledge base article on this but I forget
> the number), if you do it may cause the AI files to stop applying (but
> it may not!).

This is not a bug, it's part of the way Progress databases work.
If crash recovery has not been run, it will when you truncate
the .bi.  That changes the state of the database and that's
why it prevents applying .ai files - because the database changed
and no longer is the same as the original database.

The reason for it not doing that all the time is when you
already truncated the .bi and it had no notes to apply for
crash recovery.  Then it might as well be the same thing.

Quote:> 2. Progress are still investigating another possible bug
> where the DR system is not re using BI space (this is how we found out
> about #1).

Also not a bug, but a very unfortunate part of the circular
ring method used for the .bi.  That comes from long standing
transactions inside the database.  That prevents one cluster
from being closed and as soon as that cluster becomes the
first in the ring, all other empty clusters cannot be reused.
And so the .bi grows with new clusters.

Quote:> 3. When generating DB backups to initially set up or
> rebuild the databases on the DR machine, remeber to use the -norecover
> option, otherwise the DR system may not apply the AI files (but again,
> it may!).

Exactly - that's the solution for not forcing the crash recovery
that would modify the database and invalidate it against new .ai files.
Stay clear of that, and you're fine.

Quote:> When we implemented this (about18 months ago) Progress reccomended
> this method, but due to the amount of problems reported they no longer
> do.

Actually they do recommend it.
 
 
 

Progress, Proquiet and Snapshots

Post by Rob Lyon » Sat, 05 Jul 2003 03:52:57


Thanks for all your points, it seems yet again that the Progress community
know more than Progress themselves!

I referred to them as bugs as that it what Progress called them!

As for the 06 service pack, we installed this recently and are monitoring.
However we do not hold much hope as Progress also informed us that the BI
file growth "bug" was fixed in 03, 04 and 05, despite no sign of it in the
release notes (none in 06 either).
The reason we truncated the BI files on our DR system is Progress told us
to! 2 of the 4 databases failed to apply AI files. We were then passed to
another support person (Bodo Manke, cool name) who said that the bug was
that you could truncate the BI files on a standby database and rfutil
shouldn't let you!

I wouldn't mind some more info on the circular ring method mentioned. We
have read all that Progress can offer on how the AI roll forward works with
standby BI files, but nothing seems to explain what happens.
E.G. We have a standby DB with 2 BI extents. One stays at 40MB but the other
one seems to randomly grow for no reason,  it is currently at 680MB and
seems to jump around 80 - 120 MB at a time when it applies AI files of only
20 - 50 MB. Every few months it blows the 2GB limit and we have to rebuild
the DB. This is why Progress recommended truncating the BI. Which didn't
work.

It was the afore mentioned Bodo that told us that Progress were no longer
recommending this method because of the amount of problems reported (however
he was trying to sell us Fathom at the same time).

Regards,
Rob.




> > We currently use this method with a 9.1D system but it has problems.
> > 1. Due to a bug in progress, you can never truncate the bi files on
> > the DR machine (there is a knowledge base article on this but I forget
> > the number), if you do it may cause the AI files to stop applying (but
> > it may not!).

> This is not a bug, it's part of the way Progress databases work.
> If crash recovery has not been run, it will when you truncate
> the .bi.  That changes the state of the database and that's
> why it prevents applying .ai files - because the database changed
> and no longer is the same as the original database.

> The reason for it not doing that all the time is when you
> already truncated the .bi and it had no notes to apply for
> crash recovery.  Then it might as well be the same thing.

> > 2. Progress are still investigating another possible bug
> > where the DR system is not re using BI space (this is how we found out
> > about #1).

> Also not a bug, but a very unfortunate part of the circular
> ring method used for the .bi.  That comes from long standing
> transactions inside the database.  That prevents one cluster
> from being closed and as soon as that cluster becomes the
> first in the ring, all other empty clusters cannot be reused.
> And so the .bi grows with new clusters.

> > 3. When generating DB backups to initially set up or
> > rebuild the databases on the DR machine, remeber to use the -norecover
> > option, otherwise the DR system may not apply the AI files (but again,
> > it may!).

> Exactly - that's the solution for not forcing the crash recovery
> that would modify the database and invalidate it against new .ai files.
> Stay clear of that, and you're fine.

> > When we implemented this (about18 months ago) Progress reccomended
> > this method, but due to the amount of problems reported they no longer
> > do.

> Actually they do recommend it.

 
 
 

Progress, Proquiet and Snapshots

Post by Loki » Sat, 05 Jul 2003 04:41:04




Quote:> Thanks for all your points, it seems yet again that the Progress
> community know more than Progress themselves!

> I referred to them as bugs as that it what Progress called them!

As I said in my other post, there are bugs logged for these issues,
therefore, they are bugs.. :-)

Quote:

> As for the 06 service pack, we installed this recently and are
> monitoring. However we do not hold much hope as Progress also informed
> us that the BI file growth "bug" was fixed in 03, 04 and 05, despite
> no sign of it in the release notes (none in 06 either).

This is often the case if there is no work request associated with the
bug entry. In the bug DB, the WR# is 'Internal' (for one of the bugs I
mentioned earlier), which generally means someone was testing something
and found the problem, and it will not find it's way into the Service
Pack release notes.

Quote:> The reason we truncated the BI files on our DR system is Progress told
> us to! 2 of the 4 databases failed to apply AI files. We were then
> passed to another support person (Bodo Manke, cool name) who said that
> the bug was that you could truncate the BI files on a standby database
> and rfutil shouldn't let you!

Nice idea, but that is not enforcable. rfutil (or proutil rather) has no
idea that this db is going to be participating in future roll forward
operations, therefore we cannot stop anyone from truncating the BI as it
might be part of the "recovery operation". Perhaps a message that the DB
has recently participated in a roll forward operation, asking if you want
to go ahead with the truncate..

Quote:

> I wouldn't mind some more info on the circular ring method mentioned.
> We have read all that Progress can offer on how the AI roll forward
> works with standby BI files, but nothing seems to explain what
> happens. E.G. We have a standby DB with 2 BI extents. One stays at
> 40MB but the other one seems to randomly grow for no reason,  it is
> currently at 680MB and seems to jump around 80 - 120 MB at a time when
> it applies AI files of only 20 - 50 MB. Every few months it blows the
> 2GB limit and we have to rebuild the DB. This is why Progress
> recommended truncating the BI. Which didn't work.

What OS are you running?? Unless you are on Linux or SCO, then 2GB is no
longer a limit. BI files in V9.1B and up can be up to 32TB. SCO & Linux
are still limited to 2GB per file, but you can have multiple of them. I
know this does no solve the issue, but it can prevent you from hitting
the limit.. Potentially, backup the DB, truncate and attempt a roll
forward. If it works, great. If not, restore and try roll forward again,
letting the BI grow a little more..

The weird grwoth you describe has been seen before, but only under very
specific circumstances. Do you have the WR number from Tech support??

What is the current BI cluster size and -G on both the production and
warm spare DBs?

Quote:

> It was the afore mentioned Bodo that told us that Progress were no
> longer recommending this method because of the amount of problems
> reported (however he was trying to sell us Fathom at the same time).

Understandable.. :-) I haven't seen this before, and don't have the time
this week to run any tests, but early next week I should be able to do
some testing.

Quote:

> Regards,
> Rob.

Cheers,

Kevin

 
 
 

Progress, Proquiet and Snapshots

Post by RobL » Tue, 08 Jul 2003 23:35:40


Cheers Kev, escuse the different name, I am logging on from work!.

Work No's are (we have had several as Progress keep closing it and
re-opening it). My manager is now dealing with it so not sure of the
current status.

W302200028

W305130050

 
 
 

Progress, Proquiet and Snapshots

Post by Kevi » Wed, 09 Jul 2003 03:59:01




Quote:

> Cheers Kev, escuse the different name, I am logging on from work!.

> Work No's are (we have had several as Progress keep closing it and
> re-opening it). My manager is now dealing with it so not sure of the
> current status.

> W302200028

> W305130050

No problem.. I'll have a look as soon as I am back in the office (tomorrow
morning)..

Kevin

 
 
 

Progress, Proquiet and Snapshots

Post by Kevi » Thu, 10 Jul 2003 03:52:35




Quote:

> Cheers Kev, escuse the different name, I am logging on from work!.

> Work No's are (we have had several as Progress keep closing it and
> re-opening it). My manager is now dealing with it so not sure of the
> current status.

> W302200028

> W305130050

Hi Rob,

Had a look at the two work requests.. Both are currently closed.. As
mentioned in W305130050 bug #20030122-011 was an internal thing,
apparantley to do with internal monitoring of the bi size, not bi growth.

Since the 9.1D06 serivce pack (I see the las comm was 17/06/03) has there
been any change in the behaviour?? Does the bi still grow??

Do you still have 9.1C on the box?? If you do, do me a favour and check the
startup.pf file in $DLC, specifically for any -G entries..

Cheers,

Kevin

 
 
 

Progress, Proquiet and Snapshots

Post by Rob Lyon » Thu, 10 Jul 2003 20:23:47


Thanks Kevin,
I am not at work today so can not check, however  know -G isn't used as I
checked this before for Bodo.
Things have seemed better for a while (no recent BI growth) so it may be OK
now with 06 on.
We upgraded to 9.1D a while ago so this may have helped.

Thanks for looking into it.
Rob.




> > Cheers Kev, escuse the different name, I am logging on from work!.

> > Work No's are (we have had several as Progress keep closing it and
> > re-opening it). My manager is now dealing with it so not sure of the
> > current status.

> > W302200028

> > W305130050

> Hi Rob,

> Had a look at the two work requests.. Both are currently closed.. As
> mentioned in W305130050 bug #20030122-011 was an internal thing,
> apparantley to do with internal monitoring of the bi size, not bi growth.

> Since the 9.1D06 serivce pack (I see the las comm was 17/06/03) has there
> been any change in the behaviour?? Does the bi still grow??

> Do you still have 9.1C on the box?? If you do, do me a favour and check
the
> startup.pf file in $DLC, specifically for any -G entries..

> Cheers,

> Kevin

 
 
 

1. snapshot agent error: Another snapshot agent for the publication is running

I have created the publishers and distributor on the NT Server run SQL
Server 7.0. This server have a static internet IP.

Then I created a pull subscribers on the Windows 2000 server run SQL Server
2000. This server connect the distributor server through internet. But when
I start the snapshot agent on the distributor, the agent show failed, and
the Action message is "Another snapshot agent for the publication is
running.".

I open the jobs,  the REPL-Snapshot job have 3 step, the third step is
"Detect nonlogged agent shutdown", this step is failed.

How to slave the problem ? Thanks a lot.

2. MS repository to SQL Loader/Oracle?

3. Snapshot won't run due to existing snapshot

4. Ingres vs ms-sqlserver

5. Snapshot Replicaiton: periodic initial snapshot not yet available

6. Access System Table Question

7. Problem: rerun snapshot because current snapshot files are obsolete

8. WANTED:Lotus Agenda Apps and Deffs

9. Snapshot.base.tar.gz vs Snapshot.tar.gz

10. convert regular snapshots to snapshot groups

11. Snapshots/Snapshots Logs

12. Track the progress of Replication by progress bar?

13. Progress Bar when Backup Command is in Progress