fssnap, ufsdump and /etc/dumpdates revisited

fssnap, ufsdump and /etc/dumpdates revisited

Post by Chris Thomps » Sat, 20 Jul 2002 08:59:09



Last October there was some discussion of the problems that arise
when taking incremental dumps with ufsdump against a UFS snapshot.
The crux is that the time that ufsdump writes into /etc/dumpdates
is that when ufsdump starts, not that when the snapshot was taken.
Therefore incremental dumps taken relative to that dump may fail
to include files that were last modified during the gap.

As a result of that thread, a sneaky workround occurred to me: get
the snapshot creation time with "fssnap -i -o createtime /what/ever"
and use it to adjust the /etc/dumpdates entry after ufsdump has run.
I am not sure whether I ever posted this.

I hope not, because the scheme has a fatal flaw. The original dump's
header info contains the time it was run (that it is going to write
into /etc/dumpdates), while the incremental dump's header says that
it was taken relative to a dump with the adjusted (earlier) time.
When you do a recursive restore (ufsrestore -r), the increment gets
rejected with "Incremental volume too high" (meaning "relative to
an earlier dump than the one I just restored").

It's a little alarming that it took me nine months to notice this.
So much for all those pious words about testing one's disaster
recovery procedures...  :-(

I suppose I will just have to work on getting Sun to fix the original
bug (e.g. by having ufsdump get and use the snapshot creation time, or
at least by letting the caller specify the reference time).

Chris Thompson
Email: cet1 [at] cam.ac.uk

 
 
 

fssnap, ufsdump and /etc/dumpdates revisited

Post by Rich Tee » Sat, 20 Jul 2002 10:56:50



Quote:> Last October there was some discussion of the problems that arise
> when taking incremental dumps with ufsdump against a UFS snapshot.

I remember that thread...

Quote:> The crux is that the time that ufsdump writes into /etc/dumpdates
> is that when ufsdump starts, not that when the snapshot was taken.
> Therefore incremental dumps taken relative to that dump may fail
> to include files that were last modified during the gap.

> I suppose I will just have to work on getting Sun to fix the original
> bug (e.g. by having ufsdump get and use the snapshot creation time, or
> at least by letting the caller specify the reference time).

Am I missing something here?  When I do my incremental backups
(admitedly the file systems are not that busy), I create a new
snapshot everytime, and remove it once the backup is complete.
There's still a tiny window between the fssnap command and the
ufsdump command, but I don't think I have a problem.  Is there
a reason why you don't do your backups this way?

Curiously,

--
Rich Teer

President,
Rite Online Inc.

Voice: +1 (250) 979-1638
URL: http://www.rite-online.net

 
 
 

fssnap, ufsdump and /etc/dumpdates revisited

Post by Kjetil Torgrim Homm » Sat, 20 Jul 2002 17:44:04


[Rich Teer]:

Quote:

>   Am I missing something here?  When I do my incremental backups
>   (admitedly the file systems are not that busy), I create a new
>   snapshot everytime, and remove it once the backup is complete.
>   There's still a tiny window between the fssnap command and the
>   ufsdump command, but I don't think I have a problem.

time A:  snapshot is taken
time B:  file foo is modified
time C:  ufsdump starts

ufsdump will not see file foo, since it only changed in the real
volume, not the snapshot.  ufsdump will record time C in dumpdates.
in the next run, ufsdump will ignore the file foo, since its mtime B
is before the dumptime C.

our homegrown Veritas backup script probably has the same problem.
ugh.

--
Kjetil T.      /XXX\   /XXX\   /XXX\   /XXX\ /XX\  /XX\  /XX\  /XX\  ///
              ///X\\\ ///X\\\ ///X\\\ ///X\\///\\\///\\\///\\\///\\\///
             //// \\\X/// \\\X/// \\\X/// \XX/  \XX/  \XX/  \XX/  \XX/
            ////   \XXX/   \XXX/   \XXX/     http://folding.stanford.edu

 
 
 

fssnap, ufsdump and /etc/dumpdates revisited

Post by Joerg Schilli » Sat, 20 Jul 2002 19:00:04




>Last October there was some discussion of the problems that arise
>when taking incremental dumps with ufsdump against a UFS snapshot.
>The crux is that the time that ufsdump writes into /etc/dumpdates
>is that when ufsdump starts, not that when the snapshot was taken.
>Therefore incremental dumps taken relative to that dump may fail
>to include files that were last modified during the gap.

Interesting to hear!

I am curently under way of adding true incremental dump features to
"star" and while thinking about the problems, I fiund a lot of problems
that occur when peole try to use GNU tar for incremental backups.
However, I did not find this problem.

Quote:>As a result of that thread, a sneaky workround occurred to me: get
>the snapshot creation time with "fssnap -i -o createtime /what/ever"
>and use it to adjust the /etc/dumpdates entry after ufsdump has run.
>I am not sure whether I ever posted this.

Looks like Sun is trying to be active in the OS joke area:

fssnap -i -o createtime /dev/fssnap/0      
Erstellungszeit fr Bildschirmausdruck: Fr 19 Jul 2002 11:47:36 CEST

which is completely different from:

env LC_ALL=C fssnap -i -o createtime /dev/fssnap/0
Snapshot create time          : Fri Jul 19 11:47:36 2002

Looking at the truss output makes me believe that there is no easy way
for a backup program to find the snapshot creation time.

....

Quote:>It's a little alarming that it took me nine months to notice this.
>So much for all those pious words about testing one's disaster
>recovery procedures...  :-(

>I suppose I will just have to work on getting Sun to fix the original
>bug (e.g. by having ufsdump get and use the snapshot creation time, or
>at least by letting the caller specify the reference time).

For star, the only solution I curently can think of is to tell star to
look for the newest mtime or ctime in the FS and record this time
instead. However, this will definitely be wrong in case you don't use
a snapshot.

--



URL:  http://www.fokus.gmd.de/usr/schilling    ftp://ftp.fokus.gmd.de/pub/unix

 
 
 

fssnap, ufsdump and /etc/dumpdates revisited

Post by Joerg Schilli » Sat, 20 Jul 2002 19:06:38





>> Last October there was some discussion of the problems that arise
>> when taking incremental dumps with ufsdump against a UFS snapshot.

>I remember that thread...

>> The crux is that the time that ufsdump writes into /etc/dumpdates
>> is that when ufsdump starts, not that when the snapshot was taken.
>> Therefore incremental dumps taken relative to that dump may fail
>> to include files that were last modified during the gap.
....
>Am I missing something here?  When I do my incremental backups
>(admitedly the file systems are not that busy), I create a new
>snapshot everytime, and remove it once the backup is complete.
>There's still a tiny window between the fssnap command and the
>ufsdump command, but I don't think I have a problem.  Is there
>a reason why you don't do your backups this way?

You seem to miss that true incremental dumps will inly include the
absolute minimum of delta information.

If somebody e.g. renames a file during this "gap" period, then
the restoring next incremental dump will remove this file completely
from the new FS while it should be renamed only.

--



URL:  http://www.fokus.gmd.de/usr/schilling    ftp://ftp.fokus.gmd.de/pub/unix

 
 
 

fssnap, ufsdump and /etc/dumpdates revisited

Post by Goran Larss » Sat, 20 Jul 2002 20:06:10




Quote:> time A:  snapshot is taken
> time B:  file foo is modified
> time C:  ufsdump starts

> ufsdump will not see file foo, since it only changed in the real
> volume, not the snapshot.  ufsdump will record time C in dumpdates.
> in the next run, ufsdump will ignore the file foo, since its mtime B
> is before the dumptime C.

What about this workaround?

    1. lock fs with "lockfs -w"
    2. take snapshot of fs with "fssnap"
    3. dump snapshot using "ufsdump"
    4. monitor output of ufsdump and when it is done scanning unlock
       the fs with "lockfs -u"
    5. wait for ufsdump to complete
    6. remove the snapshot with "fssnap -d"

The manpages does not say anything about any conflicts between
"lockfs" and "fssnap".

Btw, why can't Sun be more consistent in naming new programs?
Why is the fs at the end of lockfs and at the start of fssnap?

--
G?ran Larsson     http://www.mitt-eget.com

 
 
 

fssnap, ufsdump and /etc/dumpdates revisited

Post by Pawel PIWOWARE » Sat, 20 Jul 2002 20:28:53


Quote:>     1. lock fs with "lockfs -w"
>     2. take snapshot of fs with "fssnap"

snapshot error: File system is locked

Well... keep thinkig.

Quote:>     3. dump snapshot using "ufsdump"
>     4. monitor output of ufsdump and when it is done scanning unlock
>        the fs with "lockfs -u"
>     5. wait for ufsdump to complete
>     6. remove the snapshot with "fssnap -d"

Regards
Pawel
 
 
 

fssnap, ufsdump and /etc/dumpdates revisited

Post by Goran Larss » Sat, 20 Jul 2002 21:31:59




> >     1. lock fs with "lockfs -w"
> >     2. take snapshot of fs with "fssnap"
> snapshot error: File system is locked

Oh well. Why didn't Sun document this?

--
G?ran Larsson     http://www.mitt-eget.com

 
 
 

fssnap, ufsdump and /etc/dumpdates revisited

Post by Christopher Hudnal » Sat, 20 Jul 2002 23:50:33



Quote:> our homegrown Veritas backup script probably has the same problem.
> ugh.

Veritas might be ok, since it maintains a (usually unmanagably huge)
database of backup time AND last mod time on a per-file basis.

-Chris

 
 
 

fssnap, ufsdump and /etc/dumpdates revisited

Post by Rich Tee » Sun, 21 Jul 2002 01:57:04



Quote:> time A:  snapshot is taken
> time B:  file foo is modified
> time C:  ufsdump starts

> ufsdump will not see file foo, since it only changed in the real
> volume, not the snapshot.  ufsdump will record time C in dumpdates.

Agreed, although as the two commands are run one after the
other in my backup script, the time between A and C is VERY
small (especially when the one second granularity of time is
taken into account).

--
Rich Teer

President,
Rite Online Inc.

Voice: +1 (250) 979-1638
URL: http://www.rite-online.net

 
 
 

fssnap, ufsdump and /etc/dumpdates revisited

Post by Joerg Schilli » Sun, 21 Jul 2002 02:52:38




Quote:>> time A:  snapshot is taken
>> time B:  file foo is modified
>> time C:  ufsdump starts

>> ufsdump will not see file foo, since it only changed in the real
>> volume, not the snapshot.  ufsdump will record time C in dumpdates.

>Agreed, although as the two commands are run one after the
>other in my backup script, the time between A and C is VERY
>small (especially when the one second granularity of time is
>taken into account).

Renaming a big file does not take much time.

--



URL:  http://www.fokus.gmd.de/usr/schilling    ftp://ftp.fokus.gmd.de/pub/unix

 
 
 

fssnap, ufsdump and /etc/dumpdates revisited

Post by Christopher Hudnal » Sun, 21 Jul 2002 03:01:20




> > ufsdump will not see file foo, since it only changed in the real
> > volume, not the snapshot.  ufsdump will record time C in dumpdates.

> Agreed, although as the two commands are run one after the
> other in my backup script, the time between A and C is VERY
> small (especially when the one second granularity of time is
> taken into account).

I have worked at many sites where consistent backups are absolutely
critical, which would make such risk unacceptable (Although I never
considered ufsdump as a viable backup method for those sites :).

A few really ugly workarounds come to mind:

1) TZ=Next_Timezone_To_The_West ufsdump yada yada....
I gave this a spin and it works, but could lead to duplicate backups of
the same file. I don't know if it's possible, but maybe you could
create a custom timezone with only a few minutes offset...

2) Use LD_PRELOAD to override gettimeofday(),as mentioned in another thread
as a way to test Y2K issues

3) (Ugliest of all) Start ufsdump as a background process, immediately
pstop it, THEN do the fssnap and resume ufsdump.

Hope this helps!

-Chris (King of Hideous Hacks) :)

 
 
 

fssnap, ufsdump and /etc/dumpdates revisited

Post by Kjetil Torgrim Homm » Sun, 21 Jul 2002 04:02:54


[Christopher Hudnall]:

Quote:

>   I have worked at many sites where consistent backups are
>   absolutely critical, which would make such risk unacceptable
>   (Although I never considered ufsdump as a viable backup method for
>   those sites :).

hah! :) vxdump has the same problem, but that isn't good enough for
you either.

Quote:>   A few really ugly workarounds come to mind:

>   1) TZ=Next_Timezone_To_The_West ufsdump yada yada....

I didn't realise /etc/dumpdates uses local time.  ugh.  it might be
preferable to keep TZ unset when dumping or restoring, but it may trip
up more people than maintaining status quo.

Quote:>   I gave this a spin and it works, but could lead to duplicate
>   backups of the same file. I don't know if it's possible, but maybe
>   you could create a custom timezone with only a few minutes
>   offset...

it's actually quite easy, just edit /usr/share/lib/zoneinfo/src/europe
or similar and run zic on it.  (looks like these files are only
included in Solaris 8 and up.)


  Fri Jul 19 20:59:36 MEST 2002

  Fri Jul 19 20:58:39 CEST 2002

Quote:>   2) Use LD_PRELOAD to override gettimeofday(),as mentioned in
>   another thread as a way to test Y2K issues

>   3) (Ugliest of all) Start ufsdump as a background process,
>   immediately pstop it, THEN do the fssnap and resume ufsdump.

this won't work, since you need to give ufsdump the mounted snapshot
as an argument.  I doubt it will work if ufsdump is allocated even one
timeslice before being stopped.

I think fudging /etc/dumpdates with a little Perl-script is preferable
to hack 2.  hack 1 may actually be workable :-)

--
Kjetil T.      /XXX\   /XXX\   /XXX\   /XXX\ /XX\  /XX\  /XX\  /XX\  ///
              ///X\\\ ///X\\\ ///X\\\ ///X\\///\\\///\\\///\\\///\\\///
             //// \\\X/// \\\X/// \\\X/// \XX/  \XX/  \XX/  \XX/  \XX/
            ////   \XXX/   \XXX/   \XXX/     http://folding.stanford.edu

 
 
 

fssnap, ufsdump and /etc/dumpdates revisited

Post by Rich Tee » Sun, 21 Jul 2002 04:58:45



Quote:> Renaming a big file does not take much time.

Fair point - but I think I'm fairly safe anyway.  The systems
backed up in this manner are pretty much quiescent at the time
the backup happens (around midnight), so the odds of a file being
changed are pretty slim.

--
Rich Teer

President,
Rite Online Inc.

Voice: +1 (250) 979-1638
URL: http://www.rite-online.net

 
 
 

fssnap, ufsdump and /etc/dumpdates revisited

Post by Jeff Mak » Wed, 24 Jul 2002 09:19:23




>The original dump's
>header info contains the time it was run (that it is going to write
>into /etc/dumpdates), while the incremental dump's header says that
>it was taken relative to a dump with the adjusted (earlier) time.

The obvious workaround is to adjust the timestamp in the ufsdump
output headers before it is committed to long-term storage.  This will
be difficult for people who write directly to tape.  But in my case,
at least, ufsdump output already goes through a UNIX pipe so it
wouldn't be a big deal to add another layer.  Now I just need to write
the program in the middle:

    ufsdump | header_adjuster -t snapshot_time | write_to_tape

This will get the job done until Sun provides the option to specify
the time of the dump directly when running ufsdump.

                             :: Jeff Makey

Department of Tautological Pleonasms and Superfluous Redundancies Department

 
 
 

1. /etc/dumpdates file Problem by Using ufsdump Backup Files

Hi all,

Under root, I ran:

# ufsdump 0fu /dev/rmt/0 /export/home/aaa

and then I use "ls -l" to see the /etc/dumpdates file, it is like:

# ls -l /etc/dumpdates
-rw-rw-r--   1 root     sys            0 Nov  5 10:37 /etc/dumpdates

The /etc/dumpdates file contains nothing.

and then I ran:

# ufsdump 5fu /dev/rmt/0 /export/home/aaa

the system still backup whole "/export/home/aaa" into the tape.

What's wrong with my system? How can I fix it? Thanks!

2. Ultra2 shutting down

3. Solaris 8 ufsdump hates /etc/dumpdates

4. USB Mouse and PS/2 Mouse under X

5. ufsdump not writing to dumpdates

6. kpackage not running

7. ufsdump/dumpdates problem

8. change login passwd command

9. ufsdump/fssnap - unlisted file system

10. fssnap - where is /dev/fssnap/* ?

11. /etc/dumpdates?

12. Backup and /etc/dumpdates

13. Matrox Mystique ands X.