How to get log file for rsync operation? Does rsync also delete remote files?

How to get log file for rsync operation? Does rsync also delete remote files?

Post by Goran Ivan » Mon, 05 May 2008 16:33:43



I am planning to use rsync for a daily backup to a remote backup server.

When I read the rsync manual it does not become clear on how to write a log file for the rsync operation.

I am missing options like:

rsync ..... -logfile=\home\backup\rsync.log -logmode=append ......

Did I miss these options?

I want to write (append !!) to the log file:

- Which files were transferred
- When the rsync operation took place
- How much bytes were transferred (total sum) in the rsync operation

How can I get such a log file otherwise?

BTW: Does rsync delete remote files (from previous rsync operations) if they are not existing
any more on the source system ?

Goran

 
 
 

How to get log file for rsync operation? Does rsync also delete remote files?

Post by Martin Kla » Mon, 05 May 2008 16:48:23


Goran Ivanic schrieb:

Quote:> I am planning to use rsync for a daily backup to a remote backup server.

> When I read the rsync manual it does not become clear on how to write a log file for the rsync operation.

> I am missing options like:

> rsync ..... -logfile=\home\backup\rsync.log -logmode=append ......

maybe you are looking for

--log-file=FILE         log what we're doing to the specified FILE

See man rsync.

Martin

 
 
 

How to get log file for rsync operation? Does rsync also delete remote files?

Post by Michael Heimin » Mon, 05 May 2008 20:21:28



Quote:> I am planning to use rsync for a daily backup to a remote backup server.
> When I read the rsync manual it does not become clear on how to write a log file for the rsync operation.
> I am missing options like:
> rsync ..... -logfile=\home\backup\rsync.log -logmode=append ......
> Did I miss these options?

Looks like you didn't RTFM?

      --log-file=FILE
      --log-file-format=FORMAT

Quote:> I want to write (append !!) to the log file:
> - Which files were transferred

Is send by rsync to stdout.

Quote:> - When the rsync operation took place

Simply add a time stamp to your logfile, or see what above
"--log-file" can do for you, i have never used it.

Quote:> - How much bytes were transferred (total sum) in the rsync operation

Is send by rsync to stdout.

Quote:> How can I get such a log file otherwise?

Redirect stdout + stderr to your logfile and add a time stamp
prior to firing up rsync or see what "--log-file" can do for you.

Quote:> BTW: Does rsync delete remote files (from previous rsync operations) if they are not existing
> any more on the source system ?

Yep, using "--delete" as pointed out in the man page. Something
like the following should do the trick (untested):

        rsync -avz --delete /source /destination  2>&1 >> $logfile

I left adding the time stamp as experiment for you.

Good luck

--
Michael Heiming (X-PGP-Sig > GPG-Key ID: EDD27B94)

#bofh excuse 353: Second-system effect.

 
 
 

How to get log file for rsync operation? Does rsync also delete remote files?

Post by Dan Stromber » Tue, 06 May 2008 09:55:49



> I am planning to use rsync for a daily backup to a remote backup server.

> When I read the rsync manual it does not become clear on how to write a
> log file for the rsync operation.

> I am missing options like:

> rsync ..... -logfile=\home\backup\rsync.log -logmode=append ......

> Did I miss these options?

> I want to write (append !!) to the log file:

> - Which files were transferred
> - When the rsync operation took place - How much bytes were transferred
> (total sum) in the rsync operation

> How can I get such a log file otherwise?

> BTW: Does rsync delete remote files (from previous rsync operations) if
> they are not existing any more on the source system ?

> Goran

rsync -avpl --progress --stats should be pretty close to what you want.
 
 
 

How to get log file for rsync operation? Does rsync also delete remote files?

Post by lihao0.. » Tue, 06 May 2008 13:00:50



Quote:> I am planning to use rsync for a daily backup to a remote backup server.

> When I read the rsync manual it does not become clear on how to write a log file for the rsync operation.

> I am missing options like:

> rsync ..... -logfile=\home\backup\rsync.log -logmode=append ......

> Did I miss these options?

> I want to write (append !!) to the log file:

> - Which files were transferred
> - When the rsync operation took place
> - How much bytes were transferred (total sum) in the rsync operation

> How can I get such a log file otherwise?

what *nix box are you using? I have rsync under Ubuntu and RHEL5, and
they use very different ways to handle logfile:

for Ubuntu, I use:

    rsync -aqC --delete -i --log-file=/var/log/mytest.log  SRC DEST
2>&1 1>/dev/null

(logfile are in append mode automatically. at least for my server)

for RHEL5:

    rsync -i --delete -avC SRC DEST 1>>/var/log/mytest.log 2>/dev/null

Quote:> BTW: Does rsync delete remote files (from previous rsync operations) if they are not existing
> any more on the source system ?

   use --delete option, old files which are not on SRC will be removed
from the DESC.

lihao

 
 
 

How to get log file for rsync operation? Does rsync also delete remote files?

Post by Mood » Tue, 06 May 2008 20:56:37



Quote:> I am planning to use rsync for a daily backup to a remote backup server.

> When I read the rsync manual it does not become clear on how to write a log file for the rsync operation.

> I am missing options like:

> rsync ..... -logfile=\home\backup\rsync.log -logmode=append ......

> Did I miss these options?

> I want to write (append !!) to the log file:

> - Which files were transferred
> - When the rsync operation took place
> - How much bytes were transferred (total sum) in the rsync operation

Possibly:

rsync  -auvz -e ssh server:/path/source server:/destination/path  >>
your custom-Logs file

other wise U may use sometime like below:
rsync -auvz  --log-format=FORMAT  -e ssh server:/path/source server:/
destination/path

where you may specify the log-format in your rsyncd.conf if you are
running a rsyncd ( daemon ) ( I never tried this, as I've been using
STDOUT option ^ Above ^  for logging the statistics of file
transfers...

Hope this helps..

Regards,

Quote:

> How can I get such a log file otherwise?

> BTW: Does rsync delete remote files (from previous rsync operations) if they are not existing
> any more on the source system ?

> Goran

 
 
 

How to get log file for rsync operation? Does rsync also delete remote files?

Post by Todd » Tue, 06 May 2008 21:50:44



> I am planning to use rsync for a daily backup to a remote backup server.

> When I read the rsync manual it does not become clear on how to write a log file for the rsync operation.

> I am missing options like:

> rsync ..... -logfile=\home\backup\rsync.log -logmode=append ......

> Did I miss these options?

> I want to write (append !!) to the log file:

> - Which files were transferred
> - When the rsync operation took place
> - How much bytes were transferred (total sum) in the rsync operation

> How can I get such a log file otherwise?

First, I'm not sure you cross posted to enough newsgroups.

You can redirect stderr and stdout to a logfile.

rsync blah  blah       2>&1  >> mylogfile.txt

Quote:> BTW: Does rsync delete remote files (from previous rsync operations) if they are not existing
> any more on the source system ?

It can, if you specify the --delete option.   See man rsync for more.

--
Todd H.
http://www.toddh.net/

 
 
 

How to get log file for rsync operation? Does rsync also delete remote files?

Post by Scott McMilla » Tue, 06 May 2008 23:03:53



Quote:>I am planning to use rsync for a daily backup to a remote backup server.

>When I read the rsync manual it does not become clear on how to write a log file for the rsync operation.

>I am missing options like:

>rsync ..... -logfile=\home\backup\rsync.log -logmode=append ......

>Did I miss these options?

>I want to write (append !!) to the log file:

>- Which files were transferred
>- When the rsync operation took place
>- How much bytes were transferred (total sum) in the rsync operation

>How can I get such a log file otherwise?

With simple redirection:
rsync (your options) >>yourlogfile 2>&1

Quote:

>BTW: Does rsync delete remote files (from previous rsync operations) if they are not existing
>any more on the source system ?

It can. See the --delete option(s) in the man page.

Scott McMillan

 
 
 

How to get log file for rsync operation? Does rsync also delete remote files?

Post by Guillaume Dargau » Wed, 07 May 2008 00:46:23


Quote:> How can I get such a log file otherwise?

rsync ... >> logfile
is good enough for me. Or:
--log-file=FILE         log what we're doing to the specified FILE
--log-file-format=FMT   log updates using the specified FMT

Quote:> BTW: Does rsync delete remote files (from previous rsync operations) if
> they are not existing
> any more on the source system ?

--delete                delete extraneous files from dest dirs
--
Guillaume Dargaud
http://www.gdargaud.net/
 
 
 

How to get log file for rsync operation? Does rsync also delete remote files?

Post by John Murtar » Wed, 07 May 2008 01:58:36



> I am planning to use rsync for a daily backup to a remote backup server.

> When I read the rsync manual it does not become clear on how to write a log file for the rsync operation.

> I am missing options like:

> rsync ..... -logfile=\home\backup\rsync.log -logmode=append ......

> Did I miss these options?

> I want to write (append !!) to the log file:

> - Which files were transferred
> - When the rsync operation took place
> - How much bytes were transferred (total sum) in the rsync operation

> How can I get such a log file otherwise?

> BTW: Does rsync delete remote files (from previous rsync operations) if they are not existing
> any more on the source system ?

      You really should take some time and look at a good 'man' page on
rsync.  Everything you want is supported, here a few starter options:

      -v, --verbose
              This  option  increases the amount of information you are given during the transfer.  By default, rsync works
              silently. A single -v will give you information about what files are being transferred and a brief summary at
              the end. Two -v flags will give you information on what files are being skipped and slightly more information
              at the end. More than two -v flags should only be used if you are debugging rsync.

      --delete
              This  tells rsync to delete any files on the receiving side that arent on the sending side.   Files that are
              excluded from transfer are excluded from being deleted unless you use --delete-excluded.

      --log-format=FORMAT
              This allows you to specify exactly what the rsync client logs to stdout on a per-file basis. The  log  format
              is specified using the same format conventions as the log format option in rsyncd.conf.

      Hope this helps!
--
                                          John
___________________________________________________________________
John Murtari                              Software Workshop Inc.

http://thebook.com/

 
 
 

1. Using rsync to delete remote files as they're transferred?

I have a task where I need to take a collection of files on a remote
server and transfer them, one at a time, a local server, deleting each
remote file as it is successfully transferred.  The reason for the
delete-as-transfer phase is that there are potentially a lot of files
and the task is likely to be interrupted, so it is intended to save disk
space on the remote server (there could be very long periods before a
sync actually fully completes to run a separate cleaning script).

At present I'm doing this with a script that uses ssh to initially get
the list of remote files to transfer, and then loops over each file
doing an scp and then (upon success) doing an ssh to remove the remote
file.  Nothing fancy, naturally, but after starting to use rsync for
other tasks it occurs to me that rsync would be a much better tool if it
had a mode which would allow this behavior (since the current script
involves creating many ssh sessions, exactly the thing that rsync is
designed to minimize).

Can rsync do this?

--

/  \ San Jose, CA, USA && 37 20 N 121 53 W && &tSftDotIotE
\__/ Education is a state-controlled manufactory of echoes.
    -- Norman Douglas

2. Compiling kernel ver 2.0 on a 486 with 8MB of RAM

3. RSync: move 'deleted' files to a specific folder

4. my crontab is no read anymore

5. rsync says some files can not be transferred

6. Data logging

7. Apache with rsync & stale NFS file handles

8. Font problem with X terminals

9. Exclusion of files in rsync with or without wildcards?

10. Best tool to distribute files (rdist, rsync, cfengine)

11. rsync copy directory without files?

12. Which is fastest program for copying files over network e.g rcp, ftp, scp, rsync

13. Log files getting big - how to delete?