How to clone a directory structure?

How to clone a directory structure?

Post by Martin Gro » Mon, 30 Dec 2002 07:08:23



Hi folks,

image you have a (rather large) directory structure beneath
/somedir/parent/.  This structure contains some directories containing
only directories, some only files and some both.  Now, imagine you
want to clone the directory structure to /somedir/parent-clone/.  This
clone should hold *only* the directory structure of the parent and not
any of the files contained in parent.

The clone structure will be filled with files as well, but these files
are different from parent's files.  Whenever a tree in parent is being
added or deleted, this action should be performed on the clone as well
(on demand, not via daemon).  If the action is a deletion in the
clone's tree, then for sure the files should also be deleted which are
in the clone's part which is to be deleted.

Does anyone know about a tool which is capable of doing so?

Thanks and cheers
Martin Gross

 
 
 

How to clone a directory structure?

Post by Davi » Mon, 30 Dec 2002 10:01:43



> Hi folks,

> image you have a (rather large) directory structure beneath
> /somedir/parent/.  This structure contains some directories containing
> only directories, some only files and some both.  Now, imagine you
> want to clone the directory structure to /somedir/parent-clone/.  This
> clone should hold *only* the directory structure of the parent and not
> any of the files contained in parent.

> The clone structure will be filled with files as well, but these files
> are different from parent's files.  Whenever a tree in parent is being
> added or deleted, this action should be performed on the clone as well
> (on demand, not via daemon).  If the action is a deletion in the
> clone's tree, then for sure the files should also be deleted which are
> in the clone's part which is to be deleted.

> Does anyone know about a tool which is capable of doing so?

# Makes a directory tree with all directory permissions
#
# find . -type d | cpio -pdum TARGET_DIRECTORY

--
   Confucius:  He who play in root, eventually kill tree.
Registered with the Linux Counter.  http://counter.li.org

 
 
 

How to clone a directory structure?

Post by Martin Gro » Mon, 30 Dec 2002 20:26:24



> # Makes a directory tree with all directory permissions
> #
> # find . -type d | cpio -pdum TARGET_DIRECTORY

Hi David,

Need tool to create the clone structure, thanks for the idea.
However, that's not the main problem.  Main issue is about maintaining
the clone, i.e. in case I erase a directory inside  the source/parent
tree, it should be deleted with the next "clone update" in the clone
tree as well.

If no other solution come up, I could still use find|cpio and would
only need to recursively traverse the clone for directories which need
to be deleted since they are gone in the parent; but it would be much
more need to have all that in one tool or command line expression.

Any more ideas?

 
 
 

How to clone a directory structure?

Post by Anonymou » Tue, 31 Dec 2002 04:43:11



Quote:> Need tool to create the clone structure, thanks for the idea.
> However, that's not the main problem.  Main issue is about maintaining
> the clone, i.e. in case I erase a directory inside  the source/parent
> tree, it should be deleted with the next "clone update" in the clone
> tree as well.

> If no other solution come up, I could still use find|cpio and would
> only need to recursively traverse the clone for directories which need
> to be deleted since they are gone in the parent; but it would be much
> more need to have all that in one tool or command line expression.

'rsync' is the tool for this job.

#!/bin/sh
# clone a directory stucture (no files copied)
f=/your/source/directory
t=/your/destination/directory
filelist=/tmp/files
mkdir -p $t
cd $f
# get the names of all files/sockets/etc. and remove the initial ./ part

rsync -v  -rlpogDt --delete --force --exclude-from=$filelist . $t
rm $filelist

 
 
 

How to clone a directory structure?

Post by Crd » Tue, 31 Dec 2002 10:09:37


... or may be you can try Unison  
(http://www.cis.upenn.edu/~bcpierce/unison/)

It's a file/dir sync tool, with many options.

Corrado



>> Need tool to create the clone structure, thanks for the idea.
>> However, that's not the main problem.  Main issue is about maintaining
>> the clone, i.e. in case I erase a directory inside  the source/parent
>> tree, it should be deleted with the next "clone update" in the clone
>> tree as well.

>> If no other solution come up, I could still use find|cpio and would
>> only need to recursively traverse the clone for directories which need
>> to be deleted since they are gone in the parent; but it would be much
>> more need to have all that in one tool or command line expression.

> 'rsync' is the tool for this job.

> #!/bin/sh
> # clone a directory stucture (no files copied)
> f=/your/source/directory
> t=/your/destination/directory
> filelist=/tmp/files
> mkdir -p $t
> cd $f
> # get the names of all files/sockets/etc. and remove the initial ./ part

> rsync -v  -rlpogDt --delete --force --exclude-from=$filelist . $t
> rm $filelist

 
 
 

How to clone a directory structure?

Post by Martin Gro » Thu, 02 Jan 2003 22:44:57




> > Need tool to create the clone structure, thanks for the idea.
> > However, that's not the main problem.  Main issue is about maintaining
> > the clone, i.e. in case I erase a directory inside  the source/parent
> > tree, it should be deleted with the next "clone update" in the clone
> > tree as well.

> 'rsync' is the tool for this job.

> (script with find and rsync snipped)

Cool, that's really what I needed.  I will first need to look into how
performance is - for the first steps the $filelist is a 6MB file with
some 60.000 file listings ;-)  In the final application it should end
up with somewhere around 1/3 to 1/2 of that and I am not really sure
about the performance yet ;-)  A simple argument such as --dirs-only
would probably be better, but seems not available.

But it'll work, and that's the point ;-)

Thanks, and a Happy New Year!

 
 
 

How to clone a directory structure?

Post by John-Paul Stewar » Fri, 03 Jan 2003 01:39:05





> > > Need tool to create the clone structure, thanks for the idea.
> > > However, that's not the main problem.  Main issue is about maintaining
> > > the clone, i.e. in case I erase a directory inside  the source/parent
> > > tree, it should be deleted with the next "clone update" in the clone
> > > tree as well.

> > 'rsync' is the tool for this job.

> > (script with find and rsync snipped)

> Cool, that's really what I needed.  I will first need to look into how
> performance is - for the first steps the $filelist is a 6MB file with
> some 60.000 file listings ;-)  In the final application it should end
> up with somewhere around 1/3 to 1/2 of that and I am not really sure
> about the performance yet ;-)  A simple argument such as --dirs-only
> would probably be better, but seems not available.

What about something like:

#!/bin/sh
# clone a directory stucture (no files copied)
# derived from a script by another poster
f=/your/source/directory
t=/your/destination/directory
dirlist=/tmp/dirs
mkdir -p $t
cd $f
# get the names of all directories and remove the initial ./
part

rsync -v  -rlpogDt --delete --force --include-from=$dirlist
. $t
rm $dirlist

The script is untested; use it at your own risk.  All I've
done is change the 'find' parameters to find and list the
directories _only_.  Then I've changed the 'rsync'
parameters to include only those (rather than excluding 6MB
of filenames).

Again, this script is untested and is intended only to
demonstrate the concept.

 
 
 

How to clone a directory structure?

Post by Anonymou » Fri, 03 Jan 2003 03:17:04



Quote:> Cool, that's really what I needed.  I will first need to look into how
> performance is
> ....
> A simple argument such as --dirs-only would probably be better, but
> seems not available.

Send your feature request to the 'rsync' developers.  That's how
programs get better/smarter (and bulkier...)

Quote:> But it'll work, and that's the point ;-)

Good to hear it helped.
Quote:

> Thanks, and a Happy New Year!

You are welcome.
 
 
 

How to clone a directory structure?

Post by Martin Gro » Fri, 03 Jan 2003 22:26:13



> What about something like:
...
> The script is untested; use it at your own risk.  All I've
> done is change the 'find' parameters to find and list the
> directories _only_.  Then I've changed the 'rsync'
> parameters to include only those (rather than excluding 6MB
> of filenames).

Thanks, John-Paul!  Really stupid, I am.  RTFM could have helped a
great deal here.  Your version works when a --exclude '*' is added at
being the --include-from.  However, the script can be even shorter and
become a one-line rsync command, basically:

#!/bin/sh
# clone a directory stucture (no files copied)
# Input from comp.os.linux.misc

f=/tmp/source
t=/tmp/target
cd $f
rsync -v -rlpogDt --delete --force --include '*/' --exclude '*' . $t

Really everything works - besides the only point is that the
combination "--delete --force" somehow doesn't work.  Directories
deleted in source will be deleted in target as well, but only if
target does not contain any *files*.  If the directory in target
contains nothing or directories only, it will be deleted.  But not, if
it does contain any file... As soon as the files are deleted, next
execution of rsync successfully deletes the directories.

I'll do some more checks on this, but I guess this is a bug.  Besides
this, no need to send an email to the developers, a flag
"--directories-only" is not needed since the line above does the job.

Even more funny, in the first script, the combination "--delete
--force" *does* work.  Does anyone have any more hints on what I think
is a bug?

Cheers
Martin

 
 
 

How to clone a directory structure?

Post by Martin Gro » Fri, 03 Jan 2003 22:51:17



> (snipped)

Please forget about the posting I submitted a couple of minutes ago as
reply to John-Paul's posting.  I found the "bug" - it really is not
one, but there is another option which is necessary for this rsync.
The final, working code is:

#!/bin/sh
f=/tmp/source
t=/tmp/target
cd $f
rsync -v -rlpogDt --delete --delete-excluded --force --include '*/'
--exclude '*' . $t

The important part is "--delete-excluded", which according to the man
page is necessary to "also delete excluded files on the receiving
side".  That's it ;-)

I also did a quick check on my real data structure (15k files, 414
directories) and an rsync without changes runs in the range <1 second
(the first script was somewhere around 3 min).  Just the initial rsync
(to generate the clone for the first time) took by far too long, I
stopped it after some minutes when I saw that the clone was already
there (I don't know what rsync was waiting for at that time).

Some more checks will follow, but now I have the solution I needed,
thanks to you all!

Cheers
Martin

 
 
 

1. copying a directory structure with a particular file in directories below

Hello,
I need some help/advise on how to copy a directory structure with a
particular file in directories below the start directory.

Let's say I have many text files [*.txt] with many data files and other
files [*.data, *.*]
The directory I want to copy across to another area with only the text
files but which also retains the directory structure.

i.e.,

[root_directory]
 |__[data]
       |__[directory_1]
       |__[direcotry_2]
       |    |__[directory_2.1]
       |
       |__[directory_3]
       |
       |__[directory_N]

I want to copy the directory [data] to another area with just the *.txt
files but retain the directory structure.

I tried to do this

cp -r `find /[root_directory]/[data] -name "*.txt"` .

NOTE: The above when I include the square brackets is meant to
represent the full name of the
directory and is not to be confused with anything else.

cp -r `find /root_directory/data/ -name "*.txt"` .

When doing this, all I get is the *.txt files without the directory
structure. If the *.txt files have the same name it will cause
problems. So I can copy the directory structure it would be great.

I hope you can understand the issue/problem statement from my
explanation.
If you have any questions please ask me.

Thanks,
ROuNIN

2. Driver for the Dell PERC2 Raid

3. How do you create a directory structure?

4. PPP connects to Linux host but not network.

5. Unix directory structure explainations?

6. deskjet 400 problems

7. Wanted: suggestions for file server directory structure

8. Trying to put a file with Apache and the HTTP 1.1 protocol

9. ftping directory structures

10. How one write an application that is directory structure independent from # include & libray files

11. Graphical representation of directory structure

12. <Q>: Adding another harddrive to existing directory structure

13. Standard Linux directory structure description?