Argthtjtr

I have a directory structure which looks like A1/B1/C1/files111*, A1/B1/C2/files112*, A1/B2/C1/files121*, A1/B2/C2/files122*, etc. so that all files in all directories are unique. Is there a way to use scp (or some other tool) to pull all terminal files files###* at once?

Since you didn't specify this in the question, I will presume that the directory structure is remote, and you want it local.

What you need to do is use wildcards, but escape them on the client side, so that your local shell ignores them and passes them on. Combine that with a recursive flag (-r), and it should perform how you want it:

scp -r remotemachine:some/directory/files*   /some/local/target/directory/

The important part is the backslash, as that makes your local shell ignore the asterisk, and pass it on instead. This should allow you to pull all files* from remote, including subdirectories. It will also pull any actual files whos name begin with files*.

First you have to consider what you want to happen to the files that are copied in this way, and how it may affect future copy operations. Are they going to stay there? Will they be deleted? Archived? Once you know this you can build your copy command, and then the necessary cleanup operations. Here's a sample script that first copies all existing files, then deletes them from the remote server.

There's a risk the files may change between copy and delete operations, so I'd first move them to a temp dir, then copy and delete them from there, allowing the working directory to continue regular operation.

I'd also create a cron job to retrieve these files automatically at the same time each day, and add some sanity checks to ensure the operations are completed successfully.

#!/bin/bash

HOST="RemoteServerIPorHostName"

localDir="/some/local/dir"



for remoteTargets in "/A1/B1/C1/files111*" "/A1/B1/C2/files112*" "/A1/B2/C1/files121*" "/A1/B2/C2/files122*"

do

    scp -pr user@$HOST:$remoteTargets $localDir

    ssh -t user@$HOST 'rm -rf $remoteTargets'

done

ssh user@remote "find /target/A1 -iname "*files*" -exec scp {} user@local:/desitnation ;"

• 
  ssh connects to a remote machine and execute code enclosed by the double quotes ""
  


• 
  find searchs inside of /target/A1 directory for all files containing the keyword "files". Please note you need the backslashes for case insensitive search parameter -iname and the double quotes in order to avoid leaving the previous pair too early.


• 
  -exec is another find parameter, used to execute code. In our case we execute scp and pass all search results from find to scp with the curved brackets {}.


• 
  scp copies your found files to your local or any other machine.

This worked in my test setup as I have exchanged the public keys in order to use authentication without a password.