Zyklop ◎

This program is a wrapper around rsync. It will help you:

  • if you need to sync files from remote server frequently
  • No need to keep the location of the file in your mind. It finds them for you.

Requirements

  • Python >= 2.6 (Python >= 2.7 for tests)
  • rsync installed
  • locate installed with up-to-date database on the remote system

First Steps

If you are new to ssh, setup an ssh configuration first. If you are dealing with a lot of servers, giving them an alias makes it easier to remember.

  1. Create an ssh configuration in your SSH home, e.g.:

    vim ~/.ssh/config

    You can use the following example as a starter:

    Host spameggs
Hostname  12.112.11.122
Compression yes
CompressionLevel 9
User guido

    but be sure to check the documentation or the man page (5) for ssh_config

  2. Make the config only readable for the owner:

    chmod 600 ~/.ssh/config

  3. Test if you can login to your configured host using only your alias:

    ssh spameggs

  4. Test if locate works on the remote server by searching for a test directory (here called findmesomething):

    locate findmesomething

  5. Now you can copy/sync findmesomething with zyklop:

    zyklop spameggs:findmesomething .

Examples

  1. Syncing a ZODB from remote server configured in ~/.ssh/config as spameggs. We choose not the first database, but the second:

    $ zyklop spameggs:Data.fs .
Use /opt/otherbuildout/var/filestorage/Data.fs? Y(es)/N(o)/A(bort) n
Use /opt/buildout/var/filestorage/Data.fs? Y(es)/N(o)/A(bort) y

  2. Syncing a directory providing a path segment:

    $ zyklop spameggs:buildout/var/filestorage$ .

  3. Syncing a directory which ends with blobstorage`, excluding any other blobstorage directories with postfixes in the name (e.g. blobstorage.old):

    $ zyklop spameggs:blobstorage$ .

  4. Use an absolute path if you know exactly where to copy from:

    $ zyklop spameggs:/tmp/Data.fs .

  5. Syncing a directory which needs higher privileges. We use the -s argument:

    $ zyklop -s spameggs:blobstorage$ .

  6. Dry run prints out all found remote paths and just exits:

    $ zyklop -d spameggs:blobstorage$ .
/opt/otherbuildout/var/blobstorage
/opt/otherbuildout/var/blobstorage.old
/opt/buildout/var/blobstorag

  7. Sync the first result zyklop finds automatically without prompting:

    $ zyklop -y spameggs:blobstorage$ .

Known Problems

Zyklop just hangs

This can be caused by paramiko and a not sufficient SSH setup. Make sure you can login without problems by simply issuing a:

ssh myhost

If that does not solve your problem, try to provide an absolute path from the source. Sometimes users don’t have many privileges on the remote server and the paramiko just waits for the output of a remote command:

zyklop myhost:/path/to/file .

Motivation

I’m dealing with Zope servers most of my time. Some of them have a huge Data.fs - an object oriented database. I do have in 99% of the cases an older version of the clients database on my PC. Copying the whole database will take me ages. Using rsync and simply downloading a binary patch makes updating my local database a quick thing.

To summarize, with zyklop I’d like to address two things:

  1. Downloading large ZODBs takes a long time and bandwidth. I simply don’t want to wait that long and download that much.
  2. Most of the time I can not remember the exact path where the item to copy is on the remote server.

Non Goals

Zyklop is not:

  • a Backup Solution

TODO

  • tty support: sometimes needed if SSH is configured to only allow tty’s to connect.
  • Don’t hang if only password auth is configured for SSH

Development

If you’re interested in hacking, clone zyklop on github:

CHANGES

0.6 (2013-03-01)

  • Documentation and testcoverage improvements.

0.5.2 (2013-02-12)

  • Bugfix: use one function to retrieve the username.

0.5.1 (2013-02-12)

  • Command line utility now shows version information.
  • Bugfix: now uses the ‘user’ configured in the ssh config and falls back to the environment user

0.5 (2013-02-06)

  • Added -d or –dry-run switch to only print out found remote paths by zyklop
  • Added -y or –assume-yes switch to sync the first result found.
  • Fixed b0rked README.rst

0.4 (2013-02-05)

  • Improved documentation
  • Fixed bug, which lead to a hanging command when issuing a remote command in order to find the target path

0.3 (2013-01-14)

  • Changed host, match parameters: Now you can specify the source host and path match in one parameter delimited by a column, similar to scp (e.g. foohost:/path)
  • Allow to provide an absolute path to the file you’d like to copy.

0.2 (2012-03-08)

  • Added basic support for using sudo in finding and syncing (rsync)
  • Added argparse as dependency as Python 2.6 won’t have it
  • New positional argument to provide destination to copy file/directory

API

SSH

Implemented Search

class zyklop.search.TreeNode(name='/', parent=None)[source]

A node in an abstract directory tree.

The tree will degrade to a linked list and is not balanced, nor weightened. All nodes are inserted based on their name. Duplicates are not inserted.

Top most node is labeled with /.

The TreeNode supports access to children via the __getitem__ method and comparisons with __contains__.

traverse(segms)[source]

Traverses the given segments and appends each directory to the tree as a TreeNode (leaf). A leaf is not appended if it already exists.

The path should be already split into segments, eg. /foo/bar should be passed as: ['foo', 'bar']

traverse_path(path)[source]

Traverses given path and appends them to the tree as TreeNodes. This is a wrapper method for traverse().

zyklop.search.absolute_path(node)[source]

Helper function to traverse a given TreeNode and returns an absolute path.

zyklop.search.absolute_path_helper(node, segments=None)[source]

Returns the traversed node segments to create an absolute path in reverse order.

Indices and tables