notes on fedora cloud wg #147

The issue

“Can we not overwrite the previous download location with the new one? I’d prefer for there to be sub directories for each release. In each ‘version’ in atlas that I put out I link to the actual images for that version. If we overwrite the location and delete the old images then those links get broken.”

fedora cloud ML

The relevant code

def stage_atomic_release(
        compose_id,
        compose_basedir=COMPOSE_BASEDIR,
        testing_basedir=ATOMIC_TESTING_BASEDIR,
        dest_dir=ATOMIC_STABLE_DESTINATION):
    """
    stage_atomic_release
        stage the release somewhere, this will remove the old and rsync up the
        new twoweek release
    """
    source_loc = os.path.join(compose_basedir, compose_id, "compose")
    # FIXME - need sudo until pungi perms are fixed
    rsync_cmd = [
        'sudo',
        'rsync -avhHP --delete-after',
        '--link-dest={}'.format(
            os.path.join(
                testing_basedir,
                compose_id.split('-')[-1]
            )
        ),
        "{}/".format(source_loc),
        dest_dir
    ]
    # This looks silly but it gets everything properly split for
    # subprocess.call but keeps it from looking messy above.
    rsync_cmd = ' '.join(rsync_cmd).split()
    if subprocess.call(rsync_cmd):
        log.error(
            "stage_atomic_release: rsync command failed: {}".format(rsync_cmd)
        )
        exit(3)

The desired fix

“This has been discussed offline, we have an agreed upon approach such that the stable/ directory structure will feature date stamped directories and we will keep N and N-1 stable releases there.”

How to test this?

I just ran the script in a Fedora VM a few times to figure out the required dirs and deps:

sudo dnf install -y git python python-requests python2-fedmsg-core fedmsg-relay
git clone https://pagure.io/releng.git
cd releng/scripts/
sudo mkdir -p /pub/alt/atomic/stable 
sudo mkdir -p /pub/alt/atomic/testing/20160719.0
sudo mkdir -p /mnt/koji/compose/twoweek/Fedora-Atomic-24-20160719.0/compose
./push-two-week-atomic.py -k false -r 24

Timestamped subdir

This will match the timestamp plus build number format that comes out of http://alt.fedoraproject.org/pub/alt/atomic/testing/

diff --git a/scripts/push-two-week-atomic.py b/scripts/push-two-week-atomic.py
index 575f71c..f3b7989 100755
--- a/scripts/push-two-week-atomic.py
+++ b/scripts/push-two-week-atomic.py
@@ -419,8 +419,9 @@ def stage_atomic_release(
             )
         ),
         "{}/".format(source_loc),
-        dest_dir
+        os.path.join(dest_dir, compose_id.split('-')[-1])
     ]
     # This looks silly but it gets everything properly split for
     # subprocess.call but keeps it from looking messy above.
     rsync_cmd = ' '.join(rsync_cmd).split()

Now how about the N and N-1 thing? This will let the timestamped dirs pile up.

An easy way out?

In Dusty’s quote from above, he mentions that when he links to images in vagrant’s atlas, his links are broken when the stable release changes. The links that aren’t broken are those in the testing dir, though. In fact, if I’m understanding this rsync correctly, the files that appear in stable are actually hardlinks to the files already in testing, so maybe Dusty could link straight to those…

So these two images are the same file, and the former location doesn’t go away:

http://alt.fedoraproject.org/pub/alt/atomic/testing/20160712.0/CloudImages/x86_64/images/Fedora-Atomic-Vagrant-24-20160712.0.x86_64.vagrant-libvirt.box

http://alt.fedoraproject.org/pub/alt/atomic/stable/CloudImages/x86_64/images/Fedora-Atomic-Vagrant-24-20160712.0.x86_64.vagrant-libvirt.box

Mirroring? And yet, this seems to work:

https://download.fedoraproject.org/pub/alt/atomic/testing/20160712.0/CloudImages/x86_64/images/Fedora-Atomic-Vagrant-24-20160712.0.x86_64.vagrant-libvirt.box

Actually, if these images are allowed to pile up anyway, and if mirrors are mirroring them anyway, and if this is just a matter of hard links… is there any harm to just:

diff --git a/scripts/push-two-week-atomic.py b/scripts/push-two-week-atomic.py
index 575f71c..208b3e8 100755
--- a/scripts/push-two-week-atomic.py
+++ b/scripts/push-two-week-atomic.py
@@ -411,7 +411,7 @@ def stage_atomic_release(
     # FIXME - need sudo until pungi perms are fixed
     rsync_cmd = [
         'sudo',
-        'rsync -avhHP --delete-after',
+        'rsync -avhHP',
         '--link-dest={}'.format(
             os.path.join(
                 testing_basedir,

So, for now, we could either use links from the testing dir when we want longer-lived links, or we could let the images pile up (they contain datestamps in the file names, so they won’t conflict).

Eventually, releng can get around to fixing this in the N and N-1 way, and they can worry about updating the linkage in the websites, etc.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s