The issue
“Can we not overwrite the previous download location with the new one? I’d prefer for there to be sub directories for each release. In each ‘version’ in atlas that I put out I link to the actual images for that version. If we overwrite the location and delete the old images then those links get broken.”
fedora cloud ML
The relevant code
def stage_atomic_release(
compose_id,
compose_basedir=COMPOSE_BASEDIR,
testing_basedir=ATOMIC_TESTING_BASEDIR,
dest_dir=ATOMIC_STABLE_DESTINATION):
"""
stage_atomic_release
stage the release somewhere, this will remove the old and rsync up the
new twoweek release
"""
source_loc = os.path.join(compose_basedir, compose_id, "compose")
# FIXME - need sudo until pungi perms are fixed
rsync_cmd = [
'sudo',
'rsync -avhHP --delete-after',
'--link-dest={}'.format(
os.path.join(
testing_basedir,
compose_id.split('-')[-1]
)
),
"{}/".format(source_loc),
dest_dir
]
# This looks silly but it gets everything properly split for
# subprocess.call but keeps it from looking messy above.
rsync_cmd = ' '.join(rsync_cmd).split()
if subprocess.call(rsync_cmd):
log.error(
"stage_atomic_release: rsync command failed: {}".format(rsync_cmd)
)
exit(3)
The desired fix
“This has been discussed offline, we have an agreed upon approach such that the stable/ directory structure will feature date stamped directories and we will keep N and N-1 stable releases there.”
How to test this?
I just ran the script in a Fedora VM a few times to figure out the required dirs and deps:
sudo dnf install -y git python python-requests python2-fedmsg-core fedmsg-relay git clone https://pagure.io/releng.git cd releng/scripts/ sudo mkdir -p /pub/alt/atomic/stable sudo mkdir -p /pub/alt/atomic/testing/20160719.0 sudo mkdir -p /mnt/koji/compose/twoweek/Fedora-Atomic-24-20160719.0/compose ./push-two-week-atomic.py -k false -r 24
Timestamped subdir
This will match the timestamp plus build number format that comes out of http://alt.fedoraproject.org/pub/alt/atomic/testing/
diff --git a/scripts/push-two-week-atomic.py b/scripts/push-two-week-atomic.py
index 575f71c..f3b7989 100755
--- a/scripts/push-two-week-atomic.py
+++ b/scripts/push-two-week-atomic.py
@@ -419,8 +419,9 @@ def stage_atomic_release(
)
),
"{}/".format(source_loc),
- dest_dir
+ os.path.join(dest_dir, compose_id.split('-')[-1])
]
# This looks silly but it gets everything properly split for
# subprocess.call but keeps it from looking messy above.
rsync_cmd = ' '.join(rsync_cmd).split()
Now how about the N and N-1 thing? This will let the timestamped dirs pile up.
An easy way out?
In Dusty’s quote from above, he mentions that when he links to images in vagrant’s atlas, his links are broken when the stable release changes. The links that aren’t broken are those in the testing dir, though. In fact, if I’m understanding this rsync correctly, the files that appear in stable are actually hardlinks to the files already in testing, so maybe Dusty could link straight to those…
So these two images are the same file, and the former location doesn’t go away:
Mirroring? And yet, this seems to work:
Actually, if these images are allowed to pile up anyway, and if mirrors are mirroring them anyway, and if this is just a matter of hard links… is there any harm to just:
diff --git a/scripts/push-two-week-atomic.py b/scripts/push-two-week-atomic.py
index 575f71c..208b3e8 100755
--- a/scripts/push-two-week-atomic.py
+++ b/scripts/push-two-week-atomic.py
@@ -411,7 +411,7 @@ def stage_atomic_release(
# FIXME - need sudo until pungi perms are fixed
rsync_cmd = [
'sudo',
- 'rsync -avhHP --delete-after',
+ 'rsync -avhHP',
'--link-dest={}'.format(
os.path.join(
testing_basedir,
So, for now, we could either use links from the testing dir when we want longer-lived links, or we could let the images pile up (they contain datestamps in the file names, so they won’t conflict).
Eventually, releng can get around to fixing this in the N and N-1 way, and they can worry about updating the linkage in the websites, etc.
jebpages 