Fetch downloads operation files from the remote and merges them into the local log. Encrypted media and thumbnail blobs are not downloaded. It also pulls any mk_*.enc files that are present on the remote but missing locally, so that new users added from another device become available on this one.
Fetch
Algorithm
1. Sync library directory (mk files)
remote_library_files = list all files in library/ on the remote
remote_uuid = parse UUID from the library_id_{uuid} filename in remote_library_files
assert remote_uuid == local library_id // error if mismatch
remote_mk_files = files matching library/mk_*.enc on the remote
local_mk_files = files matching mk_*.enc in local_state/library/
for each file in remote_mk_files not in local_mk_files:
download into local_state/library/ // append-only set of immutable files
2. Update last known state
temp_ops = {}
for each op file listed on the remote:
if not cached in remotes/{remote_id}/state/operations/: download into temp_ops
new_group_ids = group ids from temp_ops
cached_group_ids = group ids from remotes/{remote_id}/state/operations/
if new_group_ids is not a superset of cached_group_ids: return error
write temp_ops to remotes/{remote_id}/state/operations/
3. Merge into local log
local_ids = op_ids in operations.log
for each op file in remotes/{remote_id}/state/operations/:
if compaction file:
for each entry in the compaction:
if op_id not in local_ids: decrypt and append to operations.log
for each MediaCreation op appended:
if the media blob exists on the remote:
record media_id as present in .../state/media/media_list.json
The append to operations.log is the sharpest step in either algorithm: it mutates the
one domain shared by every activity in the library.
4. Rebuild local state
if any ops were appended:
reload in-memory library state from operations.log
Push
Push uploads all files that are considered missing on the target remote, based on its last known state.
For operation files, the client simply copies what it has locally. For media and thumbnail files, since they are downloaded lazily, the client may not have them on disk. Because the last known state of all remotes is stored locally, the push procedure can download the needed files directly from another remote and immediately upload them to the target, without performing a full fetch.
Algorithm
1. Sync library directory (mk files)
remote_library_files = list all files in library/ on the remote
remote_uuid = parse UUID from the library_id_{uuid} filename in remote_library_files
assert remote_uuid == local library_id // error if mismatch
local_mk_files = files matching mk_*.enc in local_state/library/
remote_mk_files = files matching library/mk_*.enc on the remote
for each file in local_mk_files not in remote_mk_files:
upload from local_state/library/ to library/ on the remote
2. Push operations
if pending.op exists:
append it to operations.log
delete pending.op
// determine which op groups need to be uploaded
// by comparing the local log against the last known remote state
ops_to_upload = op groups in operations.log not in last known remote state
compacted_files = compact(ops_to_upload)
upload each file in compacted_files to the remote
apply operation compaction if needed // see Compaction in Operations
update last known state: record every file uploaded or produced by
compaction in remotes/{remote_id}/state/operations/, and remove entries
for any files that compaction deleted on the remote
3. Push media and thumbnails
remote_media_list = remotes/{remote_id}/state/media/media_list.json
for each media_id in reconstructed state not in remote_media_list:
if local_state/media/YYYY/MM/{media_id}.data exists on disk:
upload the data blob
same for thumbnail
add media_id to remotes/{remote_id}/state/media/media_list.json
Duplicate operations on the remote
With concurrent clients pushing to the same remote, or a single client pushing to several remotes, the same operation group can end up uploaded more than once. This is rare in practice. An op group that appears more than once is applied only once to the local log. We do not attempt to detect or remove duplicates on the remote.