Sync

Fetch

Summary

Fetch lists and downloads remote operation files missing from the remote's last-known state, verifies that the remote has not lost any previously known CRDT dots, then applies their operations to CrdtState and appends them to the local log before rebuilding derived state. It removes local cache blobs for newly learned (and previously interrupted) hard deletions, then confirms which non-tombstoned media the remote holds and records them in that remote's media inventory. It only adds missing mk_*.enc files and does not download media or thumbnails. Fetch never modifies or removes a remote file.

Required access

• Sync policy: Acquire the per-remote sync slot and the global fetch slot.
• Remote: List and read access.
• Local:
  • Read and append to the local operation log.
  • Read and atomically write the remote's last-known operation state (remotes/{remote_id}/state/operations/).
  • Read and atomically write the merge marker (remotes/{remote_id}/state/compact_op_id_merged_to_local.json).
  • Read and opportunistically atomically update the media inventory (remotes/{remote_id}/state/media/media_list.json) under its per-remote mutex, from one listing per media/YYYY/MM/ folder holding an unconfirmed media.
  • Atomically write missing mk_*.enc files in the library directory.
• In-memory local state: Rebuild derived state from CrdtState.

Behavior

#	Behavior	Required authorization
1	Verify the remote and library identity markers, then download each mk_*.enc file present on the remote but missing from the local library. Never remove or replace an existing local master-key file.	List remote Read remote Write local key file
2	Identify the remote operation files that are not already recorded in the last-known state for the remote.	List remote
3	Download and validate those files without changing the last-known state for the remote.	Read remote
4	Verify that the resulting set of remote dots is a superset of dots previously known from this remote. If it is not, fail without writing any operation-related files. Atomically write the validated files to the last-known state for the remote.	Read local state Write local state
5	Apply individual operations from compacted files not already recorded in remotes/{remote_id}/state/compact_op_id_merged_to_local.json to crdt-state.enc and append them to the local log. Applying a duplicate is safe because each CRDT substate is idempotent. Update compact_op_id_merged_to_local.json, then rebuild derived state once.	Read remote Append local log Write local state Rebuild in-memory state
6	Remove every locally cached .data and .thumb file whose known creation metadata is now behind a MediaDeletion tombstone. Missing files are already clean; cleanup failure never reverses the durable tombstone and is retried by a later Fetch or library open.	Delete local media cache file
7	Run the media availability confirmation described below against this remote. It covers all non-tombstoned media, including trashed media, not only the ones created by the operations merged in this run, so a blob uploaded after its creation operation was merged is still discovered. Every error there is ignored, so it cannot fail an otherwise valid fetch.	List remote media folders Write local media list

Media Availability Confirmation

Summary

Confirmation establishes which media blobs a remote holds and records them in that remote's media inventory (remotes/{remote_id}/state/media/media_list.json). It is the only way a blob becomes confirmed without this client having uploaded it.

It runs as the last step of fetch and as the first step of the media stage of push, and it can also be run on its own for one remote, which is how a user repairs incomplete knowledge without performing a full fetch.

Confirmation reads nothing but media folder listings. It never reads an operation file, so it cannot become an implicit fetch.

Required access

• Sync policy: Uses the calling procedure's per-remote sync slot. When run on its own, acquire the per-remote sync slot.
• Remote: List access to media/YYYY/MM/ folders.
• Local:
  • Read the reconstructed media state.
  • Read and atomically update the media inventory under its per-remote mutex.
• In-memory local state: Not modified.

Behavior

#	Behavior	Required authorization
1	Select every data blob and every thumbnail for non-tombstoned media known to the reconstructed state that the inventory does not already confirm. Trashed media remain candidates; a hard-deleted ID never is. A blob stays a candidate until it is confirmed once, and is never probed again after that.	Read local state
2	Group those candidates by the media/YYYY/MM/ folder holding them and list each of those folders once. One listing covers every candidate stored in that folder, so the cost is one remote call per folder rather than one per blob.	List remote media folders
3	Record each blob found in the listings. The data blob and the thumbnail of one media are confirmed independently, so a data blob present on the remote never implies its thumbnail is.	Write local media list
4	Ignore every error. A folder that cannot be listed leaves its candidates unconfirmed, which is the same as never having looked. Confirmation is opportunistic bookkeeping and never fails its caller.

Confirmation only ever adds. It never records a blob as absent and never withdraws a confirmation, so an unconfirmed blob means unconfirmed, not absent from the remote. A hard-deleted ID is excluded from confirmation even if this cache still positively records it; a trashed ID remains eligible, because its encrypted blobs must be available for Restore.

Push Preparation

Summary

Push never searches for media. Everything it needs must be decided before it starts, so a separate preparation resolves, for every blob the target is missing, where to get it: from the local media cache, or from one remote already known to hold it. It resolves data blobs and thumbnails separately, so one media can take its original from one place and its thumbnail from another.

The two are not equally required. A data blob it cannot place fails preparation, because uploading the library without it would lose the media. A thumbnail it cannot place is simply left out of the plan, because a thumbnail is derived data and nothing records whether a media ever had one. Blocking a push on a thumbnail would let a media that never had one block every push of the library forever.

Resolution uses only local knowledge, the local media cache and the media inventories of the known remotes. It performs no remote access at all, so it is instant, deterministic, and usable offline.

Required access

• Sync policy: None. Preparation touches no remote and no shared local state.
• Remote: None.
• Local:
  • Read the reconstructed media state and the local media cache.
  • Read the media inventory of the target and of each candidate source remote.
  • Read the configured media source priority order.
• In-memory local state: Not modified.

Behavior

#	Behavior	Required authorization
1	Take the data blob and the thumbnail of every non-tombstoned media known to the reconstructed state as two separate items, including trashed media, and drop the ones the target's own inventory already confirms. Everything below resolves one blob, never a media: the two blobs of one media are missing, found, and fetched from independently.	Read local state Read local media list
2	For every remaining blob the local media cache holds, that cache is where to get it. A local copy is always preferred, since using it costs no download.	Read local media cache
3	For a blob that is not cached locally, where to get it is the first remote in the configured source priority order, other than the target, whose own media inventory confirms that blob.	Read local media lists
4	If there is one data blob for which preparation does not know where to get it, fail before push starts and report the media it belongs to. Nothing is uploaded and no remote is contacted.
5	Leave out any thumbnail for which preparation does not know where to get it, and carry on. That thumbnail stays unconfirmed on the target, and a later push retries it once some remote's inventory confirms it.

Unresolved media

Preparation fails as soon as there is one data blob it does not know where to get. This is not necessarily a lost media: some remote's media_list.json may simply be out of date.

The client then asks the user which remotes to confirm, which updates those inventories without fetching, and preparation is retried. An automatic push has no one to ask and fails immediately.

Push

Summary

The lifecycle-specific ordering and recovery rules are specified in Media deletion.

Push brings a target remote up to date with the local library by uploading missing master-key and operation files, then uploading any active or trashed media and thumbnails referenced by the reconstructed local state that are not already known to be on that remote. When a required media file is absent locally, push can relay it from another remote without first performing a full fetch.

Push uploads all files that are considered missing on the target remote, based on its last known state.

For hard deletion, Push first delivers the ordinary encrypted MediaDeletion operation. Only after that operation is durable on the target may it delete that media's .data and .thumb keys. It then removes the IDs from that target's local positive inventory. Remote deletion is idempotent: an absent key is already reclaimed. A failed cleanup leaves the tombstone durable and is retried on the next Push to that remote. This makes physical deletion eventual across remotes, rather than a cross-remote transaction.

The media inventory is a positive hint, never authority for a transfer. Current CRDT state decides whether an ID is eligible to upload or fetch. Therefore a device may push an active item, skip its upload because its target inventory positively remembers the blob, then later Fetch that target's tombstone and delete the local cache. Push is intentionally not fetch-first.

For operation files, the client selects operations from the complete local operation log whose dots are not covered by the target remote's cached operation files, then writes them as one immutable compaction file. For media and thumbnail files, since they are downloaded lazily, the client may not have them on disk. Because the last known state of all remotes is stored locally, the push procedure can download the needed files directly from another remote and immediately upload them to the target, without performing a full fetch.

Media source

Besides the target, push takes one media source, which is the complete description of where it may read a blob it does not have locally. Push never looks anywhere else, and never searches: it cannot decide on its own to try a remote, which is what keeps it from turning into an implicit fetch.

Media source	Meaning
None	Upload only what the local media cache holds. A blob missing from the target and absent locally fails the push, which reports the media concerned without uploading anything. This is the default, so a push can never quietly download from a remote the caller did not name.
One relay remote	One read-only remote storage, used for every blob absent locally. Its identity and library format are verified before anything is read from it.
A resolved plan	A map from each missing blob, a data blob or a thumbnail, to where to get it, either the local cache or one named remote, together with a read-only storage for each named remote. Push reads no remote absent from that map. A data blob the map does not cover fails the push rather than being searched for, while a thumbnail it does not cover is simply not uploaded. The map is produced by push preparation, described above.

Required access

• Sync policy: Acquire the per-remote sync slot.
• Remote: List, read, write, and delete access on the target. Read access on each remote named by the media source, and on no other.
• Local:
  • Read local master-key files, reconstructed derived state, and media files.
  • Read the remote's last-known operation state (remotes/{remote_id}/state/operations/) and the media inventory (remotes/{remote_id}/state/media/media_list.json).
  • Read the append-only operation log; no pending operation file exists.
  • Atomically write the remote's last-known operation state and, under its per-remote mutex, the media inventory.
• In-memory local state: Already rebuilt after each local CRDT merge.

Behavior

#	Behavior	Required authorization
1	Verify that the target remote belongs to this library, then upload local mk_*.enc files that are absent from the remote. Never remove a remote master-key file.	List remote Write remote key file
2	Read the complete local operation log and the dots covered by this remote's cached compaction files.	Read local operation log Read local cache
3	Select operation-log entries whose dots are not covered by the target remote, upload one immutable compaction file containing those operations, and record the identical ciphertext in the last-known state.	Read local operation log Read local cache Write remote operation file Write local state
4	When compaction is needed and its remote lock can be acquired, merge the applicable remote operation files, upload the replacement, delete its source files, and update the last-known state.	Read remote operation files Write and delete remote operation files Write local state
5	For every hard-deleted media with known creation metadata, delete its .data and .thumb keys from the target after the tombstone operation has been uploaded. Remove successfully reclaimed IDs from the target's local positive media inventory. Availability confirmation already ran against the current non-tombstoned state, so deletion never causes a hard-deleted blob to become an upload candidate.	Delete remote media and thumbnail Write local media list
6	After reclamation and cloud-quota admission, for each active or trashed blob still unconfirmed in the target's media inventory, read it from the local cache or from the remote the media source names for it, then ensure it exists on the target with a non-overwriting write and record what the target now holds. Each blob is handled on its own, so a media whose data blob is already confirmed is still visited for its thumbnail alone, and a media whose thumbnail is nowhere to be found simply stays unconfirmed for it.	Read local media Write remote media and thumbnail Write local media list

Duplicate operations on the remote

With concurrent clients pushing to the same remote, or a single client pushing to several remotes, the same operation can end up uploaded more than once. Merge and local-log reads deduplicate by dot; remote duplicates are harmless and are not deleted just for duplication.

Compaction Procedure

Summary

Compaction bounds the number of remote operation files by merging every known file in an overfull tier into one file in the next tier. It runs as part of push and operates only on the target remote's last-known state; it never lists or reads arbitrary remote operation files and therefore cannot become an implicit fetch.

Required access

• Sync policy: Uses push's per-remote sync slot.
• Remote: Write and delete access.
• Local: Read, write, and delete access to the target remote's last-known operation state.

Behavior

#	Behavior	Required authorization
1	When a known tier reaches its file limit, acquire the global remote operations lock. If the lock is already held, skip compaction for this push.	Create remote lock
2	Read every source file from the target's last-known state, concatenate and deduplicate their operations by dot, upload one replacement file at the next tier, and record that file in the last-known state.	Read local state Write remote operation file Write local state
3	Delete each source file from the remote and remove it from the last-known state after its deletion succeeds.	Delete remote operation file Delete local state
4	Repeat for the next tier while it exceeds its file limit, then release the lock.	Delete remote lock