dharmik chandel

01 · go-sync ·

What should happen when two devices edit the same file?

the problem
Two devices can edit the same file while one is offline. If the last upload just wins, one of those edits disappears and nobody is told.
what I considered
Last upload wins, locking files, merging edits automatically, or keeping both versions.
what I picked
Keep both. The server turns away any upload based on an old version, and the device saves its own edit as a conflicted copy next to the original.
what I'd change
My first prototype let the server pick the version, so it never caught this. There's still a tiny window where a save during a download can get overwritten.

Where I was

I was rebuilding go-sync, a sync tool that keeps a folder the same on several devices. The server could already store every version of a file, and I was starting on the part that runs on each device and actually syncs. I knew that Dropbox makes "conflicted copies". I didn't yet know how a device could tell who had changed a file.

What made it hard

Here's the case that broke my first prototype. I edit notes.txt on my laptop during a flight. Meanwhile I edit the same file on my phone, and the phone uploads it as version 2. When the laptop lands, it uploads its copy too.

The prototype's server looked up the latest version itself and saved the laptop's file as version 3. The phone's edit was gone. There was no error and no warning, so I would only have noticed if I went looking.

What I weighed

Last upload wins

This is the simplest option, and it's exactly the bug above. One person's work is lost every time two devices overlap.

Locking a file while someone edits it

A device that's offline can't ask for the lock. And a device that dies while holding the lock blocks everyone else. That rules it out for anything that has to work offline.

Merging the two edits automatically

This works for plain text, which is how git does it. But a sync tool doesn't know what's inside a file. Merging two versions of a photo or a spreadsheet byte by byte gives you a broken file.

Comparing just two copies (tried and dropped)

My first idea for the device side was to compare its copy with the server's copy and copy over whichever was newer. But "different" doesn't tell you who changed the file. If the laptop has A and the server has B, the laptop might have edited it, or the phone might have, or both did. Guessing wrong deletes someone's work.

So each device now also keeps the last version it and the server agreed on, in a small local database. Comparing both sides against that version shows who changed what. If both sides changed the file differently, the device keeps both versions.

What tipped it

Locking needs every device to be online, and merging needs to understand the file. A sync tool can count on neither. Of the two options left, only one never loses work: keep both versions and let the person combine them.

How it turned out

  • The server saves an upload only if the file is still at the version the upload started from. The check and the save are one database statement, so two uploads can't both pass. The laptop in the story above now gets "this file has moved on to version 2" instead of overwriting the phone.
  • The decision rules on the device are one small function. A test runs it on all 64 combinations of the three copies. It checks that the function never replaces an edit that hasn't been uploaded, and never overwrites a server change the device hasn't seen.
  • Other tests run two simulated devices against a real database and real file storage. They cover the flight story, an edit against a delete in both orders, a crash right after an upload, and a save landing during a download.
  • I broke each of four safety checks on purpose, one at a time, and a test failed every time.
  • I ran two real copies of the sync program against the full local setup and edited the same file on both while one was stopped. Both folders ended up with the same two files: the original name and a conflicted copy.

Commit: [add the link once it's pushed]

What I still don't know

  • There's a gap of a few microseconds between the last "has this file changed?" check and the moment a download replaces it. A save that lands exactly then is lost. I don't know yet how far to go to close it, since the full fix depends on the operating system.
  • Conflicted copies push the merging onto the person. I'd like to try merging plain-text files automatically and keeping copies only for everything else.
  • No real users have hit a conflict yet, so I don't know how confusing the copies are in practice.
  • On Mac and Windows, Notes.txt and notes.txt are the same file, but on the server they're two different files. I haven't handled that.

What I read

all decisions