lib/fs: Encode filenames containing reserved characters (fixes #1734)

[rasa]

Purpose

Many commonly used filesystems do not allow certain characters or names in file and directory names. For example, the NTFS, exFAT, FAT32 and reFS filesystems do not allow the characters /\"*:<>?| in the file name or directory name component of a path. On Windows. reserved filenames such as CON and Nul.txt, and filenames ending with a period, or space are also not allowed. Syncthing currently rejects these files causing out-of-sync errors.

Windows Subsystem for Linux (WSL), Cygwin, Msys2, git-bash, Linux's CIFS driver, and other similar systems solve this issue by encoding these characters, to allow filenames containing these characters to be saved to disk. They do this by replacing the reserved characters with UNICODE characters in the private use area (\uf000-\uf0ff). This conversion system was first developed in Interix over 25 years ago.

This PR adds an advanced folder option titled "Encoder Type" which has three options: passthrough, fat, and standard.

1. The passthrough encoder sends and receives encoded filenames without decoding them. This is how Syncthing currently works, and is the default option.
2. The fat option encodes as described above.
3. The standard option does not encode, it only decodes. While the standard encoder doesn't encode, it ignores all other encoded filenames, in case a user sets the encoder type, and then sets it back to standard.

Here's a table of the different encoders:

Name	Filesystem(s)	Encodes	Decodes?	Reserved Filenames on Windows	Comments
passthrough	Any	(none)	No	Fail	Filenames containing encoded characters are sent & received and read/written on disk as is.
fat	NTFS, exFAT, FAT32, reFS	"*:<>?|, \x01-\x1f	Yes	Fail	100% compatible with WSL/Cygwin/git-bash/Msys2, etc. On Windows, file or directory names with trailing periods or spaces are still rejected.
standard	APFS, Btrfs, ext4, etc.	(none)	Yes	Fail	Filenames containing encoded characters are not sent or received, and are ignored on disk.

Windows reserved filenames

If this PR is accepted, we may want to address Windows reserved filenames such as CON and Nul.txt, and filenames ending with a period, or space, via two new encoders:

Name	Filesystem(s)	Encodes	Decodes	Reserved Filenames on Windows	Comments
windows	NTFS, exFAT, FAT32, reFS	"*:<>?|, trailing spaces and periods, \x01-\x1f	Yes	Succeed (con encoded as co\uf06e)	On Windows, reserved filenames can be opened in File Explorer, CMD, etc. Reserved filenames are incompatible with WSL/Cygwin/git-bash/Msysw, etc, as they are encoded, whereas WSL saves reserved filenames as is.
wsl	NTFS, exFAT, FAT32, reFS	"*:<>?|, \x01-\x1f	Yes	Succeed (con saved as con via \\?\ prefix)	100% compatible with WSL/Cygwin/git-bash/Msys2, etc. On Windows, reserved filenames cannot easily be opened in File Explorer, CMD, etc. (The user would need to prefix the full path with \\?\).

Unencoded characters

None of the encoders encode NUL (\x00). While some filesystems, such as HFS+ allow file and directory names to contain slash (/) characters, none of the encoders in this PR encode this character. On Windows, this PR does not encode file and directory names containing backslashes (\). These unencoded characters will result in out-of-sync errors on systems that don't allow these characters.

Testing

All the tests pass. go vet is clean. go lint is clean. go fmt produces no output.

I have tested it with Windows, Linux, Android 11, macOS and iOS clients pushing changes to a Windows client. Directory listings showed the correct filenames in WSL, Cygwin, Msys2, and git-bash environments. I also created filenames containing reserved characters in WSL, Cygwin, Msys2 and git-bash, and these transferred successfully to Windows, Linux, Android 11, macOS, and iOS clients.

Documentation

If this PR is accepted, I will draft a PR to update the documentation.

Testing

To test this PR, simply switch to this branch on all nodes. The passthrough encoder is enabled by default.

To test the other encoders, all nodes need to be set to either standard or fat. Typically, unix like systems will use standard and Windows (NTFS), and FAT-based systems such as Android, will use fat.

While standard can't be used on FAT/NTFS-based systems without errors, fat can be used on any system, but it makes little sense, if the underlying file system is not FAT/NTFS-based.

Authorship

My name and email are already in the AUTHORS file.

Links

For reference, see:
https://cygwin.com/cygwin-ug-net/using-specialnames.html
http://msdn.microsoft.com/en-us/library/aa365247%28VS.85%29.aspx
https://en.wikipedia.org/wiki/Filename#In_Windows
https://docs.microsoft.com/en-us/windows/win32/fileio/naming-a-file

For implementations, see
https://github.com/mirror/newlib-cygwin/blob/fb01286fab9b370c86323f84a46285cfbebfe4ff/winsup/cygwin/path.cc#L435
https://github.com/billziss-gh/winfsp/blob/6e3a8f70b2bd958960012447544d492fc6a2f1af/src/shared/ku/posix.c#L1250
https://github.com/torvalds/linux/blob/master/fs/cifs/cifs_unicode.h#L27

[calmh]

I just read some comments and thought briefly about the overall thing, on the assumption that you might want this merged at some point.

[calmh]

Does this mean that file names with the magic unicode characters in them, which are synced using passthrough or in previous versions of Syncthing, will be deleted when switching to standard?

[rasa]

Thanks for reviewing. I think you're right, the passthrough encoder needs to be the default. I need to rethink this whole concept. Edit: The passthrough encoder is now the default.

[imsodin]

Doesn't this result in an inconsistent state accessible through a filesystem interface, as in a file might be filtered here, but I can stat, delete, ... it? I'd expect the same treatments for names on all filesystem methods.

[rasa]

That's a good question. Presumably, Syncthing wouldn't "see" the files to know to stat or delete them, as the directory list and walk functions ignore these files on disk. But perhaps it found them in the database, from a previous scan. If we want all fs functions to skip over these files (i.e., treat them as non-existent) I will need to add logic to do so. I think this logic could live in fs.rooted(), which can return a not found error. This is a good idea. I'll need to add some tests for this too.

[rasa]

I'm about to push these changes to this PR. Unfortunately, I recreated the add/fs-encoder branch off of main, and will force-push these changes. I hope this is acceptable. My apologies if not.

[JanKanis]

The passthrough encoder is the default, so the new encoding logic is opt-in. In fact, it only makes sense if all the nodes opt-in at once, so it's a bit of a pain to enable.

Why would that be the case? If a single node enables fat encoding, it should still receive and send the unencoded names that include the reserved characters over the wire, so from the perspective of other nodes nothing changes. Right?

And why would you want to enable the fat encoder on a non-fat linux node?

[JanKanis]

As an alternative to the proposed encoding here, Samba has to solve the same problem. There a recommended (as in: used as the primary example) replacement strategy is to replace the reserved characters with rare unicode characters that look similar:

"→¨
*→¤
/→ø
:→÷
<→«
>→»
?→¿
\→ÿ
|→¦

This has the advantage that the characters are still recognizable when viewed with another app, unlike the private use area replacements.
This will need to be configurable as what is a 'rare character' depends on locale and usage.

reference: https://www.samba.org/samba/docs/current/man-html/vfs_catia.8.html

[bt90]

This will need to be configurable as what is a 'rare character' depends on locale and usage.

To quote Shakespear: That way madness lies

[JanKanis]

To quote Shakespear: That way madness lies

The Samba guys apparently disagree. Of course there is a trade between simplicity and having everything configurable, and if there was a solution that would just work for everyone, simplicity would be the way to go. An option that would be simpler on the configuration would be to have another encoder that encodes to Samba's visually similar characters.

[rasa]

Why would that be the case? If a single node enables fat encoding, it should still receive and send the unencoded names that include the reserved characters over the wire, so from the perspective of other nodes nothing changes. Right?

If a node uses the fat encoder, the other nodes need to use either the standard encoder (non-fat filesystems), or the fat encoder (fat filesystems).

Why? Because all nodes default to the passthrough encoder, which doesn't ignore the "rare Unicode character" (encoded) filenames (say acolon\xf03a) (How Syncthing works now). The standard encoder does ignore encoded filenames.

Why? Because we want to remain backwards compatible, and we need to avoid creating both encoded filenames (acolon\xf03a), and unencoded filenames (acolon:) if the filesystem allows. Hence the three encoders.

And why would you want to enable the fat encoder on a non-fat linux node?

You wouldn't. You would enable the standard encoder. If you could guarantee the filesystem doesn't have any encoded filenames, you could leave it as the passthough encoder. But if an encoded file appears, the passthrough encoder will send it "as is", and the fat and standard encoders will then ignore it, leading to out-of-sync errors. See here for the relevant comment.

[rasa]

As an alternative to the proposed encoding here, Samba has to solve the same problem. There a recommended (as in: used as the primary example) replacement strategy is to replace the reserved characters with rare unicode characters that look similar:

"→¨
*→¤
/→ø
:→÷
<→«
>→»
?→¿
\→ÿ
|→¦

We could add easily add this encoding logic to this PR, but the characters 0x01 - 0x1f would still be rejected leading to out-of-sync errors. Not a big deal, as these characters are pretty rare in filenames, and often are created by accident.

But using this encoding doesn't solve the problem of existing files with these encoded characters in their filenames. Syncthing will see these filenames as encoded, and will then decode them, leading to out-of-sync errors. As they say, the devil's in the details.

[JanKanis]

The passthrough encoder is the default, so the new encoding logic is opt-in. In fact, it only makes sense if all the nodes opt-in at once, so it's a bit of a pain to enable.

If a node uses the fat encoder, the other nodes need to use either the standard encoder (non-fat filesystems), or the fat encoder (fat filesystems).

Why? Because all nodes default to the passthrough encoder, which doesn't ignore the "rare Unicode character" (encoded) filenames (say acolon\xf03a) (How Syncthing works now). The standard encoder does ignore encoded filenames.

Why? Because we want to remain backwards compatible, and we need to avoid creating both encoded filenames (acolon\xf03a), and unencoded filenames (acolon:) if the filesystem allows. Hence the three encoders.

As far as I can tell, escape characters can only end up on non-updated nodes if a node first enables the fat encoder, and then switches back to passthrough without removing the encoded files. As long as you don't do that, there is no problem, and there is plenty of time to update other nodes.

I haven't read through all the changes of this PR in detail, but in my understanding it should work as follows: Say node A is a linux system that allows all characters in filenames, and node B is a windows system that doesn't allow colons, among others. Currently node A sends file acolon: to B, which can't store that filename so won't save that file (and show the user an error message, if they check the folder status).
Now the installation on node B is updated with the current PR and the fat encoder is selected. Node A sends acolon: to B again, which encodes it and stores it as acolon\xf03a on its FAT (or NTFS) filesystem. But when node B advertises its files to node A, it decodes the filename before sending it to A, so A just hears that node B now also has acolon: stored. From A's perspective nothing has changed, except that B now stores the file. The encoded characters never leave B.

Only if then node B switches back to the passthrough encoder is the filename with encoded characters acolon\xf03a advertised to other nodes, and is there a risk of that filename spreading. As long as you don't switch back to passthrough (for which there isn't really any reason) there's no need to update other nodes in a hurry.

---

But using this [Samba style] encoding doesn't solve the problem of existing files with these encoded characters in their filenames. Syncthing will see these filenames as encoded, and will then decode them, leading to out-of-sync errors. As they say, the devil's in the details.

That risk always exists, also when using PUA unicode characters. They are 'private use area', so any app can use them however it wants to. There's bound to be someone somewhere trying to sync files that use those PUA characters. Which is why this encoder logic should be opt-in and allow the user to make a choice.

Although as far as I understand Syncthing, it won't lead to out-of-sync errors, Syncthing will just assume that the decoded name is the real name and sync that.

But I agree such an encoder could be added as an alternative to the fat encoder, and that it doesn't handle control characters but that probably is not worth caring about. I would guess control characters in file names are rarer than PUA unicode characters.

[rasa]

Only if then node B switches back to the passthrough encoder is the filename with encoded characters acolon\xf03a advertised to other nodes, and is there a risk of that filename spreading. As long as you don't switch back to passthrough (for which there isn't really any reason) there's no need to update other nodes in a hurry.

I agree, but since how Syncthing works now is effectively running as the passthrough encoder, downgrading Syncthing would “switch back”.

One way to solve this issue is: on startup, the passthrough encoder scans all the filenames in the database for encoded filenames. If it doesn’t find any, it switches to using the standard encoder, and updates the config file to use the standard encoder from then on. The EncoderSelected flag remains false (to indicate the user didn’t select the encoder, Syncthing did.)

If on that first scan, encoded filenames are found, the folder is paused, and the GUI would report the number of files found, and maybe the first 10 filenames, or so, and asks the user to either:

1. Ignore the files (by switching to the standard encoder permanently).
2. Accept the filenames “as is” (by using the passthrough encoder).

If the user makes a selection, the EncoderSelected flag is set to true, and the folder is unpaused. When EncoderSelected is set, the filenames are not scanned on future startups.

If EncoderSelected is false, and the standard encoder ever encounters an encoded file, it would report the filename in the GUI, and log, but ignore it.

The EncoderSelected flag can be set/reset via folder options.

For CLI users, we add an --encoded-files [ignore|accept] option. If this option is not provided, and the EncoderSelected flag is not set, and the passthrough encoder finds encoded files, it would report the encoded filenames, and pause the folder. If the CLI option is provided, the folders would be switched to use the standardmencoder, and EncoderSelected flag would be set for all the folder scanned. Encoded filenames would be reported.

Lastly, --encoded-files report would simply report any encoded filenames found for any passthrough folders, and then quit.

If this makes sense, and there is interest, I would consider adding this logic to this PR. But I would prefer this PR be accepted first, before I start on that effort.

[acolomb]

Hm. I really don't like pausing folders because some condition was detected during an upgrade. The likelihood of discovering that situation is very low, as upgrades usually happen without user interaction and without any notice. It causes trouble for someone relying on the synchronization and noticing a paused folder only after some data is missing, e.g. when used as part of a backup strategy.

[rasa]

Hm. I really don't like pausing folders because some condition was detected during an upgrade.

@acolomb It doesn't need to pause the folder. It could simply default to the standard encoder, and in the off chance the user wants to sync encoded filenames, they would need to change to the passthrough encoder.

[JanKanis]

Only if then node B switches back to the passthrough encoder is the filename with encoded characters acolon\xf03a advertised to other nodes, and is there a risk of that filename spreading. As long as you don't switch back to passthrough (for which there isn't really any reason) there's no need to update other nodes in a hurry.

I agree, but since how Syncthing works now is effectively running as the passthrough encoder, downgrading Syncthing would “switch back”.

One way to solve this issue is: on startup, the passthrough encoder scans all the filenames in the database for encoded filenames. If it doesn’t find any, it switches to using the standard encoder, and updates the config file to use the standard encoder from then on. The EncoderSelected flag remains false (to indicate the user didn’t select the encoder, Syncthing did.)

[...]

This sounds way too complicated and magical for the quite niche risk it tries to prevent. Note that unwanted encoded filenames can only spread after manual changes (changing to the fat encoder) and then downgrading or switching back the encoder. And we are not looking at data loss or data corruption, only at duplicated files or conflicts*, which a simple script could detect and fix. (Documentation could include such a script, in case people are in need of it.)
Also, this solution won't help in the upgrade one node - enable fat encoder - downgrade node scenario, as then all this new logic is also gone after the downgrade. I don't think any new code can fix that scenario as any new logic would be gone. But I also think it doesn't matter much as the chance of that happening is quite small and the result is fixable.

* at least, based on my limited knowledge of how Syncthing works. If this could result in data loss the situation is different.

If we are discussing other ways to implement this feature, let me make a different proposal:

Let there be two settings:

• encoder: [none, fat (, maybe "map restricted characters to ¨¤ø÷«»¿ÿ¦ or others)]
• disallowed file name characters (multiple choice):
  ☑ Unicode private use area characters \xf000-\xf0ff (these are used by the 'fat' encoder to encode characters that some file systems do not allow)
  (☑ optionally ¨¤ø÷«»¿ÿ¦, or others)

Selecting an encoder requires the relevant disallowed file name characters to also be checked. If an option to disallow characters is selected, Syncthing will reject incoming files containing those characters and ignore them if it finds them on its disk (unless they are used by an active encoder). In both cases a message will be shown in the GUI and the folder will show as 'out of sync'. The documentation might encourage users to verify everything still syncs correctly after changing one of these settings. Or maybe that is superfluous, changing folder settings always risks breaking the sync.

For existing folders we default to no encoder and no characters disallowed. For new folders we could default to disallowing the \xf000 range, in order to make it easier for users to enable the fat encoder. Users could just change the settings if they wanted something different.

Some considerations:

• I think this pair of settings is easier to understand for users than the proposal with both a passthrough and standard encoder, and much easier than the proposal that makes automatic changes to the encoder settings.
• Ideally, the "disallowed file name characters" should be a folder-wide setting that is automatically synced to all nodes for that folder, instead of a setting local to a node. But that may be too much to implement in a single pull request.
• This doesn't address the upgrade - enable fat - downgrade scenario from above, but as I argued there, that can't be fixed, but documentation and a script to find/fix such cases should be sufficient.
• I think it is better to really have "Up to date" mean all files are present on all nodes, instead of silently ignoring certain files. Silently ignoring files will give more problems than it solves, especially for less technically inclined users. If someone really wants silently ignored files, they can do so with .stignore.
• It should be possible to select multiple sets of disallowed characters. That setting should be the same for all nodes sharing a folder, but different nodes may want to use different encoders.
• If someone really wants all characters in filenames and have it work on fat, there could be an encoder with an escape, e.g. \xf03a is an encoded :, \xf000\xf03a is an encoded \xf03a. The only limit then would be the file name length. But that should be left to a different proposal.
• Another "disallowed characters" option that might be useful: Disallow "*/:<>?|. That would fix syncing with Windows/Fat systems the other way.

Hm. I really don't like pausing folders because some condition was detected during an upgrade.

@acolomb It doesn't need to pause the folder. It could simply default to the standard encoder, and in the off chance the user wants to sync encoded filenames, they would need to change to the passthrough encoder.

If the passthrough encoder includes extra logic to reject certain files or automagically switch to other encoders, that could lead to problems when Syncthing is (automatically) updated. IMO existing folders should not change behavior without explicit user action, so there is no risk of syncing silently breaking.

[rasa]

@JanKanis I really appreciate your feedback, and you have some great ideas that I haven't considered before. Perhaps we move this discussion to a wiki page, where we come up with a design document, and see if it gets any traction. If it does, I am willing to scrap this PR, and steal its best bits on a new one.

[JanKanis]

@rasa Sounds fine to me. While I am a developer, I don't know anything about Syncthing development procedures. I just ended up at this PR because I want to sync files from linux to android that contain reserved characters. I just had a look at the wiki and it looks like design work happens there, so go ahead.

[rasa]

@JanKanis Sorry for the long delay. I finally got the initial draft posted to #9539 . Let me know your thoughts. I know it's different from what you described, but I think it captures the essence of your ideas: only two encoders, and allowing the FAT encoder to re-encode if needed (though using a different encoding scheme, as 0xf000 would decode to NUL (in GitBash/WSL/etc.), leading to issues.

I thought long and hard about your idea that "For new folders we could default to disallowing the \xf000 range". But in the end, I think if a user wants this functionality they can add *[-]* to the folder's .stignore as you yourself said "If someone really wants silently ignored files, they can do so with .stignore." :)

If you and others give me the go ahead, I will close this PR, and start a new one, since the logic is going to be much simpler. Thanks again for the excellent feedback! /cc @calmh @AudriusButkevicius @imsodin @bt90 @tomasz1986

[JanKanis]

@rasa Nice writeup. I would like to make some comments, but I can't edit the wiki due to not being a member of the Syncthing github organisation. How about you open up the wiki for public editing?

[rasa]

@rasa Nice writeup. I would like to make some comments, but I can't edit the wiki due to not being a member of the Syncthing github organisation. How about you open up the wiki for public editing?

@JanKanis I don't know if I have the power to do that, or if the gods would look kind on me if I did.

So I moved it to #9539 instead. If you'd rather I move it to a Google Doc, so you can feedback inline, lemme know. Thanks again for your help flushing this out.