Improve documentation, and verify consistency for xattr (extended attributes) support

[thaJeztah]

Description

Posting this as a reminder / tracking ticket for issues around xattr support and consistency.

Relates to;

• VFS storage driver drops all file capabilities when creating a container #42655
• Fix file capabilites droping in Dockerfile #42934
• [20.10 backport] Fix file capabilities dropping in Dockerfile #43876
• setcap in dockerfile not persistent in parent dockerfile #43856
• Support retaining file capabilities in COPY/ADD #20435

Let me post some details from a conversation I had with @corhere to check if the patch linked above was "safe" to backport;

I haven’t tested it, but I suspect the overlay2 driver preserves all xattrs across layer Create simply because it doesn’t get in the way of overlayfs. The vfs and overlay drivers have to do a full tree-copy in userspace to Create the new layer. They both use the same copy.DirCopy function, but the vfs driver opts out of copying xattrs while the overlay driver opts into (selective) xattr copying. One operation, three drivers, three different user-visible behaviours

The only xattr copied into layer tarballs is security.capability, converted from VFS_CAP_REVISION_3 to VFS_CAP_REVISION_2. No attempt is made to record any other xattrs when diffing a layer AFAICT

On the other hand, the code to apply a layer tarball will happily apply all the xattrs in the file’s tar header
Having the vfs driver discard any trusted.overlay.opaque xattrs it encounters is a reasonable-enough thing to do since it is meaningless to anything other than overlayfs. Unconditionally preserving security.capability seems like the right thing to do. In conclusion, I’m pro-backport.

w.r.t. ISTR the tarball filtering things is to keep the images portable.

Overall, it may be somewhat surprising that attributes can be set during docker run, but can't be used (or, more factually: aren't preserved) in docker build and/or when doing a docker commit.

I think we should;

• have a closer look at discrepancies between xattrs that are supported for each storage driver (or snapshotter), and at least document them
• also document what attributes are preserved in images for portability (and explain why), and which ones are not preserved (by design).
• w.r.t the above; also check if the OCI defines anything in this area (and if not; consider describing it)
• and finally; when doing a docker import from a tarball; if there's a discrepancy there, we should probably document as well.