Conversation
Before this change the Swift backend did not implement the metadata interfaces, so --metadata had no effect: user metadata stored in X-Object-Meta-* headers was neither read nor written, and it was lost when copying or syncing between Swift and other remotes. After this change the Swift backend supports reading and writing object metadata: - Object.Metadata exposes Swift user metadata plus the standard system metadata (Cache-Control, Content-Disposition, Content-Encoding, Content-Language, Content-Type and mtime). - Object.Update and Object.Copy pass the metadata through to the upload and to the server side copy, including the large object manifest path, so no extra requests are needed for a normal copy. - Server side copies send X-Fresh-Metadata so the metadata is replaced rather than merged with the source's, and --header-upload is applied to copies as well as uploads. - MetadataInfo documents the system metadata keys and the backend now advertises the ReadMetadata, WriteMetadata and UserMetadata features. content-encoding is documented as read only: rclone's HTTP transport transparently decompresses a response carrying Content-Encoding: gzip, so writing that header would create objects rclone could not read back. For chunked uploads the metadata is only applied to the manifest object, not to the segments, which are internal to the large object. Unit tests have been added, and the documentation and the backend feature list have been updated.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this change do?
Adds reading and writing of object metadata to the Swift backend, so that
--metadataworks for Swift.Object.Metadatareturns the Swift user metadata (X-Object-Meta-*) plus the standard system metadata (cache-control,content-disposition,content-language,content-type,mtime).Object.UpdateandObject.Copyapply the metadata on upload and on server side copy, including the large object manifest path, so no extra requests are needed for a normal copy.X-Fresh-Metadataso the metadata is replaced rather than merged with the source's, and--header-uploadis applied to copies as well as uploads.content-encodingis documented as read only: rclone's HTTP transport transparently decompresses a response carryingContent-Encoding: gzip, so writing that header would create objects rclone could not read back.ReadMetadata,WriteMetadataandUserMetadata, andMetadataInfodocuments the system metadata keys.Swift's metadata column in the overview table changes from
-toRWU. This is done indocs/data/backends/swift.yaml, which generates the table.Linked issue
Fixes #10002
This supersedes #8511, which only added read support and no longer applies cleanly. In that PR @albertony asked: "Swift could probably also support writing metadata (again similar to S3)?" - this change implements that.
For new or changed backends
go run ./fstest/test_all -backends swiftpasses:This covers
TestSwiftAIO:andTestSwiftAIOsegments:(go test,vfs,operations±fast-list,sync±fast-list,bisync).Note for anyone reproducing this on Apple Silicon: the
openstackswift/saioimage has no arm64 manifest, so it needsDOCKER_DEFAULT_PLATFORM=linux/amd64.Checklist
test_allpasses for this backend and if submitting a new backend can provide a test account for the integration tester - see CONTRIBUTING.md.