The Shaman server is a file storage system that identifies files by
SHA256sum and file length. BAT can send packs there by only uploading
changed/new files. The BAT pack is reproduced at the Shaman server's
checkout directory by creating symlinks to the files in its file
storage.
Retrying sending files:
When we can defer uploading a file (that is, when we have other files to
upload as well, and we could send the current file at a later moment) we
send an `X-Shaman-Can-Defer-Upload: true` header in the file upload
request. In that case, when someone else is already uploading that file,
a `208 Already Reported` response is sent and the connection is closed.
Python's Requests library unfortunately won't give us that response if
we're still streaming the request, and raise a ConnectionError exception
instead. This exception can mean two things:
- If the `X-Shaman-Can-Defer-Upload: true` header was sent: someone else
is currently uploading that file, so defer it.
- If that header was not sent: that file is already completely uploaded
and does not need to be uploaded again.
Instead of retrying each failed file, after a few failures we now just
resend the definition file to get a new list of files to upload, then
send those. This should considerably reduce the number of HTTP calls
when multiple clients are uploading the same set of files.
This is the same version as 0.99, just released as 1.0 to note the
maturity of the code (it's been used successfully for months at the Blender
Animation Studio).
The target path is just read as string from the CLI now, to allow more
complex targets (such as URLs) that don't directly map to a path.
The Packer subclass now handles the conversion from that string to a
`pathlib.PurePath`, and specific subclasses & transfer classes can convert
those to a `pathlib.Path` to perform actual filesystem operations when
necessary.
Not sure when/how that happens, but it happened on a lighting file of the
Spring project, when linking in a nested collection of which parts where
also linked in from other blend files.
Windows fails with an OSError when `somepath.resolve()` is called and
`somepath` contains a glob pattern. As a workaround, we now `resolve()`
the parent directory, and put the filename at its end. This only works
when the glob pattern is in the filename, which is the case for BAT-
generated globs.
When a path starts with b'//' it will not be converted to a PurePath, as
the handling of such filenames is platform dependent (Windows handles those
weirdly, like appending a slash to any path).
This makes BAT skip assets that are referred to with an absolute path.
It is assumed that the receiver of the BAT pack can access those assets
at the same path.
For regular file transfers (so to a directory, not to a ZIP file or S3
storage), use multi-threaded transfer when compressing. Compressing is
CPU-bound, so using multiple threads speeds things up considerably
(packing a Spring lighting file went from 6min30 single-threaded to
2min13 multi-threaded on my machine).
Currently pointcaches on disk for cloth sims were not traced. This patch allows the tracing of pointcaches for cloth sims.
Reviewers: sybren, fsiddi
Reviewed By: sybren
Differential Revision: https://developer.blender.org/D3910
