9074421ec3
The Shaman server is a file storage system that identifies files by SHA256sum and file length. BAT can send packs there by only uploading changed/new files. The BAT pack is reproduced at the Shaman server's checkout directory by creating symlinks to the files in its file storage. Retrying sending files: When we can defer uploading a file (that is, when we have other files to upload as well, and we could send the current file at a later moment) we send an `X-Shaman-Can-Defer-Upload: true` header in the file upload request. In that case, when someone else is already uploading that file, a `208 Already Reported` response is sent and the connection is closed. Python's Requests library unfortunately won't give us that response if we're still streaming the request, and raise a ConnectionError exception instead. This exception can mean two things: - If the `X-Shaman-Can-Defer-Upload: true` header was sent: someone else is currently uploading that file, so defer it. - If that header was not sent: that file is already completely uploaded and does not need to be uploaded again. Instead of retrying each failed file, after a few failures we now just resend the definition file to get a new list of files to upload, then send those. This should considerably reduce the number of HTTP calls when multiple clients are uploading the same set of files.
Blender Asset Tracer BAT🦇
Script to manage assets with Blender.
Blender Asset Tracer, a.k.a. BAT🦇, is the replacement of BAM and blender-file
Development is driven by choices explained in T54125.
Uploading to S3-compatible storage
BAT Pack supports uploading to S3-compatible storage. This requires a credentials file in
~/.aws/credentials. Replace the all-capital words to suit your situation.
[ENDPOINT]
aws_access_key_id = YOUR_ACCESS_KEY_ID
aws_secret_access_key = YOUR_SECRET
You can then send a BAT Pack to the storage using a target s3:/ENDPOINT/bucketname/path-in-bucket,
for example:
bat pack my_blendfile.blend s3:/storage.service.cloud/jobs/awesome_work
This will upload the blend file and its dependencies to awesome_work/my_blendfile.blend in
the jobs bucket.
Paths
There are two object types used to represent file paths. Those are strictly separated.
bpathlib.BlendPathrepresents a path as stored in a blend file. It consists of bytes, and is blendfile-relative when it starts with//. It can represent any path from any OS Blender supports, and as such should be used carefully.pathlib.Pathrepresents an actual path, possibly on the local filesystem of the computer running BAT. Any filesystem operation (such as checking whether it exists) must be done using apathlib.Path.
When it is necessary to interpret a bpathlib.BlendPath as a real path instead of a sequence of
bytes, BAT first attempts to decode it as UTF-8. If that fails, the local filesystem encoding is
used. The latter is also no guarantee of correctness, though.
Type checking
The code statically type-checked with mypy.
Mypy likes to see the return type of __init__ methods explicitly declared as None. Until issue
#604 is resolved, we just do this in our code too.