198 Commits

Author SHA1 Message Date
Sybren A. Stüvel
d2e13750e9 Moved comment to correct location 2018-03-15 16:28:34 +01:00
Sybren A. Stüvel
6b51183095 Packer: store the output path of the blend file in the target directory
This allows the Flamenco Add-on to get the final path of the blend file.
2018-03-14 16:06:21 +01:00
Sybren A. Stüvel
73b264f6bc Allow relative project path 2018-03-14 16:06:21 +01:00
Sybren A. Stüvel
9ceb42af2f Moved some code into its own function 2018-03-14 16:06:21 +01:00
Sybren A. Stüvel
8340df129d BAT Pack can now exclude files based on glob patterns 2018-03-14 16:06:21 +01:00
Sybren A. Stüvel
0e9de7753c Allow Packer to be used as context manager + test for rewriting img seqs 2018-03-14 12:50:27 +01:00
Sybren A. Stüvel
c9664ca045 Report missing files in Packer.strategise() 2018-03-14 12:24:26 +01:00
Sybren A. Stüvel
575a0921c1 bat.pack.queued_copy → bat.pack.filesystem 2018-03-09 17:04:29 +01:00
Sybren A. Stüvel
ea832cd666 Added support for uploads to S3 by using the boto3 client. 2018-03-09 16:48:34 +01:00
Sybren A. Stüvel
15ca2f98ab Better handling of KeyboardInterrupt to abort file transfers 2018-03-09 16:30:46 +01:00
Sybren A. Stüvel
a2ca66a2f6 Nicer transfer queue iteration 2018-03-09 16:30:17 +01:00
Sybren A. Stüvel
ef5c46eba5 Some mypy & test tweaks 2018-03-09 16:29:30 +01:00
Sybren A. Stüvel
53a2ca4373 Only set the log level on our own logger.
Otherwise debug logging will be completely swamped with logs from other
libraries. Maybe that's nice too at some point, but that would need another
CLI arg.
2018-03-09 16:28:31 +01:00
Sybren A. Stüvel
1a37cb5ba1 Moved common code to FileTransferer class
Many implementations will want to use the same queueing mechanism.
2018-03-09 15:08:50 +01:00
Sybren A. Stüvel
3ae39104a9 Allow subclasses of Packer to specify a different file transferer class 2018-03-09 14:25:43 +01:00
Sybren A. Stüvel
cc06a191a1 Added more annotations for mypy
Function declarations that have no type annotations at all will be skipped
by mypy. Adding an explicit `-> None` tells mypy to run on those functions
too.
2018-03-09 14:23:43 +01:00
Sybren A. Stüvel
38978537bc Removed some unused code + different line wrap style 2018-03-09 13:04:26 +01:00
Sybren A. Stüvel
0778b3a6ec Defined file transferer abstract base class
This will allow for different file transfer classes uploading to different
services.
2018-03-09 13:04:14 +01:00
Sybren A. Stüvel
ac3f6142b3 Move rewritten blend files instead of copying them
This will cause a nice speedup if the temporary directory happens to be on
the same file system as the target directory for the BAT Pack. This can be
influenced by setting an environment variable; see the documentation for
[tempfile.gettempdir](https://docs.python.org/3/library/tempfile.html#tempfile.gettempdir)
2018-03-09 12:47:50 +01:00
Sybren A. Stüvel
824ca4bcb7 Rewrite blend files in temporary directory, before copying
This allows us to later support uploading to a non-local filesystem, such
as Amazon S3 cloud storage, after path rewriting is complete.
2018-03-09 12:33:15 +01:00
Sybren A. Stüvel
fdbbc3a20d Static type checking with mypy
This does introduce some not-so-nice things, like having to annotate each
`__init__` function with `-> None`. However, the benefits of having static
type checking in a complex bit of software like BAT outweigh the downsides.
2018-03-09 11:09:47 +01:00
Sybren A. Stüvel
632d01334c Renamed module tracer → trace
This is consistent with the other module names.
2018-03-09 10:02:44 +01:00
Sybren A. Stüvel
e8f41f2735 Store rewrites in AssetAction
This is in preparation of interchanging copying and rewriting.
2018-03-09 10:00:46 +01:00
Sybren A. Stüvel
d984870155 Don't repeat superclass name 2018-03-09 09:59:54 +01:00
Sybren A. Stüvel
d10264695a Tweaked logging 2018-03-08 15:30:31 +01:00
Sybren A. Stüvel
e03e0fb4d7 Reduced some log levels 2018-03-08 14:53:25 +01:00
Sybren A. Stüvel
0dab872289 Truncate UTF-8 in a way that produces valid UTF-8. 2018-03-08 14:47:41 +01:00
Sybren A. Stüvel
898f79a951 Removed unused imports 2018-03-08 14:35:35 +01:00
Sybren A. Stüvel
c80bb8232e Fix bad reference 2018-03-08 14:27:17 +01:00
Sybren A. Stüvel
e9ee5b69ec Disallow BlendPath(str) 2018-03-08 14:18:07 +01:00
Sybren A. Stüvel
f2f824ad85 Speed up BlendFileBlock instantiation with __slots__
Due to the huge number of BlendFileBlock objects created for packing a
production-size blend file, using slots here actually makes the
dependency tracer significantly (p<0.001) faster. In my test case the
speed improvement was 16% for a 'bam list' command.
2018-03-08 13:59:28 +01:00
Sybren A. Stüvel
b6527ec1ee Moved file copy queue to separate module.
Also transformed into a threading.Thread subclass, rather than using a
function + module-global state variables.
2018-03-08 13:00:56 +01:00
Sybren A. Stüvel
08f538d8c8 File I/O improvements
- Tracer now iterates over blocks in disk order.
- Packer copies files per directory, in a separate thread.
- Packer only copies files if they don't exist yet.
- Packer also copies file permissions.
2018-03-08 12:33:15 +01:00
Sybren A. Stüvel
b5418e1f5f Removed Packer._packed_paths
The info in that dict is also available in Packer._actions[...].new_path
2018-03-08 11:15:38 +01:00
Sybren A. Stüvel
8fc1072144 Formatting 2018-03-08 11:08:57 +01:00
Sybren A. Stüvel
9af913d69a Removed find_block_from_address() in favour of dereference_pointer() 2018-03-08 11:08:36 +01:00
Sybren A. Stüvel
9c9323b170 Log instead of print 2018-03-08 11:00:09 +01:00
Sybren A. Stüvel
01a9392fc1 Added unit test for Packer.strategise() 2018-03-08 11:00:02 +01:00
Sybren A. Stüvel
ccdccd69cf Speed up rewriting paths by reusing cached DNA structs & blocks 2018-03-08 10:59:45 +01:00
Sybren A. Stüvel
595f8cb0a6 Log duration of CLI commands 2018-03-07 17:14:44 +01:00
Sybren A. Stüvel
cc20b0bfd5 Sort queue of blocks to visit by blend file and on-disk order
This gives a small speedup to dependency tracing.
2018-03-07 17:14:35 +01:00
Sybren A. Stüvel
09a0866c14 Cache open blend files
This simplifies blend file handling, ensuring that blend files are only
opened once. Otherwise it would be harder to handle things like dependency
diamonds (libraries that are referenced via multiple paths through multiple
other libraries).
2018-03-07 17:13:47 +01:00
Sybren A. Stüvel
c5a02423cb Added --profile CLI flag for running the Python profiler 2018-03-07 15:50:04 +01:00
Sybren A. Stüvel
e2123f8090 Path rewriting for sequences seems to be working too. 2018-03-06 16:41:16 +01:00
Sybren A. Stüvel
71dd5bc11b Path rewriting when packing.
Doesn't work with sequences, nor with split dirname/basename fields.
2018-03-06 16:06:36 +01:00
Sybren A. Stüvel
433ad8f16a Moved packer code into separate class, in preparation for smarter behaviour 2018-03-06 11:58:38 +01:00
Sybren A. Stüvel
39474548d2 Pack: Made project path explicit 2018-03-06 11:41:28 +01:00
Sybren A. Stüvel
6d1cae1225 Tracer: don't warn about missing expanders for LI and DATA blocks 2018-03-06 11:40:56 +01:00
Sybren A. Stüvel
5c8e074c68 Implemented simple 'bat pack' CLI command
No path rewriting yet, just simply copying the dependencies. They are
copied to the target directory + the path relative to the current working
directory. For example:

bat pack scenes/05-barber/05_02_C-agent_focused/05_02_C.lighting.blend /tmp/packer

will write to
/tmp/packer/scenes/05-barber/05_02_C-agent_focused/05_02_C.lighting.blend
and will copy ./path/to/file → /tmp/packer/path/to/file
2018-03-02 17:59:04 +01:00
Sybren A. Stüvel
f2694df602 Don't warn about missing readers in blend2asserts
It's still logged at debug level, but it's aimed at developers and not
artists/end users.
2018-03-02 15:45:26 +01:00