autour_de_minuit/blender-asset-tracer
8
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 4 Wiki Activity
95 Commits 1 Branch 41 Tags
38978537bcb6fc0652c5f83642343419c67ddf23
Commit Graph
95 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Sybren A. Stüvel 38978537bc Removed some unused code + different line wrap style 2018-03-09 13:04:26 +01:00
Sybren A. Stüvel 0778b3a6ec Defined file transferer abstract base class 
This will allow for different file transfer classes uploading to different
services.
 2018-03-09 13:04:14 +01:00
Sybren A. Stüvel ac3f6142b3 Move rewritten blend files instead of copying them 
This will cause a nice speedup if the temporary directory happens to be on
the same file system as the target directory for the BAT Pack. This can be
influenced by setting an environment variable; see the documentation for
[tempfile.gettempdir](https://docs.python.org/3/library/tempfile.html#tempfile.gettempdir)
 2018-03-09 12:47:50 +01:00
Sybren A. Stüvel 824ca4bcb7 Rewrite blend files in temporary directory, before copying 
This allows us to later support uploading to a non-local filesystem, such
as Amazon S3 cloud storage, after path rewriting is complete.
 2018-03-09 12:33:15 +01:00
Sybren A. Stüvel 384f1d5107 Compare float with assertAlmostEqual 2018-03-09 11:20:38 +01:00
Sybren A. Stüvel fdbbc3a20d Static type checking with mypy 
This does introduce some not-so-nice things, like having to annotate each
`__init__` function with `-> None`. However, the benefits of having static
type checking in a complex bit of software like BAT outweigh the downsides.
 2018-03-09 11:09:47 +01:00
Sybren A. Stüvel 632d01334c Renamed module tracer → trace 
This is consistent with the other module names.
 2018-03-09 10:02:44 +01:00
Sybren A. Stüvel e8f41f2735 Store rewrites in AssetAction 
This is in preparation of interchanging copying and rewriting.
 2018-03-09 10:00:46 +01:00
Sybren A. Stüvel d984870155 Don't repeat superclass name 2018-03-09 09:59:54 +01:00
Sybren A. Stüvel d10264695a Tweaked logging 2018-03-08 15:30:31 +01:00
Sybren A. Stüvel e03e0fb4d7 Reduced some log levels 2018-03-08 14:53:25 +01:00
Sybren A. Stüvel 0dab872289 Truncate UTF-8 in a way that produces valid UTF-8. 2018-03-08 14:47:41 +01:00
Sybren A. Stüvel 898f79a951 Removed unused imports 2018-03-08 14:35:35 +01:00
Sybren A. Stüvel c80bb8232e Fix bad reference 2018-03-08 14:27:17 +01:00
Sybren A. Stüvel bfc1421df9 Updated README 2018-03-08 14:24:16 +01:00
Sybren A. Stüvel be747ffa6e Ignore build & dist dirs 2018-03-08 14:19:29 +01:00
Sybren A. Stüvel e9ee5b69ec Disallow BlendPath(str) 2018-03-08 14:18:07 +01:00
Sybren A. Stüvel f2f824ad85 Speed up BlendFileBlock instantiation with __slots__ 
Due to the huge number of BlendFileBlock objects created for packing a
production-size blend file, using slots here actually makes the
dependency tracer significantly (p<0.001) faster. In my test case the
speed improvement was 16% for a 'bam list' command.
 2018-03-08 13:59:28 +01:00
Sybren A. Stüvel 88f23a4097 Also test bat pack --noop 2018-03-08 13:02:18 +01:00
Sybren A. Stüvel b6527ec1ee Moved file copy queue to separate module. 
Also transformed into a threading.Thread subclass, rather than using a
function + module-global state variables.
 2018-03-08 13:00:56 +01:00
Sybren A. Stüvel 0df94c74cd Also test Packer.execute() 2018-03-08 12:41:19 +01:00
Sybren A. Stüvel 08f538d8c8 File I/O improvements 
- Tracer now iterates over blocks in disk order.
- Packer copies files per directory, in a separate thread.
- Packer only copies files if they don't exist yet.
- Packer also copies file permissions.
 2018-03-08 12:33:15 +01:00
Sybren A. Stüvel b5418e1f5f Removed Packer._packed_paths 
The info in that dict is also available in Packer._actions[...].new_path
 2018-03-08 11:15:38 +01:00
Sybren A. Stüvel 8fc1072144 Formatting 2018-03-08 11:08:57 +01:00
Sybren A. Stüvel 9af913d69a Removed find_block_from_address() in favour of dereference_pointer() 2018-03-08 11:08:36 +01:00
Sybren A. Stüvel 9c9323b170 Log instead of print 2018-03-08 11:00:09 +01:00
Sybren A. Stüvel 01a9392fc1 Added unit test for Packer.strategise() 2018-03-08 11:00:02 +01:00
Sybren A. Stüvel ccdccd69cf Speed up rewriting paths by reusing cached DNA structs & blocks 2018-03-08 10:59:45 +01:00
Sybren A. Stüvel 595f8cb0a6 Log duration of CLI commands 2018-03-07 17:14:44 +01:00
Sybren A. Stüvel cc20b0bfd5 Sort queue of blocks to visit by blend file and on-disk order 
This gives a small speedup to dependency tracing.
 2018-03-07 17:14:35 +01:00
Sybren A. Stüvel 09a0866c14 Cache open blend files 
This simplifies blend file handling, ensuring that blend files are only
opened once. Otherwise it would be harder to handle things like dependency
diamonds (libraries that are referenced via multiple paths through multiple
other libraries).
 2018-03-07 17:13:47 +01:00
Sybren A. Stüvel c5a02423cb Added --profile CLI flag for running the Python profiler 2018-03-07 15:50:04 +01:00
Sybren A. Stüvel e2123f8090 Path rewriting for sequences seems to be working too. 2018-03-06 16:41:16 +01:00
Sybren A. Stüvel 71dd5bc11b Path rewriting when packing. 
Doesn't work with sequences, nor with split dirname/basename fields.
 2018-03-06 16:06:36 +01:00
Sybren A. Stüvel 433ad8f16a Moved packer code into separate class, in preparation for smarter behaviour 2018-03-06 11:58:38 +01:00
Sybren A. Stüvel 92f91b42cf Added test case for Windows-style paths 2018-03-06 11:45:35 +01:00
Sybren A. Stüvel 39474548d2 Pack: Made project path explicit 2018-03-06 11:41:28 +01:00
Sybren A. Stüvel 6d1cae1225 Tracer: don't warn about missing expanders for LI and DATA blocks 2018-03-06 11:40:56 +01:00
Sybren A. Stüvel 9b2254a302 Included radon in requirements 
This is mostly used to keep an eye on cyclomatic complexity using
`radon cc blender_asset_tracer -s --min C`
 2018-03-02 18:04:55 +01:00
Sybren A. Stüvel 5c8e074c68 Implemented simple 'bat pack' CLI command 
No path rewriting yet, just simply copying the dependencies. They are
copied to the target directory + the path relative to the current working
directory. For example:

bat pack scenes/05-barber/05_02_C-agent_focused/05_02_C.lighting.blend /tmp/packer

will write to
/tmp/packer/scenes/05-barber/05_02_C-agent_focused/05_02_C.lighting.blend
and will copy ./path/to/file → /tmp/packer/path/to/file
 2018-03-02 17:59:04 +01:00
Sybren A. Stüvel f2694df602 Don't warn about missing readers in blend2asserts 
It's still logged at debug level, but it's aimed at developers and not
artists/end users.
 2018-03-02 15:45:26 +01:00
Sybren A. Stüvel 8009ff1e47 Added block expansion 
The expansion process follows pointers and library links to construct
the full set of actually-used data blocks. This set consists of all data
blocks in the initial blend file, and all *actually linked-to* data
blocks in linked blend files.

I've also removed non-recursive dependency listing.
 2018-03-02 15:44:07 +01:00
Sybren A. Stüvel 677d388a15 Convert BlendPath to pathlib.Path using local filesystem encoding 
UTF-8 is tried first, and if that fails the local filesystem encoding is
used.
 2018-03-02 13:42:49 +01:00
Sybren A. Stüvel 8bb130d336 Moved cdefs one module up 2018-03-02 13:36:08 +01:00
Sybren A. Stüvel 7167d51730 Added BlendFileBlock.id_name property 
Evaluated only once, so safe to call multiple times without producing
excessive disk I/O. Returns None instead of raising KeyError when there is
no (b'id', b'name') field.
 2018-03-02 11:26:02 +01:00
Sybren A. Stüvel 2bb9cffa49 Renamed block_walkers to blocks2assets 2018-03-02 10:06:41 +01:00
Sybren A. Stüvel f46e761f09 Removed base_index parameter in favour of explicit array handling 
The `base_index` parameter is confusing and only works in a limited number
of cases. Having explicit functions to deal with those cases is preferred.
 2018-03-01 12:50:22 +01:00
Sybren A. Stüvel 2c0cc6003d Updated block_walkers docstring 2018-03-01 10:35:51 +01:00
Sybren A. Stüvel d93516d34f Better naming of functions in block_walkers.py 2018-03-01 10:31:41 +01:00
Sybren A. Stüvel 204626b671 Fix issue when library blend file does not exist 2018-02-28 17:23:43 +01:00
Sybren A. Stüvel 6252b041d5 Implemented JSON dependency logging output 2018-02-28 17:18:18 +01:00
Sybren A. Stüvel a7706f21b6 Nicer CLI listing of dependencies 2018-02-28 17:02:34 +01:00
Sybren A. Stüvel 2af8d94cb9 Perform recursion after handling all blocks of current file 
This way file access isn't interleaved and all dependencies of one file are
reported before moving to the next.
 2018-02-28 17:02:19 +01:00
Sybren A. Stüvel 55302004ee Test BlockUsage.files() 2018-02-28 16:51:27 +01:00
Sybren A. Stüvel ca964181fe Beginning of CLI interface + protection against infinite recursion. 2018-02-28 16:47:13 +01:00
Sybren A. Stüvel 8ae400acbe Support particle simulation point cache data 
Also see https://developer.blender.org/T53562
 2018-02-28 14:11:48 +01:00
Sybren A. Stüvel 86af05e823 Added recursion into library blend files. 2018-02-28 12:42:27 +01:00
Sybren A. Stüvel cb5eff2dcb Some docstrings 2018-02-28 11:56:41 +01:00
Sybren A. Stüvel 7541922b74 Also report library blocks. 
Those are needed for recursive tracing of dependencies.
 2018-02-28 11:55:10 +01:00
Sybren A. Stüvel 0d7bbb34f0 Reduced cyclomatic complexity of BlockUsage.__init__() 2018-02-28 11:17:27 +01:00
Sybren A. Stüvel a5bc52523d Some more comments 2018-02-28 11:16:58 +01:00
Sybren A. Stüvel 65b690e998 Added block walker implementations + tests 2018-02-27 17:30:28 +01:00
Sybren A. Stüvel 6b9c0a3f95 Support slash notation for BlendPath 
This mimicks the slash notation of pathlib.Path
 2018-02-27 17:29:44 +01:00
Sybren A. Stüvel 15cd74cda4 Start of dependency tracer work. 2018-02-26 18:16:15 +01:00
Sybren A. Stüvel 59c0b6df4c Allow block.get() to return the dna.Field 
This is needed by the upcoming dependency tracer.
 2018-02-26 18:15:14 +01:00
Sybren A. Stüvel a56e985cdc Unify behaviour of block[path] and block.get(path) 2018-02-26 18:14:23 +01:00
Sybren A. Stüvel 127f357e49 Marked set() as ported 2018-02-23 15:54:13 +01:00
Sybren A. Stüvel bce5f4b670 Ported get_file_offset() and renamed to abs_offset() 2018-02-23 15:53:26 +01:00
Sybren A. Stüvel aed1827ff7 Moved pad_up_4() to inner function as it's only used there. 2018-02-23 15:44:57 +01:00
Sybren A. Stüvel 55075c09b6 Ported get_data_hash() and renamed to hash() 2018-02-23 15:42:48 +01:00
Sybren A. Stüvel db14cc5074 Test segfault 2018-02-23 14:46:47 +01:00
Sybren A. Stüvel 49d0f41b40 Test using BlendFile as context 2018-02-23 14:26:57 +01:00
Sybren A. Stüvel 9750a91571 Fixed file leak when opening corrupted file. 2018-02-23 14:26:57 +01:00
Sybren A. Stüvel eb8281ea82 Fix leaking file object when loading non-blend file 2018-02-23 14:26:57 +01:00
Sybren A. Stüvel 8e0b135eb3 Fixed recompressing after modification 2018-02-23 14:26:57 +01:00
Sybren A. Stüvel 67751d9a3e Fixed loading compressed files 2018-02-23 14:26:57 +01:00
Sybren A. Stüvel b42a090f32 Support for modifying string/bytes fields. 2018-02-23 14:26:57 +01:00
Sybren A. Stüvel efc60f437b Improved BFB.set() a bit, not yet tested at all. 2018-02-23 14:26:57 +01:00
Sybren A. Stüvel 934a8e210e Clarified type refinement, simplified API, custom exception for get_pointer 
Type refinement is now only done with BlendFileBlock.refine_type(), and
no longer with sdna_index_refine parameters to various functions. This
simplifies the API at the expense of having to call two simple functions
instead of one more complex one.
 2018-02-23 14:26:57 +01:00
Sybren A. Stüvel 9c8766a28c str → repr 2018-02-23 12:24:50 +01:00
Sybren A. Stüvel 42413c4d99 Added some TODOs 2018-02-23 12:24:50 +01:00
Sybren A. Stüvel 74ef3877c6 Renamed internal func and removed an iteration over self.blocks 2018-02-23 12:24:50 +01:00
Sybren A. Stüvel 8d32ab513f Recursively add offsets of Struct.field_from_path() 2018-02-23 12:24:50 +01:00
Sybren A. Stüvel 480b82337e Some simplifications 2018-02-22 17:44:50 +01:00
Sybren A. Stüvel 968068d21d Better exception message 2018-02-22 17:44:50 +01:00
Sybren A. Stüvel 63c7e4fcef Ported iteration over properties 2018-02-22 17:44:50 +01:00
Sybren A. Stüvel 6175f6c592 Updated README 2018-02-22 16:50:34 +01:00
Sybren A. Stüvel 7165d121bd Ported BlendFileBlock.get_recursive_iter() 
Also simplified get_file_offset(), because its optional parameters are
never used anyway.
 2018-02-22 16:45:16 +01:00
Sybren A. Stüvel 87300df6a3 Fixed issue reading array items and renamed some stuff 2018-02-22 16:14:10 +01:00
Sybren A. Stüvel 81dd2c9721 Ignore blend file backups 2018-02-22 15:52:27 +01:00
Sybren A. Stüvel 9c7791f762 Implemented/fixed/ported loading of blend files and getting blocks & props 2018-02-22 15:52:10 +01:00
Sybren A. Stüvel e66b1edaf4 Implemented + tested dna.Struct.field_get 2018-02-22 14:50:24 +01:00
Sybren A. Stüvel 03bf2ac69f Added unit testing with py.test 2018-02-22 14:50:24 +01:00
Sybren A. Stüvel dcda85c3e6 Ported dna.Name, dna.Field, and started with dna.Struct 2018-02-22 14:50:24 +01:00
Sybren A. Stüvel 0532634d13 Initial check-in, not yet working and has old, unported BAM code 2018-02-22 14:50:10 +01:00