Shaman server support

The Shaman server is a file storage system that identifies files by
SHA256sum and file length. BAT can send packs there by only uploading
changed/new files. The BAT pack is reproduced at the Shaman server's
checkout directory by creating symlinks to the files in its file
storage.

Retrying sending files:

When we can defer uploading a file (that is, when we have other files to
upload as well, and we could send the current file at a later moment) we
send an `X-Shaman-Can-Defer-Upload: true` header in the file upload
request. In that case, when someone else is already uploading that file,
a `208 Already Reported` response is sent and the connection is closed.
Python's Requests library unfortunately won't give us that response if
we're still streaming the request, and raise a ConnectionError exception
instead. This exception can mean two things:

- If the `X-Shaman-Can-Defer-Upload: true` header was sent: someone else
  is currently uploading that file, so defer it.
- If that header was not sent: that file is already completely uploaded
  and does not need to be uploaded again.

Instead of retrying each failed file, after a few failures we now just
resend the definition file to get a new list of files to upload, then
send those. This should considerably reduce the number of HTTP calls
when multiple clients are uploading the same set of files.

blender_asset_tracer/pack/shaman/client.py

@@ -0,0 +1,121 @@
+# ***** BEGIN GPL LICENSE BLOCK *****
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License
+# as published by the Free Software Foundation; either version 2
+# of the License, or (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software Foundation,
+# Inc., 59 Temple Place - Suite 330, Boston, MA  02111-1307, USA.
+#
+# ***** END GPL LICENCE BLOCK *****
+#
+# (c) 2019, Blender Foundation - Sybren A. Stüvel
+
+import urllib.parse
+
+import requests.adapters
+
+
+class ShamanClient:
+    """Thin wrapper around a Requests session to perform Shaman requests."""
+
+    def __init__(self, auth_token: str, base_url: str):
+        self._auth_token = auth_token
+        self._base_url = base_url
+
+        http_adapter = requests.adapters.HTTPAdapter(max_retries=5)
+        self._session = requests.session()
+        self._session.mount('https://', http_adapter)
+        self._session.mount('http://', http_adapter)
+        self._session.headers['Authorization'] = 'Bearer ' + auth_token
+
+    def request(self, method: str, url: str, **kwargs) -> requests.Response:
+        full_url = urllib.parse.urljoin(self._base_url, url)
+        return self._session.request(method, full_url, **kwargs)
+
+    def get(self, url, **kwargs):
+        r"""Sends a GET request. Returns :class:`Response` object.
+
+        :param url: URL for the new :class:`Request` object.
+        :param kwargs: Optional arguments that ``request`` takes.
+        :rtype: requests.Response
+        """
+
+        kwargs.setdefault('allow_redirects', True)
+        return self.request('GET', url, **kwargs)
+
+    def options(self, url, **kwargs):
+        r"""Sends a OPTIONS request. Returns :class:`Response` object.
+
+        :param url: URL for the new :class:`Request` object.
+        :param kwargs: Optional arguments that ``request`` takes.
+        :rtype: requests.Response
+        """
+
+        kwargs.setdefault('allow_redirects', True)
+        return self.request('OPTIONS', url, **kwargs)
+
+    def head(self, url, **kwargs):
+        r"""Sends a HEAD request. Returns :class:`Response` object.
+
+        :param url: URL for the new :class:`Request` object.
+        :param kwargs: Optional arguments that ``request`` takes.
+        :rtype: requests.Response
+        """
+
+        kwargs.setdefault('allow_redirects', False)
+        return self.request('HEAD', url, **kwargs)
+
+    def post(self, url, data=None, json=None, **kwargs):
+        r"""Sends a POST request. Returns :class:`Response` object.
+
+        :param url: URL for the new :class:`Request` object.
+        :param data: (optional) Dictionary, list of tuples, bytes, or file-like
+            object to send in the body of the :class:`Request`.
+        :param json: (optional) json to send in the body of the :class:`Request`.
+        :param kwargs: Optional arguments that ``request`` takes.
+        :rtype: requests.Response
+        """
+
+        return self.request('POST', url, data=data, json=json, **kwargs)
+
+    def put(self, url, data=None, **kwargs):
+        r"""Sends a PUT request. Returns :class:`Response` object.
+
+        :param url: URL for the new :class:`Request` object.
+        :param data: (optional) Dictionary, list of tuples, bytes, or file-like
+            object to send in the body of the :class:`Request`.
+        :param kwargs: Optional arguments that ``request`` takes.
+        :rtype: requests.Response
+        """
+
+        return self.request('PUT', url, data=data, **kwargs)
+
+    def patch(self, url, data=None, **kwargs):
+        r"""Sends a PATCH request. Returns :class:`Response` object.
+
+        :param url: URL for the new :class:`Request` object.
+        :param data: (optional) Dictionary, list of tuples, bytes, or file-like
+            object to send in the body of the :class:`Request`.
+        :param kwargs: Optional arguments that ``request`` takes.
+        :rtype: requests.Response
+        """
+
+        return self.request('PATCH', url, data=data, **kwargs)
+
+    def delete(self, url, **kwargs):
+        r"""Sends a DELETE request. Returns :class:`Response` object.
+
+        :param url: URL for the new :class:`Request` object.
+        :param kwargs: Optional arguments that ``request`` takes.
+        :rtype: requests.Response
+        """
+
+        return self.request('DELETE', url, **kwargs)