Setting up an Amazon S3-Compatible Object Store¶
Installation and configuration of the TeamDrive S3 Daemon
s3d is only
required if TSHS (see TeamDrive Scalable Hosting Storage for details) is not enabled and you have an
Amazon S3 compatible object store you wish to use as a secondary storage tier.
Currently, Amazon S3, OpenStack Swift and the Ceph Object Gateway (version 0.8 or higher, using the S3-compatible API) are supported.
s3d is a process that runs in the background and provides secondary
storage by transferring files to the object store. It does this by monitoring
the hosted data directory structure and transferring files to the object store
when a file reaches an age as specified in the service’s configuration. The
configuration settings also allows you to specify what files are eligible to
be transferred via the use of pattern matching.
Because all object store requests will be signed using the current timestamp,
it’s essential that the system time is accurate when running
Make sure that the NTP service is installed and running. See the chapters
about NTP configuration in the installation guides for details.
The configuration of
s3d is performed by changing the relevant
configuration settings using the Host Server Administration Console.
Log into your Host Servers Administration Console at
https://hostserver.yourdomain.com/admin/ and click on Settings.
The S3 daemon Settings all begin with
The following information is needed by
s3d to connect to the object
- This setting specifies the type of S3 storage. Valid options are: Amazon or OpenStack.
- Your object store’s domain name, e.g.
- The name of the Bucket in the object store that will contain the Space data. The bucket must already exist.
If you are setting up multiple TeamDrive Hosting servers it is important that they do not use the same Bucket. Doing so can result in data loss.
- Your S3 access (public) key, used to access the specified bucket.
- Your S3 secret (private) key used to access the specified bucket.
- Set this to
Trueto enable the synchronization of data stored by the Host Server (Space data) to the specified bucket on an S3 compatible object store. Note that the synchronization won’t start until the
s3dservice has been started (see below).
- These options control the way S3 is accessed, for example the number of
parallel threads during upload, whether to use multipart upload, etc. The
options may also contain
- When S3 redirect is enabled, the Host Server will redirect the Client to a
download directly from the object store, when appropriate. This helps to
offload traffic from the Host Server to the object store. If set to
False, the Host Server fetches the requested object from the object store and serves it to the Client directly.
Starting and Stopping the
You can use the
/etc/init.d/s3d init script to start and stop s3d. The
S3SyncActive needs to be set to
s3d will abort with a corresponding error message.
Enabling the S3 Daemon means that any new data in existing Spaces and new Spaces will be transferred to the object store. Currently, there is no automatic way to return back to a pure file-based Host Server setup — data that was moved to S3 stays in S3. Disabling the S3 secondary storage tier would result in Clients no longer being able to access their Space data.
s3d with the command
service s3d start, check the log
/var/log/s3d.log for startup messages. You can use
status to check if
s3d is up and running.
s3d should be added to the processes to be started at boot time. To do
this execute the following commands as root:
[root@hostserver ~]# chkconfig s3d on
Optional configuration parameters¶
The following optional configuration options can be modified in the
S3Options configuration setting:
- The minimum block size that can be used for multi-part upload. The valid range of this parameter will be determined by the implementation of the object store being used.
- The maximum number of parts a file can be divided into for upload. The valid range of this parameter will be determined by the implementation of the object store being used.
- Files larger than this will not be transferred to S3. The default, 0, means all files are uploaded.
- The thread pool size to use for multi-part uploads. The default, 0, means single threaded.
- Amazon S3 uses the bucket name as part of the domain name
BucketAsSubdomain=1), while other S3-compatible object stores (e.g. OpenStack) include the bucket name as part of the path name after the domain (
BucketAsSubdomain=0). This option usually does not have to be set explicitly, as the
S3Brandsetting determines this value automatically.
OpenStack configuration parameters¶
S3Brand is set to
OpenStack then 2 additional parameters are
S3Options, to enable the generation of temporary URLs.
Temporary URLs give clients temporary direct access to objects in the object
store, helping to reduce network traffic on the Host Server. This requires the
S3EnableRedirect to be set to
- This is the path component of the OpenStack Authorization URL. For example
if the OpenStack Authorization URL is
OpenStackAuthPathwould be /v1/AUTH_a422b2-91f3-2f46-74b7-d7c9e8958f5d30.
- Your OpenStack temp URL key used to generate temporary URLs.
Your OpenStack administrator should be able to provide you with the OpenStack Authorization URL and the corresponding temp URL key.
The generation of new temp URL keys will invalidate older keys. It is important that once the temp URL key has been set for your TeamDrive Hosting server that no new keys are being generated.
Enabling Object Store Traffic Usage Processing¶
When S3 storage has been enabled, TeamDrive Clients access the data in the object store directly. The traffic required by these operations is recorded by reading the S3 access log files.
This means that setting
True changes the way Traffic
usage is calculated. When
False, the Hosting Service
is able to record all traffic usage in the
pspace database. When
True the S3 access logs myst be downloaded and parsed
to get the required information.
At present, only Amazon S3 supports the concept of providing access log
files via a dedicated log bucket. If your object store doesn’t support
providing access logs, leave the following four settings empty. The host
server will calculate the file size of each returned redirect URL to the
external object store as traffic for this space. This might be not exact,
because a client might send a download request for the same file again in
case that the download was not complete. The download will then start at
the offset where the former download stopped. A new redirect will not
result in a full file download in this case. In case your object store
offers access logs, the traffic usage processing expects the Amazon S3
access log format and the log files must contain the string
See Server Access Logging in the Amazon S3 documentation for details
on how to enable server access logging.
The calculation of traffic usage requires an external Perl-based tool named
aws to be installed. Install the aws tool as described on the aws tool
home page. The tools
s3delete need to be in the
PATH of the
In addition, the following configuration settings have to be defined:
- Name of the bucket that contains the S3 access logs. Note that if this setting is empty, the object store traffic will not be calculated. You must configure your object store to save the access logs to this bucket. Please refer to your object store documentation to determine how this is done.
- Local path in the filesystem to download above access logs for further
processing. By default, this path is set to
- If set to
True, the processed access logs will be moved to the folder defined in the next setting
- If logs need to be kept for own additional analysing, then the S3 log files will be moved to this directory once traffic usage Processing is complete.
By default, the task that calculates the S3 traffic runs every 10 minutes.