Release Notes - Version 3.x¶
Change Log - Version 3.7¶
3.7.11 (2021-02-18)¶
Fixed a bug in the cleanup code of s3d, that is used after space rollback. A directory object returned from S3 was not correctly handled.
The Apache module (mod_pspace) will now retry S3 startup (object store access) if this fails on startup of the module (HOSTSERVER-781).
Updated jquery.js version to 3.5.1 (HOSTSERVER-788).
Added download logging (HOSTSERVER-787). This functionality can be used to limit the number of downloads of a file, from a certain client device in a certain amount of time. Published files are not effected by this limit.
If the download quota is exceeded, the Host Server returns a HTTP_TOO_MANY_REQUESTS (429) error.
New settings (
EnableDownloadLogging
,DownloadLimit
,DownloadLogGrouping
,DownloadLogRetention
andDownloadRatePeriod
) allow this to be configured globally for the Host Server.Download logging can also be enabled at the Depot level by setting a “download limit”. This value overriders the global
DownloadLimit
setting. See Download Logging for details.In the Admin Console the download log can be viewed and queried from the “Log Files” menu item.
3.7.10 (2020-05-15)¶
A number of changes have been made to prevent the publishing of a web-site using the publish file functionality (HOSTSERVER-770). This includes:
- Added the
EnableDirectLink
setting which makes it possible to disable the direct link feature (dl=1). - Added the
ForceDownloadList
setting: This a list of content types and file endings that force a download in place of displaying the file in the browser. - Added the
PublicRewritesInstalled
setting which indicates that certain re-write rules have been added to the Host Server, which allow the TeamDrive client to generate public URLs that vary in the first component depending on the space (see PublicRewritesInstalled).
3.7.9 (2019-07-19)¶
The S3 server name (setting
S3Server
) may now be specified as a URL, not just a domain. This allows you to set the protocol to HTTPS, and specify an alternative port (HOSTSERVER-767).If a protocol is specified in the URL, then the this overrides the value of the S3RedirectProtocol setting.
Fixed the setting S3RedirectProtocol, which was ignored by the Host Server apache module (mod_pspace).
3.7.8 (2019-03-29)¶
- S3Daemon: Set CURLOPT_NOSIGNAL in order to prevent crash which occurs when libcurl recieves a signal due to a timeout in domain lookup
- Set yvva dependency to 1.4.6
3.7.7 (2019-02-26)¶
- A unique index has been added to the the Owner.UserName field where it was missing (HOSTSERVER-763).
- If
HttpsUsedByPublish
is set toTrue
, the Host Server will now return an error when trying to access a published file using HTTP (instead of HTTPS) (HOSTSERVER-762). - A error occurred when adding/removing a large number of users to/from a depot. This is due to field size limitations in the RepositoryChanges table. Excessively long user lists are now truncated, and the suffix: ”, ... and N others” is added (REGSERVER-1379).
- The “getdepotdata” API call was incorrectly returning the “ ” HTML entity
in the
<changelist>
details. - Fixed a bug when setting the owner of a repository, if the repository had no a history entry was not created.
- Added timeouts for all S3 operations. The connection timeout is set to 2 minutes, and the timeout for the entire S3 operation is set to 30 minutes (HOSTSERVER-758). This is to prevent the background task from hanging in the request to get the S3 logs.
- Returning data from encrypted files could hang in Tdp3File::send_file() if there was an error on the channel (HOSTSERVER-760).
3.7.6 (2018-10-22)¶
- Fixed a bug in the calculation of Space disk usage when correcting spaces that have a negative disk usage. The bug resulted in a overflow error being thrown by the “Check Spaces with Limit” auto task (HOSTSERVER-757).
- Fixed a bug when deleting an owner (user that has been deleted on the Registration Server): if the user was a user of a depot, then: either (1) the user was not removed or (2) a repository history entry was not inserted.
3.7.5 (2018-10-11)¶
Spaces marked as having a data retention period may not be deleted over the API (HOSTSERVER-751).
These spaces must either be deleted using the TeamDrive client, or on the Host Server Admin Console. Depots containing spaces with a data retention period are also subject to this restriction.
Deleting a depot on the Admin Console will now delete all spaces in the depot.
Undeleting all spaces and restoring all spaces belonging to a depot is now possible. When a depot contains deleted spaces the button “Undeleted Spaces” and “Spaces Restored” appear in the Admin Console on the depot page. Note that if the depot has been deleted, then you must undelete the depot first (HOSTSERVER-726).
An error will occur if you click “Spaces Restored”, and not all spaces in the depot have been copied back to an active volume on the host. In this case, the restore of some spaces may be complete while other remain deleted.
As before, undeleted and restore are possible at the space level, however, this will not be allowed if the repository of the space has been deleted.
NOTE! The Admin Console setting:
ShowDeletedObjects
must be set to true in order to see depots and spaces that have been deleted.Added
SpaceDeletionDelay
(Resource Management) setting which specifies the time between a space being deleted and it actually being removed from disk (HOSTSERVER-727). During this time the space can be undeleted.Added
AllowedLoginIPList
(Admin Console) setting which can be used to restrict login to the Admin Console to certain IP addresses (HOSTSERVER-723).HTML templates can now be customised by setting a header and a footer HTML “snippet” (HOSTSERVER-729) at the depot level.
Note that the placeholders
[[HEADER]]
and[[FOOTER]]
have been added to the relevant HTML templates for this purpose.The global settings:
DefaultTemplateFooter
andDefaultTemplateHeader
are used as default values if nothing is specified for a depot.HTML template can use conditional sections (). This have the following form:
[[IF:<placeholder>]]
...[[ENDIF:<placeholder>]]
and
[[IFNOT:<placeholder>]]
...[[ENDIF:<placeholder>]]
where
<placeholder>
may be any valid placeholder:HEADER
,FOOTER
,FILE-NAME
,ERROR-MESSAGE
,ERROR-CODE
andPUBLIC-URL
.The
IF
sections are displayed if the specified placeholder is not empty and non-zero (in, the case if ERROR-CODE`).IFNOT
sections are displayed if the placeholder value is empty or zero.Operations that append to log files now return the log offset of the position after the block written (HOSTSERVER-740).
The Host Server now supports at rest encryption of public files.
The new HTML template: “decryption-failed.html”, will be returned if the public URL does not contain a correct or valid decryption key.
The Host Server now supports “shorted URLs” for public files. A short URL may be requested before upload of a public file begins (HOSTSERVER-722).
A new HTML template has been added: “upload-incomplete.html”. This template is returned if upload of a public file has been started, but is not yet complete. This is necessary because, in the case of large files, the TeamDrive Client may make the public URL available before the upload is complete.
Note
For short URL public files to work correctly, you must remove the
action="..."
attribute from the<form>
tags, in the “enter-password.html” and “password-wrong.html” templates. The default templates have already been updated.Published files are now encrypted at rest. The key must provided in the URL on upload and download (HOSTSERVER-732).
3.7.4 (2018-07-17)¶
- Fixed crash in background task when a Registration Server was not available during synchronisation of owner data (HOSTSERVER-720).
- The “getdepotdata” API call now returns a
<flags>
tag which may include therestrict-access
flag value (see Retrieve Depot Information for details). - The “createdepot” API call no longer automatically creates a “contract number” for a depot which starts with “WEB#”.
- Fixed a bug that prevented the synchronisation of data with foreign Registration Server. The error in log was: “RROR -24903 (0): Authorization failed: device 99999 not found” (HOSTSERVER-725).
- When moving spaces from one depot to another, then disk usage and traffic was not always recalculated correctly (HOSTSERVER-731).
- Under certain circumstance published files that were part of a snapshot were not deleted after expiry, although access to the file was prevented (HOSTSERVER-733).
- Under some circumstance the Host Server set the “Traffic limit
exceeded” flag, even when the
EnforceTrafficLimit
setting was set toFalse
(HOSTSERVER-735). - The Host Server now records the name of the user that made changes to a depot. Previously this information was not always available as it was placed in the comments. This function requires the use of Registration Server version 4.0 or later (HOSTSERVER-736).
3.7.3 (2017-11-01)¶
Improved the reporting and logging of connection errors that may occur when the Host Server contacts the Registration Server.
During Host Server Setup it is now possible to specify a proxy to use in order to contact the Registration Server. The
NoProxyList
setting must be specified after setup, if required.Improved input checking on setup of the Host Server. The Registration Name may not contain in special characters. Domain Names may not contain any spaces, and must include at least one ‘.’ character (HOSTSERVER-715).
The Host Server will now prevent access to a Depot if all users are removed from the access list. Previously, Depots reverted to unrestricted access when the last user was removed from the access list.
The Depot users in the access list are now displayed in the Admin Console. Only the Depot owner and users in this list are allowed to create Spaces in the Depot. However, users not in the list are not prevented from using existing Spaces in the Depot.
Deleting a Depot in the Admin Console now works the same as deleting a Depot via the API: the Depot is simply marked as deleted (HOSTSERVER-712).
If the setting
ShowDeletedObjects
isFalse
, then Depots marked as deleted will not be visible in the Depot list. However, such Depots can be reached by clicking on the Depot link in a Space belonging to the Depot.Note that deleting a Depot currently just prevents new Spaces from being created in the Depot. Existing Spaces are still accessable.
Moved index on SpaceID, MetaType from MetaData to MetaDataOptions table. This index was previously created on the wrong table (HOSTSERVER-716).
Added support for “If-Modified-Since” header. If sent, and BLOB data has not been modified since the specified time, the server will now send a “304 Not Modified” result. This is in order to support caching proxoies (HOSTSERVER-709).
Added
NonCachingProxies
setting. This is a list of the host names or pseudonyms of proxies that are downstream from the Host Server but do not cache any data (HOSTSERVER-711).Version 3.7.3 requires YVVA runtime version 1.4.4.
3.7.2 (2017-08-14)¶
- The TPD v3 call “restsnap” will now delete all meta data created for the Space after the last modify time of the snapshot (HOSTSERVER-708).
- Fixed a database deadlock in TDP v3 call “addmeta” (HOSTSERVER-707).
- Moved
EnableProxyCaching
to “Client Settings” (HOSTSERVER-706). - A space change history entry is now made when a Space is deleted (HOSTSERVER-705).
3.7.1 (2017-06-20)¶
This is the initial public release of version 3.7.
This version requires the YVVA runtime 1.4.0 or later.
Host Server Functionality¶
The Host Server supports Point-in-Time recovery. Using this functionality the TeamDrive Client is able to rollback a Space to a previous point in time. See details in chapter Snapshot Backup and Point-in-Time Recovery.
Added “Outgoing Connection” settings:
UseProxy
,ProxyHost
,NoProxyList
,ConnectionTimeout
andNetworkTimeout
(see Outgoing Connections) (HOSTSERVER-676).Added support for Read Notifications. Read Notifications are disabled by default for all Spaces. This feature must be explicitly enabled for a Space by the TeamDrive Client.
The setting
DefaultReadNotificationtMaxAge
determines the maximum age of read notifications, if this value has not been explicitly set at the Space level settings (HOSTSERVER-681).Fixed a bug which caused an error when moving a Space from one depot to another using the Admin Console (HOSTSERVER-680).
When accumulating traffic, the process now checks the access time to ensure that traffic is only accumulated for the current month. This fixes the problem that the Object Store log processing can generate traffic changes that occurred in the previous month (HOSTSERVER-702).
Administration Console¶
Note
Please clear the browser cache after the server update.
Snapshot relevant parameters can be set per Space. Changes are recorded in the change history of the Space.
Read Notification settings can be set per Space. Changes are recorded in the change history of the Space.
Added the list of background tasks (“Auto Tasks”) to the Admin Console. The list indicates when a task last ran and the result. Clicking on a task allows the user to Activate or Deactivate tasks (HOSTSERVER-697).
This page also shows the status of the Host Server services: td-hostserver, s3d and tshs (if active). The command used to determine the status is:
service td-hostserver/s3d/tshs status
If this system is not correctly installed an error will be displayed.
Change Log - Version 3.6¶
3.6.3 (2017-02-15)¶
- Admin Console: Fixed the select owner dialog on the “Space Depots: Details” page. Entering a name filter was not working (HOSTSERVER-664).
- Revised chapter Host Server Virtual Appliance with CentOS 7
3.6.2 (2017-01-24)¶
- Missing BLOBs are now logged a Trace level (HOSTSERVER-648).
- Added path and BLOB name to error log output of the v2 protocol (HOSTSERVER-649).
- Updating a Depot in the Hosting Server Admin Console could fail with the error: “Owner is empty” (HOSTSERVER-646).
- Fixed incorrect XML sent in the Reg Server search call used to retrieve owner/user details (HOSTSERVER-650).
- Removed “Space has been disabled” messages from log (HOSTSERVER-647).
3.6.1 (2016-11-15)¶
The Host Server now retrieves details of Owners from the Registration Server. The email address and Provider Code of the Owner will be updated within 24 hours if changed in the Registration Server. So-called “magic usernames” are now only displayed when the Host Server has no email address for a user (HOSTSERVER-629).
See Sync Owner Data Task for details.
When adding a Space Owner in the Admin Console, the Host Server will now check that the user is a registered TeamDrive User. Note that that it is now possible to add an Owner by specifying the Email address only (HOSTSERVER-640).
The statistics poll method now returns the Space owner details to the TeamDrive Client (HOSTSERVER-639).
Admin Console: fixed error handling when creating an Owner using the Admin Console.
Admin Console: fixed an error when using the Admin Console to create a new Depot.
The Client will now be prevented from inserted Space meta data with zero length (HOSTSERVER-644).
It is now possible to set the system settings
ServiceHostURL
andRegServerURL
to the domain name of the Host Server or Registration Server instead of a complete URL. The Host Server will not automatically convert this to a URL as required (HOSTSERVER-645).The setup process of the Host Server will continue to set these values to complete URLs, although the input fields during setup only require domain names.
3.6.0 (2016-09-01)¶
The Host Server 3.6 requires the YVVA runtime version 1.3.8 or later. Please follow the upgrade instructions in In-place Upgrading from 3.0.013 or 3.5 to 3.6
Host Server Functionality¶
- Improved restore functionality (HOSTSERVER-635).
- The name of the user and the deletion time are now recorded when a Space is deleted (HOSTSERVER-507). Note: this only works if TDP v3 is active, and you are using TeamDrive client version 4.2 or later.
- Added system settings:
NotifyVolumeEmail
,NotifyVolumeWarningLevel
,NotifyVolumeCriticalLevel
. The background task, Volume Warning, has been added to send an email notification when the the disk usage of a volume exceeds the specified levels. - Fixed upgrade from version 3.0.011 (HOSTSERVER-618).
- Added the setting
S3RedirectProtocol
. This setting determines the protocol to be used for redirects to S3, or other Object Stores (HOSTSERVER-622). - Added support for Azure blob storage (HOSTSERVER-583)
- The Host now supports the “Range” header (HOSTSERVER-577). This enables the direct streaming of videos. Note, only one range per call is supported.
Administration Console¶
- Added functionality to restore a deleted space (HOSTSERVER-633).
- Host Server settings have now been divided into groups (HOSTSERVER-574).
- The modification time of settings in now displayed in the Admin Console (HOSTSERVER-575).
API¶
- Added
movespace
API call, which moves a Space to another Depot (HOSTSERVER-636). - The
getspacedata
API call now accepts the following additional tags:<includedeleted>
,<resultoffset>
and<resultlimit>
(HOSTSERVER-461). - The
getdeptdata
API call now accepts the tags<includedchanges>
which specifies if the change history should be returned with the details of the Depot (HOSTSERVER-497).
Change Log - Version 3.5¶
3.5.8 (2016-09-27)¶
- Fixed problem when using directory scan on XFS with CentOS7. The “teamdrive-volume-id” file was not being correctly created (HOSTSERVER-643).
- The volume ID is now checked on startup of the Apache module (mod_pspace). Previously it was only checked when a Space was created.
3.5.7 (2016-08-29)¶
Note
The Host Server version 3.5.7 requires YVVA runtime version 1.3.8 or later.
Note
Updating the Host Server on CentOS 7 with “yum update” might
update the apache to a newer version. This update could re-install the
deleted “conf”-files in the folder /etc/httpd/conf.modules.d/
and will
prevent starting the apache. Please follow the modified instruction to
disable all modules in the “conf”-files instead of deleting them as
described in configure-apache-24
- Fixed the “back” button after clicking on a link in the Admin Console.
- Fixed restore function: it was possible that an incorrect log offset was calculated after restore (HOSTSERVER-632).
- Organised the settings into groups in the documentation (HOSTSERVER-630). The same grouping is used in the Admin Console in Host Server 3.6.
- The Depot document returned with SERVERFLAGS= contained an invalid terminator. This caused the document to be incorrectly interpreted by the Client and Registration Server (HOSTSERVER-631).
3.5.6 (2016-07-13)¶
- Fixed the traffic reset task. If the setting StatisticRest is blank, then the trask does not run. A quick workaround for this bug is to set the variable to “0”. This must be done directly in the database, on the table pspace.Setting, column Value (HOSTSERVER-623).
3.5.5 (2016-06-09)¶
- Added missing yvva compatibility to td-hostserver background task configuration file
3.5.4 (2016-06-07)¶
Note
The Host Server version 3.5.4 requires YVVA runtime version 1.3.6 or later.
- Fixed a bug that could result in the TeamDrive Client reporting
traffic limit reached, when
EnforceTrafficLimit
is set toFalse
(HOSTSERVER-621). - Added support for CentOS 7 with Apache 2.4
- Fixed the link in to Volumes in the Host overview page (HOSTSERVER-619).
- Fixed dialog used to set the owner of a Depot (HOSTSERVER-616).
- Minor API documentation fix: the position of the
<etl>
tag has been changed, and the order of tags in reply’s now matches the order returned by the server (HOSTSERVER-496). - Admin Console: The Storage and Transfer columns incorrectly showed “MiB MB” as units (HOSTSERVER-612).
- The Host Server was incorrectly setting the Volume full Status bit on Spaces, when the Depot disk limit was reached (HOSTSERVER-611). This error will be corrected automatically.
- Fixed a bug that prevented long running MD5 checks from working correctly.
- An error in the TDP version 3 prevented files from being deleted when the depot was full (HOSTSERVER-610).
3.5.3 (2016-02-02)¶
- Fixed lost password functionality in admin web interface (HOSTSERVER-604).
- Added the
DownloadContentType
setting which may be used to specify the content type of encrypted data returned by Host Server (HOSTSERVER-602). - API function “deletespace” no longer returns an error when deleting a Space that has already been deleted. However, the API also does not return an error if the Space does not exist at all, or if the Space is in another Depot. In these cases, the delete call is just ignored (HOSTSERVER-429).
- Fixed a bug in mod_pspace: if a recently published file was deleted and then published again, the result could be that the file on the server has 0 bytes (HOSTSERVER-601).
- The tags
<disclimit>
and<trafficlimit>
in the “setdepot” call are now optional. - Added
<etl>
tag to the “getspacedata” API-call. The “Traffic Limit Reached” bit will also be removed from the status returned by this call (HOSTSERVER-411).
3.5.2 (2015-12-08)¶
- Fixed bug in schema definition for FileSize column in PublicFile table
- Fixed bug with comparison of timestamp to DATE value in the database because of daylight savings time corrections (HOSTSERVER-578).
- Fixed TD3 Protocol crash in loadSpaces() (HOSTSERVER-580).
- Fixed return of .tdsv files
- Fixed disk usage calculation error in case of host server is connected to an object store (HOSTSERVER-576).
- Fixed duplicate object store log files processing in case of identical or missing S3ToProcessPath and S3ProcessedPath (HOSTSERVER-586)
- Fixed adding external traffic in API-call “getspacedata” (HOSTSERVER-587)
- Fixed retrieval of public file where name comtains reservied URL characters (HOSTSERVER-581)
- Correctly log last.log.lock when reading and writing log files and if no maximum len is given, return the entire log
- Fixed error when adding MOVE action to database –> Illegal mix of collations (HOSTSERVER-589)
- Fixed TD3Protocol: Empty reply for getblob (HOSTSERVER-595)
- Fixed exclude “Error getting size from ...” in case of zero download for object store access log processing (HOSTSERVER-593)
- Corrected RepositoryChanges table duplicate constants
- S3Daemon: Fixed error ‘The Content-MD5 you specified did not match what we received.’ It was possible that the checksum value stored in the database did not match that of the actual file (HOSTSERVER-591).
- S3Daemon: Fixed problem with multipart uploads. If an attempt to transfer a zero length file to S3 it would fail but would try again later so it was stuck in an endless loop (HOSTSERVER-588).
- Added Functionality to move space from one depot to another. The host Admin Console now provides a “Move...” button which can be used to move Spaces to a selected Depot. A new API function, movedepotspaces(), allows the same function to be performed via the API (HOSTSERVER-546). Client version 4.1.2 required to update the new space owner correctly.
3.5.1 (2015-10-09)¶
Documentation¶
- Fixed description of Background Tasks
- Added ssl configuration hint in case of upgrading a server to version 3.5
- Added description for the html templates for password protected published files
Host Server Functionality¶
- Usability: Added a default html template folder to avoid conflicts with customized html templates (HOSTSERVER-572)
- Administration: Fixed divide by zero error in case of depot size and traffic limit are zero (HOSTSERVER-570)
- Administration: German translation is disabled. Only english web interface is supported (HOSTSERVER-569)
- Administration: The new background task for API log cleanup will be created with status enabled instead of disabled. The usage could be controlled using the setting “APILogEntryTimeout” (HOSTSERVER-568)
- Usability: Added html template “url-invalid.html” for expired or invalid token in case of access a published file (HOSTSERVER-567)
- Security improvement: Limit access to allowed log files (HOSTSERVER-564)
- S3 daemon: Added bandwidth limitation for the S3 daemon (HOSTSERVER-563)
- Administration: Added filter (<, >, =) for Space-IDs and Depot-IDs (HOSTSERVER-562)
- Administration: Added setting “APILogEntryTimeout” to define a period in days for deleting api logs (HOSTSERVER-561)
- Administration: Fixed truncated “Add New Admin User”-Button (HOSTSERVER-560)
- Administration: Fixed access to ping.xml (HOSTSERVER-558)
- Administration: Fixed s3d.log file name for log file display (HOSTSERVER-557)
- S3 daemon: Fixed crash in case of multipart upload (HOSTSERVER-556)
- Administration: Fixed displaying info text for “TimeDiffTolerance” setting (HOSTSERVER-553)
3.5.0 (2015-09-21)¶
TeamDrive Host Server Version 3.5 is the next major release following after version 3.0.013.
Note
Please note the the version numbering scheme for the Host Server has been changed starting with version 3.5. The first two digits of the version string now identify a released version with a fixed feature set. The third digit, e.g. “3.5.1” now identifies the patch version, which increases for every public release that includes backwards-compatible bug or security fixes. A fourth digit identifies the build number and usually remains at zero, unless a rebuild/republishing of a release based on the same code base has to be performed (e.g. to fix a build or packaging issue that has no effect on the functionality or feature set).
Version 3.5 contains the following features and notable differences to version 3.0.013. See Change Log - Version 3.0.013 for a detailed description of the change history for that version.
Host Server Functionality¶
- Security enhancement: Files can now be published with an expiration date after which an auto task on the Host Server will automatically remove the published files again. Additionally, published files can now be protected by a password. This functionality requires support on the TeamDrive Client side, which is implemented in versions 4.1 of the TeamDrive Client. For entering the password in a html page, a few templates were added. The templates could be customized and will not overwritten when updating to a newer Host Server version.
- Security enhancement: A request for a published file no longer returns the
actual file directly, except in the case where the request comes from tools
like
wget
orcurl
. Instead, the document returned is an HTML file containing JavaScript calls that load the actual file using a temporary URL. This solves a potential security problem in which URLs of published documents can be inadvertently disclosed to unintended recipients in the following scenario: A TeamDrive user publishes a document that contains URLs pointing to a third-party website (e.g. a PDF or office document). The user, or an authorized recipient of the published URL, clicks on a hyperlink embedded in the document. At that point, the referrer header discloses the document’s publish URL to the third-party website. Someone with access to that header, such as the webmaster of the third-party website, could then access the link to the published document. (HOSTSERVER-316) - A new Client/Server protocol, supporting parallel polling of Spaces for increased throughput/performance, batched delete operations (e.g. emptying the Trash) and “soft” locking of files. These features require support on the TeamDrive Client side, which is scheduled to be implemented in future versions of the TeamDrive Client.
- Performance improvement: The Host Server now uses a database table instead
of action files in the Space Volume’s file system for signalling actions
like uploading or deleting files to the object store. As a result,
s3d
no longer has to perform a full scan of all Space Volumes to look for new or changed files. (HOSTSERVER-284) Additionally, the MD5 digest of a file is also stored in this table, sos3d
does not need to perform a recalculation of the checksum before uploading the file to the object store. During an upgrade from a previous version, any remaining action tag files in the file system will be imported into the database. Afterwards, the server settingImportS3tagFiles
should be set toFalse
. - The S3 daemon
s3d
now only performs a full scan of all Space Volumes once per day by default, looking for old files to be transferred to the object store. The age of these files is set via the settings variableMaxFileAge
. The maximum file age should be set long enough to ensure that no file that may still be in the process of being uploaded by a Client will be sent to the Object Store, otherwise the Client would have to restart the upload from scratch.
Administration Console¶
- Security improvement: Added support for managing multiple user/administrator accounts. There are 2 types of users: Superuser and Administrator. Only the Superuser may manage other users. The Administrator may view all users and only update his own user account. (HOSTSERVER-366)
- Security improvement: Disabled auto completion on the login form. (HOSTSERVER-379)
- Security improvement: The complexity of entered passwords is now indicated. (HOSTSERVER-374)
- Security improvement: it is now possible to enable two-factor authentication via email. If enabled, the user is required to enter a security code provided via email in addition to his username and password.
- Security improvement: On login, the user will get an error if he has another logged in session. To proceed, the user must check the checkbox titled: “Close my other login sessions”. (HOSTSERVER-376, HOSTSERVER-377)
- Security improvement: The following events are now logged at the “notice” level: login, logout, failed login attempts and changes to user accounts.
- Security improvement: the amount of search results (e.g. Spaces, Depots or
users) is now limited to a maximum defined by the
MaxRecordsDisplayed
setting, which can only be changed by the Superuser. - Administration: It is now possible to change a Depot’s status (e.g. enabled, disabled, deleted)
- Administration: Added support for viewing selected server log files and the Host Server API log. (HOSTSERVER-348, HOSTSERVER-243)
- Administration: It is now possible to track and display modifications made to Space Depots (e.g. via API calls coming from the Registration Server or via the Host Server Admin Console). (HOSTSERVER-388)
- Administration: When creating a new Space Volume via the Administration Console, the system now checks if the directory actually exists on the file system before creating the Volume. (HOSTSERVER-349)
- Usability: References like Depot Names, Volume names and owners in the Space list are now clickable, to improve the quick navigation between pages. (HOSTSERVER-390)
- Usability: Objects like Spaces or Depots that have been marked as deleted
are now hidden in result lists by default. They can be made visible again by
changing the setting
ShowDeletedObjects
fromfalse
totrue
. (HOSTSERVER-442) - Usability: Administration Console now better visualizes errors like missing Space Volumes.
- Usability: Units displayed for disk space or traffic usage now use the correct units (e.g. MiB, or GiB), to avoid confusion caused by conversions between different units. Space and traffic levels are now displayed in percent instead of absolute units.
Administration / Installation¶
- Administration: The Host Server’s log levels have been aligned with the ones
used by the Registration Server and the Yvva Runtime Environment. Valid log
levels are: 1 (Error), 2 (Warning), 3 (Notice), 4 (Trace), 5 (Debug).
In production mode the default log level is 3 (Notice).
Setting the log file name to
syslog
will now send log output to the local syslog service. You can add an optional “Log Identity after a colon in the log file name, for example:syslog:my-log-id
. The default Log Identity is name of the program, e.g.s3d
ortshs
. - Administration: The central log file
/var/log/td-hostserver.log
is the central log location for all Yvva-based components (e.g. the Host Server API, Administration Console ortd-hostserver
background service); the log files used in previous versions (e.g./var/log/mod_yvva.log
,/var/log/p1_autotask.log
,/var/log/pbvm.log
) will no longer be used. - Administration: TSHS now supports the additional commands
disable-s3-host
,enable-s3-host
anddelete-s3-host
that allow for disabling/removing the synchronization of objects to an S3-compatible object store. Callingdisable-s3-host
marks a host entry as “disabled”. Callingdelete-s3-host
deletes a host entry unless the entry is referenced by a file. In this case the entry will be marked as deleted. If an entry is marked as disabled or deleted, no further data will be uploaded to the object store. However, accessing existing objects from the object store will continue to work. Callingenable-s3-host
will re-enable the synchronization of objects to the object store, including the upload of all objects that have been uploaded to TSHS while the object store was marked as disabled. If a disabled or deleted host is marked as current, then TSHS will generate an error on each write attempt. - Administration: Added an auto task that can be enabled to send out notification emails if a Space Volume’s disk utilization reaches a configurable level.
- Administration: Added an auto task that removes published files that have reached their expiry time.
- Administration: Added an auto task that can be enabled to delete API log
entries older than 30 days from the
hostapilog
table. - Installation: TSHS now supports reading options from a configuration file.
The default is
/etc/tshs.conf
. The default options that were previously stored in the TSHS init script/etc/init.d/tshs
have now been moved to the configuration file instead. (HOSTSERVER-303) - Installation: Optionally configure email support (required when using two-factor authentication). (HOSTSERVER-437)
- Installation: The initial Host Server setup process now asks for both a user name and password for the Superuser account. (HOSTSERVER-438)
- Installation: Host Server 3.5 now requires Yvva Runtime Environment version 1.2 or later. This version is included in the Host Server’s yum package repository and will be installed automatically.
- Installation: The distribution now contains the tool
mys3
, which can be used to interact with an S3 compatible object store.
API¶
- Changes to a Space Depot performed by the API functions
addusertodepot
anddeleteuserfromdepot
are now added to the Depot’s change log. - The MD5 checksum value calculated over API requests no longer needs to be passed in lowercase when submitting the request. (HOSTSERVER-426)
- For debugging purposes, erroneous API requests are now logged to the API requests table as well. (HOSTSERVER-465)
Change Log - Version 3.0.013¶
3.0.013.15 (2015-08-17)¶
- S3: Fixed bug with high IO, upload could not proceed and other uploads will be blocked. (HOSTSERVER-529)
3.0.013.14 (2015-06-04)¶
- S3: Fixed bug in parsing S3 access log entries for traffic calculation
(resolves
Error getting spaceid
errors intd-hostserver.log
). Additionally, the S3 log analyser script now only downloads and processes objects from the log bucket that contain the stringaccess_log-
. (HOSTSERVER-500) - mod_pspace: Added support for calculating traffic from S3-compatible object
stores that do not support access logging via log buckets in the way that
Amazon S3 does it. Now, if a redirect to S3 is performed and
S3LogBucketName
has not been specified, the request length will be logged as bytes sent. (HOSTSERVER-499) - s3d: The S3 daemon has now been split into two processes, a worker process and a watchdog process. If the worker process dies, the watchdog will restart it. Killing the watchdog process will also kill the worker process. The watchdog will always try to restart the worker, but depending on the frequency with which the worker is dying the watchdog will wait before trying to restart it. The minimum wait is 3 seconds, the maximum is 30 minutes. (HOSTSERVER-508)
3.0.013.13 (2015-05-11)¶
- mod_pspace/s3d: Added workaround to handle a deviation in the Ceph 0.8 Object
Store S3 API: the “list multipart upload parts” API request returns
ListMultipartUploadResult
instead oflistpartsresult
(see BUG#11494 in the Ceph bug tracker for details). (HOSTSERVER-484) - mod_pspace: Added missing call to
s3d_delete()
when an “Upload to file that has already been transfered to S3” is detected. Due to the missing call, Clients could end up in an endless loop, showing a “wrong md5” error in the log file. (TDCLIENT-2045) - mod_pspace: Added new module option
watched_space_id
that can be used to trace Client accesses to a specific Space for debugging purposes. See Tracing Client Accesses to a Single Space for details. (HOSTSERVER-486)
3.0.013.12 (2015-04-14)¶
- s3d: Uploading the
last.log
file failed with a checksum error if the log was written to before the upload was complete.s3d
now only transfers the data size used when calculating the checksum. This will allow thelast.log
file to grow while being uploaded to S3. (HOSTSERVER-474) - s3d: Fixed unsafe object references during multi-part uploads which may have
lead to
s3d
crashes. (HOSTSERVER-454) - Installation: The
td-hostserver
RPM package will no longer reset the permissions and ownerships of the/spacedata
and/spacedata/vol01
directories to700
andapache:apache
during an update, if they had been changed by the administrator after the initial installation. Depending on how the Space Volume is mounted, the RPM installation could fail with an error likeerror: unpacking of archive failed on file /spacedata
. A new installation will still create the directories using these permissions/ownerships by default. (HOSTSERVER-401) - Host Server: Converted the type of the
StatisticRest
setting fromINT
toDATE
, to avoid an error that could occur when updating from very old Host Server Versions (theresetTraffic()
auto task failed with anInvalid integer literal
error). This also fixes a potential issue that could result in the reset routine being run multiple times on the day the traffic is reset. (HOSTSERVER-478) - Documentation: Fixed link structure in the HTML documentation so that clicking Next and Previous within a document works as expected. (HOSTSERVER-471)
3.0.013.11 (2015-03-30)¶
- Administration Console: Updated logo and favicon.
- Host Server: Updated some error messages by replacing “Repository” with “Depot”. Ensure that a Space Depot that has been marked as “Deleted” no longer allows the creation of new Spaces. (HOSTSERVER-456)
- mod_pspace: Reduced logging of errors by only logging Client accesses to deleted Spaces as an error if the Space status is zero. (HOSTSERVER-449)
- mod_pspace: Fixed a crashing bug that could occur in rare situations. (HOSTSERVER-457)
- s3d: Fix unsafe access to the thread pool that may have caused
s3d
to crash in certain situations. (HOSTSERVER-454) - s3d: Fixed a problem that caused a crash if a multipart upload was interrupted before completion and then restarted again. The parts list could have holes in it for the parts that were successfully uploaded in the first try.
- Documentation: Added section that instructs the user to perform a
yum update
after installing the VM image. Reformatted the 3.0.013 release notes and replaced the table with regular sections for improved readability. - Documentation: Added Failover and Scalability chapter to the Administration Guide, added description of the startup sequence/dependencies to the Installation Guides. (HOSTSERVER-431)
3.0.013.10 (2015-01-26)¶
- s3d: Fixed a problem that caused a crash from time to time. The crash would occur if a request for an object’s header timed out or was interrupted.
- Host Server: Fixed bug in the calculation of
DiskUsed
for Space Volumes that did not contain any Spaces. (HOSTSERVER-452) - Administration Console: The Volume repair button now only appears if a repair is actually required (previously it appeared whenever there was an error on the volume).
- Installation: added a new RPM package
td-hostserver-doc-html
that contains the Host Server documentation in HTML format, installed in the Host Server’s Apache document root/var/www/html/td-hostserver-doc/
. Access to the documentation can be restricted by editing/etc/httpd/conf.d/td-hostserver-doc.httpd.conf
. (HOSTSERVER-450) - Installation: fixed bug in upgrading from older versions and the
hostapilog
database did not get created. (HOSTSERVER-446)
3.0.013.9 (2015-01-14)¶
- mod_pspace/s3d: fixed
unexpected object "vol01/..." starting with 'vol' was found in the bucket...
error , which prevented the Apache module from starting. This error could occur after updating from a previous version if S3 was already enabled, and the old object format (prefixed by volume name) was used on an S3 compatible object store. (HOSTSERVER-447)
3.0.013.8 (2015-01-13)¶
- API: Added missing
activatedepot
API command and added new tag<changeinfo>
to add a free form comment to the change history of the following API commands:activatedepot
,assignusertodepot
,createdepotwithoutuser
,deactivatedepot
,deletedepot
. Updated API version to 3.0.004. (HOSTSERVER-337) - Installation: fixed typo in the installation script that adds the
RewriteRules to
ssl.conf
. Added RewriteRule in preparation for accepting Client requests for Space data via SSL/TLS (not supported yet). - Installation: the binary tarball distribution now includes debug versions of
the Host Server binaries (
s3d-debug
andtshs-debug
) and Apache module (mod_pspace-debug.so
, to better support analyzing possible crashing bugs. (HOSTSERVER-445) - Installation: fixed possible upgrade error from previous versions: moving
the MySQL table
pbpg.Keys
to thepspace
database failed if an emptypspace.Keys
table already existed. (HOSTSERVER-441)
3.0.013.7 (2014-12-12)¶
- Fixed error in creating an index during the initial MySQL table creation (HOSTSERVER-440)
3.0.013.6 (2014-12-09)¶
- Installation: fixed possible upgrade error from 3.0.011 when the MySQL
database
pbpg
still existed, but theKeys
table was already moved to thepspace
database (HOSTSERVER-427) - Fixed bug in which failed Auto Tasks were not executed anymore (HOSTSERVER-407)
- mod_pspace: fixed possible crash when system settings are NULL (e.g. in an upgrade scenario from 3.0.011 to 3.0.013, when httpd was started before yvvad performed the required schema updates)
- mod_pspace: Fixed possible “Admin API: AES decode error- curruption detected” error when updating from older versions (timing issues could result in the generation of duplicate private keys) (HOSTSERVER-420, HOSTSERVER-422)
- Increased the size of the
S3Options
settings field from 200 to 2000 chars, to accommodate longer option strings required for certain OpenStack environments (HOSTSERVER-425) - Installation: updated
RewriteRule
sets in the httpd configuration files (removed obsolete/depot
rule, HOSTSERVER-424)
3.0.013.5 (2014-09-26)¶
- mod_pspace: fixed a Space corruption bug that could occur when updating from a previous Host Server version to version 3.0.013 and Space Volumes were using a non-standard naming scheme (not “volxxx”)
- Admin Console: added “Repair” button that allows performing an automatic repair of Volumes affected by the corruption bug. Clients will be notified to perform a Space Restore operation on affected Spaces.
3.0.013.4 (2014-09-18)¶
- Admin Console: fixed 404 errors when opening the Admin URL without a trailing slash (HOSTSERVER-398)
- Admin Console: the input focus is now automatically set to the password field (HOSTSERVER-392)
- s3d: Fixed bug in path deletion on S3: if the path ended with ‘/’ it wasn’t being deleted.
- s3d: exceptions are now logged in
/var/log/s3d.log
3.0.013.3 (2014-09-05)¶
- mod_pspace: Replaced the previously used MD5 implementation with calls to the MD5 routines provided by OpenSSL (yielding a 70% performance improvement when calculating MD5 checksums on large files) (HOSTSERVER-355)
- mod_pspace: consolidated brand-specific settings into one place and disabled multi-part uploads for OpenStack
- mod_pspace: Fixed bug where failed uploads (resulting in MD5 checksum failures) would still be accounted for as bytes written in the Space usage statistics (HOSTSERVER-352)
- Fixed autotask
resetTraffic()
to properly reset the traffic for Spaces that had theSPACE_TRAFFIC_FULL
status flag enabled. (HOSTSERVER-353) - Installation: security enhancement: set
ServerTokens
toProd
andServerSignature
toOff
inhttpd.conf
to disable displaying the Apache Server version and OS version in the HTTP headers and on error pages (HOSTSERVER-357) - mod_pspace: Disabled unnecessary buffering of files fetched from S3 object store and passed back to the client. (HOSTSERVER-356)
- tshs:
add-s3-host
will ping the S3 service before actually adding the host details. - Admin Console: security enhancement: don’t display the version and build number on the login page and https redirection page (HOSTSERVER-359)
- Security enhancement: disabled unneded HTTP methods in
td-hostserver.httpd.conf
(ony allow GET, POST, PUT, disable HEAD, OPTIONS, TRACE) (HOSTSERVER-361) - Virtual appliance security enhancement: set
ServerTokens
toProd
andServerSignature
toOff
inhttpd.conf
to disable displaying the Apache Server version and OS version in the HTTP headers and on error pages (HOSTSERVER-357)
3.0.013.2 (2014-07-14)¶
- To avoid confusion, the S3-related configuration option
openStackAuthURL
was renamed toopenStackAuthPath
3.0.013.1 (2014-07-11)¶
Host Server Version 3.0.013 is the next major release following after version 3.0.011 (Version 3.0.012 was an internal release that has not been published).
Version 3.0.013 contains the following features and notable differences to version 3.0.011:
- The TeamDrive Host Server installation can now be performed via RPM on Red Hat Enterprise Linux 6 and derivative distributions, which significantly improves the installation procedure and the process of applying updates.
- The initial setup and registration of a Host Server is now fully web-based.
It’s no longer necessary to provide a
hosting.txt
orproperties
file. Instead, all the required information can be entered in a web form. - The entire Host Server configuration is now stored in the MySQL database. This includes configuration settings for S3 daemon and TSHS.
- The web-based TeamDrive Hosting Service Administration Console has been improved significantly, by simplifying the work flows for common administration tasks and fixing several usability issues.
- TSHS, the TeamDrive Scalable Hosting Storage and the TeamDrive S3 Daemon provide additional scalability options to expand the storage capabilities of a TeamDrive Hosting Service.
- It’s now possible to generate a monthly report that contains detailed statistics about all existing Depots and Spaces within these depots, including the monthly traffic and disk usage.
- The Host Server no longer depends on the PrimeBase Application Environment.
Instead, it now uses the Yvva Runtime Environment, which replaces the
following components:
mod_yvva
replacesmod_pbas
for providing the web-based Administration Console and API. The stand-alonepbas
instance is no longer required. As a consequence, thepbur
MySQL database which was used by PBAS to manage user accounts and privileges is no longer required and has been removed.yvvad
replacespbac
for running background tasks. The formerp1_autotask
background task PBAC instance is now provided by the servicetd-hostserver
, which usesyvvad
.yvva
replacespbac
for command line operations that involve executing PBT code on the shell.
- The installation location of the TeamDrive PBT code has been changed from
/home/teamdrive/pbas
to/opt/teamdrive/hostserver/
. - The
sakgen
binary that used to be installed in/home/teamdrive/sakh
is no longer required. Instead, the functionality to encrypt Space Depot access keys is now provided by thetshs
binary. - All TeamDrive Host Server processes now run under the user ID used by the
Apache http Server (
apache
). A dedicatedteamdrive
user account is no longer required. - By default, the MySQL databases are now installed in the default location
/var/lib/mysql
instead of/spacedb
, which made it difficult to enable SELinux on the MySQL instance. - For security reasons, the MySQL credentials required for accessing the MySQL
Database are no longer stored in the default MySQL configuration file
/etc/my.cnf
. Instead, the[p1db]
options group has now been moved into a dedicated configuration file/etc/td-hostserver.my.cnf
, only readable by theapache
user. - The Apache httpd Server configuration file has been renamed from
teamdrive.conf
totd-hostserver.httpd.conf
. - The overall robustness of the TeamDrive Host Server has been improved by issuing more meaningful error messages and performing more safety and consistency checks.
- Each Space Volume now contains a file
teamdrive-volume-id
that contains a unique global volume ID, to ensure that multiple volumes are mounted to the correct location.
Change Log - Version 3.0.011¶
3.0.011.6 (YYYY-MM-DD)¶
- HOSTSERVER-228: Add settings for
ClientPollFrequency
andStatisticPollFactor
- HOSTSERVER-241: Moved
[p1db]
group frommy.cnf
to a dedicated configuration file/etc/td-hostserver.my.cnf
to improve security and packaging.
3.0.011.5 (2014-04-22)¶
- HOSTSERVER-224: Added
SpaceStatisticEnabled
and SpaceStatisticExportPath` - Updated
teamdrive.conf
Apache configuration file: wrapped long lines and updateds3daemon
file locations to match the defaults suggested in the Installation Manual - HOSTSERVER-191: Fixed Magic Username problem with
sakgen
by enclosing them with single quotes to avoid the shell from expanding them as variables. Fixed “bad file descriptor error”
3.0.011.4 (2014-03-12)¶
- Fixed HOSTSERVER-99: created database migration script
mysql/v3.0.010_to_v3.0.011.sql
to update the table structures, move the Keys table from databasepbpg
topspace
and renamed databasetd2apilog
tohostapilog
. - Removed default
API_SALT
in sql script. - Improved
hosting.txt
value validation.
3.0.011.3 (2014-03-03)¶
- Updated version number in
pbstab
from “4546” to “4547” - Fixed HOSTSERVER-172: The default MySQL table definition file
mysql/p1space_schema.sql
contained a wrong value for the configuration variablePathToSAKConverter
. Instead of/home/teamdrive/sakh/sakgen
it should have been/home/teamdrive/sakh/
.
3.0.011.2 (2014-02-07)¶
- Updated sample
hosting.txt
file: no trailing slash afterREGSERVERURL
- Updated and completed Translation files (grammar, typos, obsolete terms)
- Set
PathToSAKConverter
configuration variable to/home/teamdrive/sakh/sakgen
by default - Added S3Daemon config and script files to the installation package
- Fixes to object store access log processing
3.0.011.1 (2014-02-04)¶
- Added parsing and error handling for
API_IP_LIST
andAPI_SALT
from thehosting.txt
. - pbstab: changed log file from
/home/teamdrive/pbas/setup/pbac.log
to/var/log/p1_autotask.log
(HOSTSERVER-145) pbstab
: fixed wrong path top1ctl.dal
- Fixed setting space status bit
- Fixed autotask debug output
- Fixed typos and obsolete reference to p1ctl from the translation files
- Changed configuration variable 340 “Protocol Log File”
in
pbas.env
from “<< Default Log >>” to “/var/log/pbas.log” - note that this file needs to be created and assigned to the user running the PBAS instance (touch /var/log/pbas.log ; chown teamdrive:teamdrive /var/log/pbas.log
) - Fixed HOSTSERVER-150: removed reference to td2apilog database
3.0.011.0 (2014-01-28)¶
- First build of the 3.0.011 branch, using the scripted build