Troubleshooting

List of relevant configuration files

/etc/httpd/conf.d/td-hostserver.httpd.conf:

The configuration file that loads and enables the TeamDrive Host Server-specific modules for the the Apache HTTP Server:

  • mod_pspace.so: this Apache module provides the actual Host Server functionality by accepting incoming data from the TeamDrive clients as well as delivering data to other clients upon request.
  • mod_yvva.so: this Apache module is responsible for providing the web-based Host Server Administration Console as well as the Host Server API interface.
/etc/logrotate.d/td-hostserver:
This file configures how the log files belonging to the TeamDrive Host Service are being rotated. See the logrotate(8) manual page for details.
/etc/td-hosting.conf:
This file defines how the td-hostserver background service is started using the yvvad daemon.
/etc/td-hostserver.my.cnf:
This configuration file defines the MySQL credentials used to access the pspace MySQL database. It is read by the Apache modules mod_yvva and mod_pspace as well as the yvvad daemon that runs the td-hostserver background tasks and the yvva command line client.
/etc/yvva.conf:
This configuration file contains configuration settings specific to the Yvva Runtime Environment that are shared by all Yvva components, namely the mod_yyva Apache module, the yvvad daemon and the yvva command line shell.
/etc/tshs.conf:
This configuration file defines a number of maintenance tasks performed by the tshs background service.

List of relevant log files

In order to debug and analyse problems with the Host Server configuration, there are several log files that you should consult:

/var/log/td-hostserver.log:

The log file for the Yvva Application Server module which provides the web-based Host Server Administration Console and API. Consult this log file when you have issues with associating the Host Server with the Registration Server, errors when issuing API requests or problems with the Administration Console. You can increase the amount of logging by changing the Yvva setting log-level from error to trace or debug in /etc/httpd/conf.d/td-hostserver.httpd.conf:

<Location /yvva>
  SetHandler yvva-handler
  YvvaSet root-path=/opt/teamdrive/hostserver
  YvvaSet mysql-cnf-file=/etc/td-hostserver.my.cnf
  YvvaSet log-file=/var/log/td-hostserver.log
  YvvaSet log-level=error
</Location>

After changing these values, you need to restart the Apache HTTP Server service using service httpd restart.

This log file is also used by the td-hostserver background task. Check this one to verify that background tasks are being processed without errors. The log file location can be configured by changing the file name passed to the log-file option in the configuration file /etc/td-hosting.conf. The log level can be increased by changing the default value error for the log-level option to trace or debug. Changing these values requires a restart of the td-hostserver background process using service td-hostserver restart.

/var/log/mod_pspace.log:
This log file contains error messages related to the mod_pspace Apache module, particularly when using an S3 compatible object store or TSHS. It needs to be writable by the user that the Apache HTTP Server runs under (apache by default). The log file location is configured by the server setting ModuleLogFile and the amount of logging can be changed by adjusting the server setting ModuleLogLevel via the Host Server Administration Console. The value defines the maximum level of logging of messages logged: 1 = Error, 2 = Warning, 3 = Notice, 4 = Trace, 5 = Debug. Changing these values requires restarting the Apache HTTP Server.
/var/log/httpd/:
The Apache HTTP Server’s log files (e.g. error_log) might also contain additional relevant error messages (e.g. from mod_pspace) that should be checked. The amount of logging is affected by the ModuleLogLevel setting described above.
/var/log/tshs.log:
This log file contains errors and other messages generated by the tshs background service. The log file location and amount of output are defined in file /etc/tshs.conf, via the options log-file and log-level. Possible values in the order of verbosity are protocol, error, warning, trace, debug. The default is warning.
/var/log/s3d.log:
This log file is written by the TeamDrive S3 daemon s3d and provides log messages and errors specific to the s3d background service. The log file location is defined in the init script /etc/init.d/s3d.

Enable Logging with Syslog

As outlined in List of relevant log files, the TeamDrive Host Server logs critical errors and other notable events in various log files by default.

Starting with Host Server version 3.5 and Yvva 1.2, it is now possible to redirect the log output of some server components to a local syslog instance as well.

Note

Please note that other components of the TeamDrive Host Server, e.g. mod_pspace, s3d or tshs currently do not provide syslog support. This limitation may be lifted in future versions of the TeamDrive Host Server software.

Syslog support is an essential feature for auditing, security and/or compliance reasons, as it allows you to funnel all log messages into a centralized syslog server.

This makes it easier to monitor the logs for critical events or errors and prevents tampering with the log files in case of a security breach. It also helps to maintain control over the disk space utilization on the server, as growing log files can’t accidentally fill up the file system.

To enable syslog support, the log file name in the log-file setting has to be replaced with the keyword syslog. Optionally, a custom process identifier can be supplied, by appending it to the syslog keyword, using a colon as the separator, e.g. log-file=syslog:my_process_identifier. If not used, the default process identifier will be used, which is the name of the program executable.

To enable syslog support for the Yvva-based td-hostserver backround service, edit the log-file setting in file /etc/td-hosting.conf as follows:

log-file=syslog:td-hostserver

You need to restart the td-hostserver background service via service td-hostserver restart in order to activate this change. If the log-level is set to debug you will now see log messages appearing in /var/log/messages:

Jun 23 11:57:33 localhost td-hostserver: notice: yvvad startup
Jun 23 11:57:33 localhost td-hostserver: notice: Using config file:
/etc/td-hosting.conf
Jun 23 11:57:33 localhost td-hostserver: notice: No listen port
Jun 23 11:57:33 localhost td-hostserver: notice: yvvad running in repeat 60
(seconds) mode

To enable syslog support for the Host Server API and Administration Console, edit the YvvaSet log-file setting in file /etc/httpd/conf.d/td-hostserver.httpd.conf:

YvvaSet log-file=syslog

You need to restart the Apache HTTP Server via service httpd restart in order to activate this change. If the log-level is set to debug you will now see log messages appearing in /var/log/messages:

Jun 23 12:06:04 localhost mod_yvva: notice: mod_yvva 1.2.1 (May 21 2015
11:00:12) startup OK

Tracing Client Accesses to a Single Space

For debugging issues with a specific Space, it might be useful to enable more verbose tracing of activity between the Host Server and the TeamDrive Clients accessing this Space.

For this purpose, access to that Space can be traced by providing the Space’s ID to the option watched_space_id in /etc/httpd/conf.d/td-hostserver.httpd.conf as follows:

<Location /primespace>
      SetHandler pspace-handler
      MySQLCnf /etc/td-hostserver.my.cnf

      watched_space_id <space ID>

      # Necessary to ignore the extra Range-header
      # (see Range-header note in the documentation)
      RequestHeader unset Range
</Location>

Restart the Apache HTTP Server with service httpd restart. Any activity on the selected Space will now be logged into the log file /var/log/mod_pspace.log.

Note

Remove this option and restart the Apache HTTP Server once you’ve finished analyzing the problem, to avoid uncontrolled growth of the log file.

Common errors

Web Installation: “500 Internal Server Error”

This error can be triggered by several error conditions. Check the log file /var/log/td-hostserver.log for details.

Some common errors include:

[Error] -12036 (2002): Can't connect to local MySQL server through socket
'/var/lib/mysql/mysql.sock' (25)
[Error] "startup.yv" (80)

The local MySQL Server’s socket file can’t be opened. This could either be a permission problem, or the MySQL Server is simply not available. Check that MySQL is actually up and running (e.g. by running service mysqld status) and restart it, if necessary. If the error persists, check the MySQL error log file (usually /var/log/mysqld.log) for hints.

Similarly, an error like the following one indicates that a remote MySQL Server might not be answering (e.g. because of a firewall rule or because it’s not running):

[Error] -12036 (2003): Can't connect to MySQL server on
'mysql.yourdomain.com' (107)
[Error] "startup.yv" (80)

If you see Access denied errors like the following one:

[Error] -12036 (1045): Access denied for user 'teamdrive'@'localhost' (using
password: YES)
[Error] "startup.yv" (32)

Either the username or password used to connect to the MySQL Server are wrong. Double check that the MySQL username and password provided in /etc/td-hostserver.my.cnf are correct, e.g. by trying to connect to the MySQL server using these credentials with the mysql command line client.

If you see the following error when connecting to a remote MySQL Server:

[Error] -12036 (1130): Host 'regserver.yourdomain.com' is not allowed to
connect to this MySQL server
[Error] "startup.yv" (80)

Check the TeamDrive MySQL user’s privileges on the remote MySQL server, e.g. by running SHOW GRANTS FOR `teamdrive`@`regserver.yourdomain.com`; and make sure that this user is allowed to connect to the MySQL server from the Registration Server’s host.

Errors When Registering the Host Server

If the Host Server Registration fails, check /var/log/td-hostserver.log on the Host Server as well as /var/log/td-regserver.log on the Registration Server for hints (/var/log/pbt_mod.trace for Registration Server versions before version 3.5) . See the Troubleshooting chapter in the Registration Server Installation Manual for details.

MySQL Errors When Upgrading From an Older Host Server Version

If you observe Access denied or Unknown database errors from the MySQL server like the following ones after starting the updated TeamDrive Host Server using an older MySQL table structure:

[Note] DROP DATABASE pbpg;
[Error] -12036 (1044): Access denied for user 'teamdrive'@'localhost' to
database 'hostapilog'
[Error] "p1setup.pbt" P1Setup:upgradeSettings(328)
[Error] "p1setup.pbt" P1Setup:setupDatabase(14)
[Error] "p1setup.pbt" (506)

Unknown database:

[Error] -12036 (1049): Unknown database 'hostapilog'
[Error] "p1setup.pbt" P1Setup:upgradeSettings(328)
[Error] "p1setup.pbt" P1Setup:setupDatabase(14)
[Error] "p1setup.pbt" (506)
[Error] "p1_shared.pbt" (2)

Double check that the hostapilog database actually exists and that the teamdrive user has the required privileges to access it.

Create the database using CREATE DATABASE hostapilog; and grant the required privileges using GRANT ALL PRIVILEGES ON `hostapilog`.* TO 'teamdrive'@'localhost';. Restart the TeamDrive Service again using service td-hostserver restart, it should now conclude the schema conversion.

If you observe a Can't connect to local MySQL server error like the following one in /var/log/httpd/error_log:

[notice] mod_pspace 1.6.17 Loaded; Build May  6 2015 12:42:39;
Crash-Reporting-Disabled
[error] Failed to boot Admin API: MySQL 2002:
Can't connect to local MySQL server through socket
'/var/lib/mysql/mysql.sock' (2)

or in /var/log/td-hostserver.log:

[Error] -12036 (2002): Can't connect to local MySQL server
through socket '/var/lib/mysql/mysql.sock' (2)

Double check that the MySQL Server is up and running and that the socket configuration setting in the [mysqld] group in /etc/my.cnf matches the one in /etc/td-hostserver.my.cnf.

The default value is /var/lib/mysql/mysql.sock. If the value in my.cnf is different, e.g. /tmp/mysql.sock, we suggest to revert back to the default value there instead of changing it in td-hostserver.my.cnf (unless you have an explicit reason to change the default socket path, of course).

Restart MySQL and the TeamDrive Hosting Services after changing this value.

Admin Console: Clicking on “Host” Results in a “500 Internal Server Error”

If you observe an error message like the following when clicking on Host in the Host Server Administration Console:

500 Internal Server Error
ERROR -1: TshsMain: void CSDBConn::connect(CSDB.cc:1116) MySQL 1044: Access
denied for user 'teamdrive'@'localhost' to database 'tshs_admin'

Or:

500 Internal Server Error
ERROR -1: TshsMain: void CSDBConn::connect(CSDB.cc:1116) MySQL 1049: Unknown
database 'tshs_admin'

You likely changed the setting TSHSEnabled to True, but did not configure the MySQL settings for accessing the tshs_admin database in /etc/td-hostserver.my.cnf.

If you changed the setting by accident, simply set TSHSEnabled back to False.

Otherwise, consult the chapter TeamDrive Scalable Hosting Storage in the Team Drive Host Server Administration Guide for details on how to enable and configure TSHS properly.

“Duplicate key” MySQL errors when updating the database

If you observe “Duplicate key” errors in the Traffic or Owner tables when upgrading these to the latest schema version, you first need to manually remove the duplicates via the MySQL client or another tool like MySQL Workbench. Older versions of the Host Server database schema did not have UNIQUE constraints on some columns, which caused the creation of duplicate entries. For the Traffic table, this usually only affects older traffic accounting information that can safely be removed.

Duplicates in the Owner table are likely caused by user names or email addresses that refer to the same user account, but using different capitalization. In this case it helps to cross-reference the affected users with their information in the Registration Server Database - likely one of these accounts has not been actively used and can be deleted. Please contact support@teamdrive.net if you need assistance in resolving these conflicts.

Admin API Error: MySQL 1040: Too many connections

On a busy server, you might observe one of the following error messages in the Apache HTTP Server’s error log file from time to time:

[error] Failed to boot Admin API: MySQL 1040: Too many connections
[error] [client xxx.xxx.xxx.xxx] (500)Unknown error 500: Admin API Error:
MySQL 1040: Too many connections

In /var/log/td-hostserver.log you might observe a similar error:

[Error] -12036 (1040): Too many connections
[Error] "startup.yv" (80)

This error indicates that the number of child processes spawned by the Apache HTTP Server (e.g. when many TeamDrive Clients attempt to connect to the Host Server concurrently), causes the MySQL Server to run out of threads for handling the incoming database connections.

By default, the MySQL Server is configured to accept 151 concurrent connections. Each Apache child process can establish up to two MySQL connections (one for mod_pspace and one for mod_yvva, depending on what kind of requests it needs to serve). Therefore, the maximum number of connections should be adjusted to be at least 1.5 times the maximum number of child processes spawned by the Apache HTTP Server (defined by the MaxClients directive in the Apache HTTP Server configuration file /etc/httpd/conf/httpd.conf).

The value can be changed by adding the system variable max_connections to the [mysqld] configuration group in the MySQL Server configuration file /etc/my.cnf, e.g.:

[mysqld]
datadir=/var/lib/mysql
max_allowed_packet=4M
max_connections=350
socket=/var/lib/mysql/mysql.sock
user=mysql

You need to either restart the MySQL server in order to apply this change, or change the value at run-time, by running the following SQL statement as the MySQL root user:

mysql> SET GLOBAL max_connections=350;

Keep in mind that increasing the maximum number of connections also increases the memory requirements of the MySQL Server. For more details, please consult the MySQL Server and Apache HTTP Server documentation: