Troubleshooting¶
List of relevant configuration files¶
/etc/httpd/conf.d/td-hostserver.httpd.conf
:The configuration file that loads and enables the TeamDrive Host Server-specific modules for the the Apache HTTP Server:
mod_pspace.so
: this Apache module provides the actual Host Server functionality by accepting incoming data from the TeamDrive clients as well as delivering data to other clients upon request.mod_yvva.so
: this Apache module is responsible for providing the web-based Host Server Administration Console as well as the Host Server API interface.
/etc/logrotate.d/td-hostserver
:- This file configures how the log files belonging to the TeamDrive Host
Service are being rotated. See the
logrotate(8)
manual page for details. /etc/td-hosting.conf
:- This file defines how the
td-hostserver
background service is started using theyvvad
daemon. /etc/td-hostserver.my.cnf
:- This configuration file defines the MySQL credentials used to access the
pspace
MySQL database. It is read by the Apache modulesmod_yvva
andmod_pspace
as well as theyvvad
daemon that runs thetd-hostserver
background tasks and theyvva
command line client. /etc/yvva.conf
:- This configuration file contains configuration settings specific to the Yvva
Runtime Environment that are shared by all Yvva components, namely the
mod_yyva
Apache module, theyvvad
daemon and theyvva
command line shell. /etc/tshs.conf
:- This configuration file defines a number of maintenance tasks performed by
the
tshs
background service.
List of relevant log files¶
In order to debug and analyse problems with the Host Server configuration, there are several log files that you should consult:
/var/log/td-hostserver.log
:The log file for the Yvva Application Server module which provides the web-based Host Server Administration Console and API. Consult this log file when you have issues with associating the Host Server with the Registration Server, errors when issuing API requests or problems with the Administration Console. You can increase the amount of logging by changing the Yvva setting
log-level
fromerror
totrace
ordebug
in/etc/httpd/conf.d/td-hostserver.httpd.conf
:<Location /yvva> SetHandler yvva-handler YvvaSet root-path=/opt/teamdrive/hostserver YvvaSet mysql-cnf-file=/etc/td-hostserver.my.cnf YvvaSet log-file=/var/log/td-hostserver.log YvvaSet log-level=error </Location>
After changing these values, you need to restart the Apache HTTP Server service using
service httpd restart
.This log file is also used by the
td-hostserver
background task. Check this one to verify that background tasks are being processed without errors. The log file location can be configured by changing the file name passed to thelog-file
option in the configuration file/etc/td-hosting.conf
. The log level can be increased by changing the default valueerror
for thelog-level
option totrace
ordebug
. Changing these values requires a restart of thetd-hostserver
background process usingservice td-hostserver restart
./var/log/mod_pspace.log
:- This log file contains error messages related to the
mod_pspace
Apache module, particularly when using an compatible object store or TSHS. It needs to be writable by the user that the Apache HTTP Server runs under (apache
by default). The log file location is configured by the server settingModuleLogFile
and the amount of logging can be changed by adjusting the server settingModuleLogLevel
via the Host Server Administration Console. The value defines the maximum level of logging of messages logged: 1 = Error, 2 = Warning, 3 = Notice, 4 = Trace, 5 = Debug. Changing these values requires restarting the Apache HTTP Server. /var/log/httpd/
:- The Apache HTTP Server’s log files (e.g.
error_log
) might also contain additional relevant error messages (e.g. frommod_pspace
) that should be checked. The amount of logging is affected by theModuleLogLevel
setting described above. /var/log/tshs.log
:- This log file contains errors and other messages generated by the
tshs
background service. The log file location and amount of output are defined in file/etc/tshs.conf
, via the optionslog-file
andlog-level
. Possible values in the order of verbosity areprotocol
,error
,warning
,trace
,debug
. The default iswarning
. /var/log/s3d.log
:- This log file is written by the TeamDrive S3 daemon
s3d
and provides log messages and errors specific to thes3d
background service. The log file location is defined in the init script/etc/init.d/s3d
.
Enable Logging with Syslog¶
As outlined in List of relevant log files, the TeamDrive Host Server logs critical errors and other notable events in various log files by default.
Starting with Host Server version 3.5 and Yvva 1.2, it is now possible to
redirect the log output of some server components to a local syslog
instance as well.
Note
Please note that other components of the TeamDrive Host Server, e.g.
mod_pspace
, s3d
or tshs
currently do not provide syslog support.
This limitation may be lifted in future versions of the TeamDrive Host
Server software.
Syslog support is an essential feature for auditing, security and/or compliance reasons, as it allows you to funnel all log messages into a centralized syslog server.
This makes it easier to monitor the logs for critical events or errors and prevents tampering with the log files in case of a security breach. It also helps to maintain control over the disk space utilization on the server, as growing log files can’t accidentally fill up the file system.
To enable syslog support, the log file name in the log-file
setting has to
be replaced with the keyword syslog
. Optionally, a custom process
identifier can be supplied, by appending it to the syslog
keyword, using a
colon as the separator, e.g. log-file=syslog:my_process_identifier
. If not
used, the default process identifier will be used, which is the name of the
program executable.
To enable syslog support for the Yvva-based td-hostserver
backround
service, edit the log-file
setting in file /etc/td-hosting.conf
as
follows:
log-file=syslog:td-hostserver
You need to restart the td-hostserver
background service via service
td-hostserver restart
in order to activate this change. If the log-level
is set to debug
you will now see log messages appearing in
/var/log/messages
:
Jun 23 11:57:33 localhost td-hostserver: notice: yvvad startup
Jun 23 11:57:33 localhost td-hostserver: notice: Using config file:
/etc/td-hosting.conf
Jun 23 11:57:33 localhost td-hostserver: notice: No listen port
Jun 23 11:57:33 localhost td-hostserver: notice: yvvad running in repeat 60
(seconds) mode
To enable syslog support for the Host Server API and Administration Console,
edit the YvvaSet log-file
setting in file
/etc/httpd/conf.d/td-hostserver.httpd.conf
:
YvvaSet log-file=syslog
You need to restart the Apache HTTP Server via service httpd restart
in
order to activate this change. If the log-level
is set to debug
you
will now see log messages appearing in /var/log/messages
:
Jun 23 12:06:04 localhost mod_yvva: notice: mod_yvva 1.2.1 (May 21 2015
11:00:12) startup OK
Tracing Client Accesses to a Single Space¶
For debugging issues with a specific Space, it might be useful to enable more verbose tracing of activity between the Host Server and the TeamDrive Clients accessing this Space.
For this purpose, access to that Space can be traced by providing the Space’s
ID to the option watched_space_id
in
/etc/httpd/conf.d/td-hostserver.httpd.conf
as follows:
<Location /primespace>
SetHandler pspace-handler
MySQLCnf /etc/td-hostserver.my.cnf
watched_space_id <space ID>
# Necessary to ignore the extra Range-header
# (see Range-header note in the documentation)
RequestHeader unset Range
</Location>
Restart the Apache HTTP Server with service httpd restart
. Any activity on
the selected Space will now be logged into the log file
/var/log/mod_pspace.log
.
Note
Remove this option and restart the Apache HTTP Server once you’ve finished analyzing the problem, to avoid uncontrolled growth of the log file.
Common errors¶
Web Installation: “500 Internal Server Error”¶
This error can be triggered by several error conditions. Check the log file
/var/log/td-hostserver.log
for details.
Note
If there is no error in the log, then the problem may be that SELinux is still enabled. Please see: Disable SELinux for details.
Some common errors include:
[Error] -12036 (2002): Can't connect to local MySQL server through socket
'/var/lib/mysql/mysql.sock' (25)
[Error] "startup.yv" (80)
The local MySQL Server’s socket file can’t be opened. This could either be a
permission problem, or the MySQL Server is simply not available. Check that
MySQL is actually up and running (e.g. by running service mysqld status
)
and restart it, if necessary. If the error persists, check the MySQL error log
file (usually /var/log/mysqld.log
) for hints.
Similarly, an error like the following one indicates that a remote MySQL Server might not be answering (e.g. because of a firewall rule or because it’s not running):
[Error] -12036 (2003): Can't connect to MySQL server on
'mysql.yourdomain.com' (107)
[Error] "startup.yv" (80)
If you see Access denied
errors like the following one:
[Error] -12036 (1045): Access denied for user 'teamdrive'@'localhost' (using
password: YES)
[Error] "startup.yv" (32)
Either the username or password used to connect to the MySQL Server are wrong.
Double check that the MySQL username and password provided in
/etc/td-hostserver.my.cnf
are correct, e.g. by trying to connect to the
MySQL server using these credentials with the mysql
command line client.
If you see the following error when connecting to a remote MySQL Server:
[Error] -12036 (1130): Host 'regserver.yourdomain.com' is not allowed to
connect to this MySQL server
[Error] "startup.yv" (80)
Check the TeamDrive MySQL user’s privileges on the remote MySQL server, e.g.
by running SHOW GRANTS FOR `teamdrive`@`regserver.yourdomain.com`;
and
make sure that this user is allowed to connect to the MySQL server from the
Registration Server’s host.
Errors When Registering the Host Server¶
If the Host Server Registration fails, check /var/log/td-hostserver.log
on
the Host Server as well as /var/log/td-regserver.log
on the Registration
Server for hints (/var/log/pbt_mod.trace
for Registration Server versions
before version 3.5) . See the Troubleshooting chapter in the Registration
Server Installation Manual for details.
MySQL Errors When Upgrading From an Older Host Server Version¶
If you observe Access denied
or Unknown database
errors from the MySQL
server like the following ones after starting the updated TeamDrive Host
Server using an older MySQL table structure:
[Note] DROP DATABASE pbpg;
[Error] -12036 (1044): Access denied for user 'teamdrive'@'localhost' to
database 'hostapilog'
[Error] "p1setup.pbt" P1Setup:upgradeSettings(328)
[Error] "p1setup.pbt" P1Setup:setupDatabase(14)
[Error] "p1setup.pbt" (506)
Unknown database
:
[Error] -12036 (1049): Unknown database 'hostapilog'
[Error] "p1setup.pbt" P1Setup:upgradeSettings(328)
[Error] "p1setup.pbt" P1Setup:setupDatabase(14)
[Error] "p1setup.pbt" (506)
[Error] "p1_shared.pbt" (2)
Double check that the hostapilog
database actually exists and that the
teamdrive
user has the required privileges to access it.
Create the database using CREATE DATABASE hostapilog;
and grant the
required privileges using GRANT ALL PRIVILEGES ON `hostapilog`.* TO
'teamdrive'@'localhost';
. Restart the TeamDrive Service again using
service td-hostserver restart
, it should now conclude the schema
conversion.
If you observe a Can't connect to local MySQL server
error like the
following one in /var/log/httpd/error_log
:
[notice] mod_pspace 1.6.17 Loaded; Build May 6 2015 12:42:39;
Crash-Reporting-Disabled
[error] Failed to boot Admin API: MySQL 2002:
Can't connect to local MySQL server through socket
'/var/lib/mysql/mysql.sock' (2)
or in /var/log/td-hostserver.log
:
[Error] -12036 (2002): Can't connect to local MySQL server
through socket '/var/lib/mysql/mysql.sock' (2)
Double check that the MySQL Server is up and running and that the socket
configuration setting in the [mysqld]
group in /etc/my.cnf
matches the
one in /etc/td-hostserver.my.cnf
.
The default value is /var/lib/mysql/mysql.sock
. If the value in my.cnf
is different, e.g. /tmp/mysql.sock
, we suggest to revert back to the
default value there instead of changing it in td-hostserver.my.cnf
(unless
you have an explicit reason to change the default socket path, of course).
Restart MySQL and the TeamDrive Hosting Services after changing this value.
Admin Console: Clicking on “Host” Results in a “500 Internal Server Error”¶
If you observe an error message like the following when clicking on Host in the Host Server Administration Console:
500 Internal Server Error
ERROR -1: TshsMain: void CSDBConn::connect(CSDB.cc:1116) MySQL 1044: Access
denied for user 'teamdrive'@'localhost' to database 'tshs_admin'
Or:
500 Internal Server Error
ERROR -1: TshsMain: void CSDBConn::connect(CSDB.cc:1116) MySQL 1049: Unknown
database 'tshs_admin'
You likely changed the setting TSHSEnabled
to True
, but did not
configure the MySQL settings for accessing the tshs_admin
database in
/etc/td-hostserver.my.cnf
.
If you changed the setting by accident, simply set TSHSEnabled
back to
False
.
Otherwise, consult the chapter TeamDrive Scalable Hosting Storage in the Team Drive Host Server Administration Guide for details on how to enable and configure TSHS properly.
“Duplicate key” MySQL errors when updating the database¶
If you observe “Duplicate key” errors in the Traffic
or Owner
tables
when upgrading these to the latest schema version, you first need to manually
remove the duplicates via the MySQL client or another tool like MySQL
Workbench. Older versions of the Host Server database schema did not have
UNIQUE constraints on some columns, which caused the creation of duplicate
entries. For the Traffic
table, this usually only affects older traffic
accounting information that can safely be removed.
Duplicates in the Owner
table are likely caused by user names or email
addresses that refer to the same user account, but using different
capitalization. In this case it helps to cross-reference the affected users
with their information in the Registration Server Database - likely one of
these accounts has not been actively used and can be deleted. Please contact
support@teamdrive.net if you need assistance in resolving these conflicts.
Admin API Error: MySQL 1040: Too many connections¶
On a busy server, you might observe one of the following error messages in the Apache HTTP Server’s error log file from time to time:
[error] Failed to boot Admin API: MySQL 1040: Too many connections
[error] [client xxx.xxx.xxx.xxx] (500)Unknown error 500: Admin API Error:
MySQL 1040: Too many connections
In /var/log/td-hostserver.log
you might observe a similar error:
[Error] -12036 (1040): Too many connections
[Error] "startup.yv" (80)
This error indicates that the number of child processes spawned by the Apache HTTP Server (e.g. when many TeamDrive Clients attempt to connect to the Host Server concurrently), causes the MySQL Server to run out of threads for handling the incoming database connections.
By default, the MySQL Server is configured to accept 151 concurrent
connections. Each Apache child process can establish up to two MySQL
connections (one for mod_pspace
and one for mod_yvva
, depending on
what kind of requests it needs to serve). Therefore, the maximum number of
connections should be adjusted to be at least 1.5 times the maximum number of
child processes spawned by the Apache HTTP Server (defined by the
MaxClients
directive in the Apache HTTP Server configuration file
/etc/httpd/conf/httpd.conf
).
The value can be changed by adding the system variable max_connections
to
the [mysqld]
configuration group in the MySQL Server configuration file
/etc/my.cnf
, e.g.:
[mysqld]
datadir=/var/lib/mysql
max_allowed_packet=4M
max_connections=350
socket=/var/lib/mysql/mysql.sock
user=mysql
You need to either restart the MySQL server in order to apply this change, or change the value at run-time, by running the following SQL statement as the MySQL root user:
mysql> SET GLOBAL max_connections=350;
Keep in mind that increasing the maximum number of connections also increases the memory requirements of the MySQL Server. For more details, please consult the MySQL Server and Apache HTTP Server documentation: