.. _troubleshooting: Troubleshooting =============== List of relevant configuration files ------------------------------------ ``/etc/httpd/conf.d/td-hostserver.httpd.conf``: The configuration file that loads and enables the TeamDrive Host Server-specific modules for the the Apache http Server: * ``mod_pspace.so``: this Apache module provides the actual Host Server functionality by accepting incoming data from the TeamDrive clients as well as delivering data to other clients upon request. * ``mod_yvva.so``: this Apache module is responsible for providing the web-based Host Server Administration Console as well as the Host Server API interface. ``/etc/logrotate.d/td-hostserver``: This file configures how the log files belonging to the TeamDrive Host Service are being rotated. See the ``logrotate(8)`` manual page for details. ``/etc/td-hosting.conf``: This file defines how the ``td-hostserver`` background service is started using the ``yvvad`` daemon. ``/etc/td-hostserver.my.cnf``: This configuration file defines the MySQL credentials used to access the ``pspace`` MySQL database. It is read by the Apache modules ``mod_yvva`` and ``mod_pspace`` as well as the ``yvvad`` daemon that runs the ``td-hostserver`` background tasks and the ``yvva`` command line client. ``/etc/yvva.conf``: This configuration file contains configuration settings specific to the Yvva Runtime Environment that are shared by all Yvva components, namely the ``mod_yyva`` Apache module, the ``yvvad`` daemon and the ``yvva`` command line shell. ``/etc/tshs.conf``: This configuration file defines a number of maintenance tasks performed by the ``tshs`` background service. List of relevant log files -------------------------- In order to debug and analyse problems with the Host Server configuration, there are several log files that you should consult: ``/var/log/mod_yvva.log``: The log file for the Yvva Application Server module which provides the web-based Host Server Administration Console and API. Consult this log file when you have issues with associating the Host Server with the Registration Server, errors when issuing API requests or problems with the Administration Console. You can increase the amount of logging by changing the Yvva setting ``log-level`` from ``error`` to ``trace`` or ``debug`` in ``/etc/httpd/conf.d/td-hostserver.httpd.conf``:: SetHandler yvva-handler YvvaSet root-path=/opt/teamdrive/hostserver YvvaSet mysql-cnf-file=/etc/td-hostserver.my.cnf YvvaSet log-level=error After changing these values, you need to restart the Apache http Server service using ``service httpd restart``. ``/var/log/td-hostserver.log``: The log file for the ``td-hostserver`` background task. Check this one to verify that background tasks are being processed without errors. The log file location can be configured by changing the file name passed to the ``log-file`` option in the configuration file ``/etc/td-hosting.conf``. The log level can be increased by changing the default value ``error`` for the ``log-level`` option to ``trace`` or ``debug``. Changing these values requires a restart of the ``td-hostserver`` background process using ``service td-hostserver restart``. ``/var/log/mod_pspace.log``: This log file contains error messages related to the ``mod_pspace`` Apache module, particularly when using TSHS. It needs to be writable by the user that the Apache http Server runs under (``apache`` by default). The log file location is configured by the server setting ``ModuleLogFile`` and the amount of logging can be changed by adjusting the server setting ``ModuleLogLevel`` via the Host Server Administration Console. The value defines the maximum level of logging of messages logged: 1 = Protocol, 2 = Error, 3 = Warning, 4 = Trace, 5 = Debug. Changing these value requires restarting the Apache http Server. ``/var/log/httpd/``: The Apache httpd Server's log files (e.g. ``error_log``) might also contain additional relevant error messages (e.g. from ``mod_pspace``) that should be checked. The amount of logging is affected by the ``ModuleLogLevel`` setting described above. ``/var/log/tshs.log``: This log file contains errors and other messages generated by the ``tshs`` background service. The log file location and amount of output are defined in file ``/etc/tshs.conf``, via the options ``log-file`` and ``log-level``. Possible values in the order of verbosity are ``protocol``, ``error``, ``warning``, ``trace``, ``debug``. The default is ``warning``. ``/var/log/s3d.log``: This log file is written by the TeamDrive S3 daemon ``s3d`` and provides log messages and errors specific to the ``s3d`` background service. The log file location is defined in the init script ``/etc/init.d/s3d``. Common errors ------------- Errors When Registering the Host Server ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ If the Host Server Registration fails, check ``/var/log/mod_yvva.log`` on the Host Server as well as ``/var/log/pbt_mod.trace`` on the Registration Server for hints. See the Troubleshooting chapter in the Registration Server Installation Manual for details. MySQL Errors When Upgrading From an Older Host Server Version ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ If you observe ``Access denied`` or ``Unknown database`` errors from the MySQL server like the following ones after starting the updated TeamDrive Host Server using an older MySQL table structure:: 140618 10:56:37 [Note] DROP DATABASE pbpg; 140618 10:56:37 [Error] -12036 (1044): Access denied for user 'teamdrive'@'localhost' to database 'hostapilog' 140618 10:56:37 [Error] "p1setup.pbt" P1Setup:upgradeSettings(328) 140618 10:56:37 [Error] "p1setup.pbt" P1Setup:setupDatabase(14) 140618 10:56:37 [Error] "p1setup.pbt" (506) ``Unknown database``:: 140618 10:24:31 [Error] -12036 (1049): Unknown database 'hostapilog' 140618 10:24:31 [Error] "p1setup.pbt" P1Setup:upgradeSettings(328) 140618 10:24:31 [Error] "p1setup.pbt" P1Setup:setupDatabase(14) 140618 10:24:31 [Error] "p1setup.pbt" (506) 140618 10:24:31 [Error] "p1_shared.pbt" (2) Double check that the ``hostapilog`` database actually exists and that the ``teamdrive`` user has the required privileges to access it. Create the database using ``CREATE DATABASE hostapilog;`` and grant the required privileges using ``GRANT ALL PRIVILEGES ON `hostapilog`.* TO 'teamdrive'@'localhost';``. Restart the TeamDrive Service again using ``service td-hostserver restart``, it should now conclude the schema conversion. If you observe a ``Can't connect to local MySQL server`` error like the following one in ``/var/log/httpd/error_log``:: [Thu Jun 19 16:20:50 2014] [notice] mod_pspace 1.5.04 Loaded; Build Jun 19 2014 13:24:58;Crash-Reporting-Disabled [Thu Jun 19 16:20:50 2014] [error] Failed to boot Admin API: MySQL 2002: Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2) or in ``/var/log/td-hostserver.log``:: 140619 16:42:06 [Error] -12036 (2002): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2) Double check that the MySQL Server is up and running and that the ``socket`` configuration setting in the ``[mysqld]`` group in ``/etc/my.cnf`` matches the one in ``/etc/td-hostserver.my.cnf``. The default value is ``/var/lib/mysql/mysql.sock``. If the value in ``my.cnf`` is different, e.g. ``/tmp/mysql.sock``, we suggest to revert back to the default value there instead of changing it in ``td-hostserver.my.cnf`` (unless you have an explicit reason to change the default socket path, of course). Restart MySQL and the TeamDrive Hosting Services after changing this value. Admin Console: Clicking on "Host" Results in a "500 Internal Server Error" ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ If you observe an error message like the following when clicking on **Host** in the Host Server Administration Console:: 500 Internal Server Error ERROR -1: TshsMain: void CSDBConn::connect(CSDB.cc:1116) MySQL 1044: Access denied for user 'teamdrive'@'localhost' to database 'tshs_admin' Or:: 500 Internal Server Error ERROR -1: TshsMain: void CSDBConn::connect(CSDB.cc:1116) MySQL 1049: Unknown database 'tshs_admin' You likely changed the setting ``TSHSEnabled`` to ``True``, but did not configure the MySQL settings for accessing the ``tshs_admin`` database in ``/etc/td-hostserver.my.cnf``. If you changed the setting by accident, simply set ``TSHSEnabled`` back to ``False``. Otherwise, consult the chapter *TeamDrive Scalable Hosting Storage* in the Team Drive Host Server Administration Guide for details on how to enable and configure TSHS properly. "Duplicate key" MySQL errors when updating the database ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ If you observe "Duplicate key" errors in the ``Traffic`` or ``Owner`` tables when upgrading these to the latest schema version, you first need to manually remove the duplicates via the MySQL client or another tool like MySQL Workbench. Older versions of the Host Server database schema did not have UNIQUE constraints on some columns, which caused the creation of duplicate entries. For the ``Traffic`` table, this usually only affects older traffic accounting information that can safely be removed. Duplicates in the ``Owner`` table are likely caused by user names or email addresses that refer to the same user account, but using different capitalization. In this case it helps to cross-reference the affected users with their information in the Registration Server Database - likely one of these accounts has not been actively used and can be deleted. Please contact support@teamdrive.net if you need assistance in resolving these conflicts. Admin API Error: MySQL 1040: Too many connections ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ On a busy server, you might observe one of the following error messages in the Apache http Server's error log file from time to time:: [error] Failed to boot Admin API: MySQL 1040: Too many connections [error] [client xxx.xxx.xxx.xxx] (500)Unknown error 500: Admin API Error: MySQL 1040: Too many connections In ``/var/log/mod_yvva.log`` you might observe a similar error:: 141107 12:11:44 [Error] -12036 (1040): Too many connections 141107 12:11:44 [Error] "startup.yv" (80) This error indicates that the number of child processes spawned by the Apache http Server (e.g. when many TeamDrive Clients attempt to connect to the Host Server concurrently), causes the MySQL Server to run out of threads for handling the incoming database connections. By default, the MySQL Server is configured to accept 151 concurrent connections. Each Apache child process can establish up to two MySQL connections (one for ``mod_pspace`` and one for ``mod_yvva``, depending on what kind of requests it needs to serve). Therefore, the maximum number of connections should be adjusted to be at least 1.5 times the maximum number of child processes spawned by the Apache http Server (defined by the ``MaxClients`` directive in the Apache http Server configuration file ``/etc/httpd/conf/httpd.conf``). The value can be changed by adding the system variable ``max_connections`` to the ``[mysqld]`` configuration group in the MySQL Server configuration file ``/etc/my.cnf``, e.g.:: [mysqld] datadir=/var/lib/mysql max_allowed_packet=4M max_connections=350 socket=/var/lib/mysql/mysql.sock user=mysql You need to either restart the MySQL server in order to apply this change, or change the value at run-time, by running the following SQL statement as the MySQL root user:: mysql> SET GLOBAL max_connections=350; Keep in mind that increasing the maximum number of connections also increases the memory requirements of the MySQL Server. For more details, please consult the MySQL Server and Apache http Server documentation: https://dev.mysql.com/doc/refman/5.6/en/too-many-connections.html https://httpd.apache.org/docs/2.2/mod/mpm_common.html#maxclients http://fuscata.com/kb/set-maxclients-apache-prefork