Importing User Accounts via CSV Files

Instead of manually creating individual user accounts via the Administration Console as described in chapter Adding Users Manually, it is possible to import multiple user accounts into the Registration Server’s database from a file containing the account information as a CSV (comma-separated values) list.

The CSV import can be enabled and configured via the Provider Settings located in the CSVIMPORT group. See the TeamDrive Registration Server Reference Guide for more details on these settings.

There are two options on how to upload the CSV file:

  • Upload the import file to a “hot folder”, which can be configured using the CSV_IMPORT_DIR setting. The upload can be performed by an external system, e.g. via rsync, scp or sftp, or via a local cron job (e.g. using wget or any other tool to pull the file from a remote location).
  • Upload the import file manually via the Registration Server Administration Console.

The data import via a hot folder is performed by an Auto Task which polls a directory for files containing CSV data at a defined interval (once every hour by default). See “CSV Import” Task for details.

CSV File Structure

A CSV import can be used to create new users and update existing user accounts.

When a user is created, the setting CLIENT/USER_IDENTIFICATION_METHOD (see USER_IDENTIFICATION_METHOD) specifies how a account will be identified by the user. In other words, the name used during login. This can either by a username or an email address. Your import file must conform to this specification.

If a user already exists, then the import can recognise this fact and update a user record, instead of creating a new one. In this case the setting CSVIMPORT/CSV_IDENTITY_COLUMN (see :ref:``) specifies the import column that uniquely identifies the user.

Note that the value of this column may not change, in order for the update to work. Only set CSV_IDENTITY_COLUMN to a value other than username, if you know that this field is never updated.

The CSV file must contain the following fields, separated by comma or semicolon:

username:This is the a globally unique name for a user account. Usernames are unique over all TeamDrive Registration Servers. A username must be specified if USER_IDENTIFICATION_METHOD or CSV_IDENTITY_COLUMN is set to username. This field may be omitted if USER_IDENTIFICATION_METHOD is set to default or email. In this case you will create a user account which is only identified by an email address. In order to do this, CSV_IDENTITY_COLUMN must be set to a value other than username. Once set, the username may not change. Registration Server version 3.6 will not allow users to be created with usernames that look like email addresses (that contain the “@” character). Such usernames are still allowed as reference to users created by previous versions of the Registration Server. By default, this is also the CSV_IDENTITY_COLUMN column.
email:Registration email address of the user. This value is not optional. The Registration Server ensures that the email is unique per Provider. There may be additional uniqueness contraints imposed by the global settings EmailGloballyUnique and UserEmailUnique. If the Provider setting ISOLATED_EMAIL_SCOPE is set to True then the global settings are ignored (see ISOLATED_EMAIL_SCOPE).
password:A password for the user. If empty, the user can define a password during the initial registration process as described in the Reference Guide. Changes to an existing user password will be ignored.
distributor:The Provider Code of the user’s Provider.
reference:A free text field which can be used to assign an external reference ID (e.g. a cost center). If CSV_IDENTITY_COLUMN is set to this column, then CLIENT/EXT_USER_REFERENCE_UNIQUE should be set to ``True` for the Provider.
department:A free text field which can be used to set a department reference for the user. May be changed, if the provider setting CSVIMPORT/CSV_ALLOW_SET_DEPARTMENT is set to True (which is the default). If CSVIMPORT/DISABLE_MISSING_CSV_USERS is set to True then this field must be identical for all records in the import file (i.e. you must use one import file per Department).
language:Language code of the user. This value may change.
authid:This field is optional. It can contain a unique ID that can be used as an alternative reference. If CSV_IDENTITY_COLUMN is set to this column, then the value in this column will be used to identify the user on update In this case, the value in this column may not change. This is the same ID that is used by an external authentication service such as AD or LDAP. As a result, if the column is used you must be certain that it conforms to the usage of any existing or future external authentication service (this is the value $ldap_user_id_attr referred to in Configuring External Authentication using Microsoft Active Directory / LDAP).

Example file structure (without an authid field):

username;email;password;distributor;reference;department;language
TeamDriveUser1;TD_User1@yourdomain.com;;password1;1234;Int1;EN
TeamDriveUser2;TD_User2@yourdomain.com;;password2;1342;Int2;DE
TeamDriveUser3;TD_User3@yourdomain.com;;password3;1452;Int2;DE

Note

Note that even though the CSV file contains a field to define a user’s provider code, this value is currently not used. Instead, the provider code is defined by the user that uploads the CSV file via the Administration Console or by the directory the file is located in. If you need to upload user accounts for multiple providers, create one file per provider account and upload them separately.

Enable CSV Upload via the Administration Console

You can enable the CSV import functionality via the Administration Console by adding the provider setting CSVIMPORT/CSV_IMPORT_ACTIVE and setting it to True via the Administration Console.

Additionally, the Auto Task “CSV Import” must be enabled, by setting its status to Enabled via the Administration Console (Server Management -> Manage Auto Tasks). Change the frequency to the desired time interval in which this Auto Task should be executed. For testing purposes, it might make sense to set it to a very short frequency (e.g. 1 minute).

Note

The import of a single user requires about 1 second. Make sure that the Auto Task’s Frequency allows enough time for the currently running job to finish before another task is started.

If your list of users does not change frequently, it might make sense to keep the Auto Task disabled and only activate it temporarily, after a new CSV file has been uploaded.

In this mode, the CSV files and result logs are stored in the Registration Server’s database and can be managed via the Administration Console.

To upload your CSV user data manually via the Administration Console, follow the instructions outlined in chapter Adding Users via CSV File Import.

Uploading CSV Files to a Directory

As an alternative to the manual upload via the Administration Console, you can define a directory on the Registration Server that will be scanned for new CSV files periodically.

This so called “hot folder” allows for an automated process to create or disable user accounts by uploading updated CSV files using tools like scp, sftp or rsync from another server. An example directory structure can be created in /var/tmp using the following command (replace XXXX with your provider code):

[root@regserver ~]# install -m 700 -d /var/tmp/csvimport/XXXX/error
[root@regserver ~]# install -m 700 -d /var/tmp/csvimport/XXXX/success
[root@regserver ~]# chown -R apache:apache /var/tmp/csvimport
[root@regserver ~]# tree /var/tmp/csvimport
csvimport/
`-- XXXX
    |-- error
    `-- success

3 directories, 0 files

In addition to activating CSV import via the CSV_IMPORT_ACTIVE setting as outlined above, you need to add and configure the following Provider Settings via the Administration Console:

CSVIMPORT/CSV_USE_FILESYSTEM:
Set this option to True to use a directory on the Registration Server for uploading user account information in a CSV file. You should only enable this setting after you created the required directory structure and updated the following settings accordingly. Changing this setting to True will automatically add the following settings to your Provider Settings.
CSVIMPORT/CSV_UPLOAD_DIR:
This directory is the location for uploading new CSV files that should be processed by the import script (e.g. /var/tmp/csvimport/XXXX/ in the example above). The name must end with a slash. Each provider must to use a different directory. It must be readable and writable for the Linux user that the CSV import job is running under (apache by default).
CSVIMPORT/CSV_SUCCESS_DIR:
This directory contains the log files for successful CSV imports (e.g. /var/tmp/csvimport/XXXX/success in the example above). The name must end with a slash. It must be readable and writable for the Linux user that the CSV import job is running under (apache by default).
CSVIMPORT/CSV_ERROR_DIR:
This directory contains the log files for failed CSV imports (e.g. /var/tmp/csvimport/XXXX/error in the example above). The name must end with a slash. It must be readable and writable for the Linux user that the CSV import job is running under (apache by default).

Now copy the CSV file containing your user accounts into the directory defined in CSV_UPLOAD_DIR (e.g. /var/tmp/csvupload/XXXX in the example above).

Note

Please ensure that the file’s ownership and permissions are set correctly, so that the Auto Task can delete the file after it has been processed.

After the Auto Task has been executed, the file will be imported into the database and processed. Afterwards, you can review the processing status via the Administration Console (Manage Servers -> CSV user imports).

Customizing the CSV Import

The CSV import can be further customized using the following Provider settings:

CSVIMPORT/CSV_ALLOW_SET_DEPARTMENT:
Set this to False, if the department may not be changed by the CSV import of an existing user account.
CSVIMPORT/CSV_IDENTITY_COLUMN:
This setting specifies which field in the CSV file will be used to identify users during the CSV import (options are: username, email, reference and authid)
CSVIMPORT/DISABLE_MISSING_CSV_USERS:
If set to True, any user account not present in the CSV file will be disabled on the Registration Server. In this mode, your CSV user file always needs to contain all active user accounts. In addition, an import file may only contain the users of one Department, if the Department field is used.
CLIENT/USER_IDENTIFICATION_METHOD:
This setting determines how a user account is identified by the user. In other words, what name is used on login to TeamDrive. See USER_IDENTIFICATION_METHOD for more details.
CLIENT/EXT_USER_REFERENCE_UNIQUE:
If you wish to use the reference columns to identify user when updating user accounts during import, then this value must be set to True. See EXT_USER_REFERENCE_UNIQUE for more details).