Security


The virtualization technology used in the GPUBox software separates the application and the operating system layer from the pool of GPU devices beneath it. We can easily distinguish the two types of communication occurring within the GPUBox infrastructure: managing and computation. Basically, the first type of communication allows for the management of the GPU devices and prepares the environment, while the other protocol is used strictly for computational purposes. The principal objective of security for GPUBox is to protect and control the access to allocated GPU devices.

The main concept of the security in GPUBox is token-based authentication and authorization. After successfully signing in, a user has to present a valid token to continue working within the GPUBox infrastructure. To keep the users' passwords, tokens, and other important data secure, communication in GPUBox can be encrypted to prevent sniffing or a man-in-the-middle attack. The security protocol can be switched off by the appropriate configuration parameters, but we advise keeping it enabled and only using the non-secure communication for debugging purposes. The communication related to managing the infrastructure is secure, the computation protocol - except for the handshake - is not ciphered. The encryption process of high-volume data would required a huge overhead of CPU cycles.

The capability of logging and reporting information, such as records of the attempts to access the resources, is a necessary tool in modern applications. GPUBox is able to identify and verify the users, the origin of the requests, and which resources were requested by which user. When the request cannot be identified, the threat is detected and immediately reported.

The key features of the GPUBox security include the following:

  • communication is encrypted where it is needed
  • authentication and authorization is token-based
  • users and groups
  • logging and reporting
  • Users and groups

    A person who wants to use any resource from the GPUBox infrastructure must be identified at the first stage before gaining access to the GPUs. Each user is identified by the userid and then verified by a password. At the time of creating a user account, an administrator defines his userid and sets the temporary password. The UserID and password identify and verify a user to the GPUBox infrastructure.

    A user can be assigned to one of two groups:

  • superusers with group ID 0
  • users with group ID 1
  • Regarding the associated group and role, the security of GPUBox distinguishes three types of users:

    Type GroupID Description
    User 1 User can perform the following actions:
  • allocate the GPUs
  • drop the GPUs
  • list the free GPU devices
  • list his own GPU allocations
  • manage his own password and token
  • perform the computation on the allocated GPUs
  • use the gpubox command
  • use GPUBox Console in user mode
  • Superuser 0 An administrator-type account that has access to do all of the actions necessary to maintain the GPUBox infrastructure. This level of access allows a user to create and control the users, as well as manage the GPU devices. Besides the user privileges, superusers can also:
  • create, delete, change user's accounts
  • drop the GPU on behalf of other users
  • list the devices and monitor their parameters
  • list the other users' allocations
  • list and remove processes
  • view the logs of OServer and GPUServers
  • view and change the configuration of OServer and GPUServers
  • maintain the orphans allocations
  • manage Monitor1
  • use the gpubox and agpubox commands
  • use GPUBox Web Console in superuser mode
  • Infrastructure user 0 Has the same privileges as a superuser, but is dedicated to the communication between OServer and GPUServers. The same account can be used for managing the GPUBox infrastructure and for enabling communication, although we highly advise that you use separate accounts.

    The GPUBox security allows for the use of the same account from different systems. Moreover, a user does not have to be a particular individual. The same user ID can be shared by one or more people. However, in terms of security, sharing the same user ID with more individuals can be undesirable, because accountability can be lost in this type of situation.
    The account is disabled after five consecutive failed password attempts. Besides logging and accountability issues, sharing the same account can have a negative impact on the other users. For this reason, using the same account as an infrastructure user and as an administrator can freeze the entire GPUBox infrastructure communication capability as a consequence of a disabled administrator/infrastructure user account. In such a case, to restore the communication, another administrator would have to enable the infrastructure user and restart the GPUServers.
    Sharing accounts has other disadvantages as well; people using the same account will be able to:

  • see other's allocation
  • drop other's GPUs
  • We highly advise using a separate account for each person using the GPUBox infrastructure.

    Each user has several attributes. The administrator can display all of them by issuing the agpubox u|user -l|--list -u|--user=<userid>command:

    $ agpubox u -l -u=mary
    ID                 :2
    UserID             :mary
    Username           :Mary
    Valid from         :2013-05-25 22:26:56
    Valid to           :2999-12-13 00:59:59
    Created            :2013-05-25 22:26:56
    Attempted login     :0
    Last successful login :2013-05-28 19:26:58
    Last failed login  :2013-05-28 13:40:50
    Enabled            :yes
    Group ID           :1
    Max GPU            :8
    Comment            :Created from 203.0.113.152 by GPUBox administrator (gpubox)

    The meaning of each attribute is described in the User's attributes section.

    Tokens

    As a consequence of the authentication process, the user obtains a token. The password given in the -p|--password parameter for the gpubox token command for the gpubox token subcommand enables the user to download the token tied with his account and save it to the Client's configuration file:

  • $HOME/.gpubox with the 600 permissions

    Linux

  • %LOCALAPPDATA%\gpubox.config

    Windows

  • Authentication and authorization mechanisms in the GPUBox infrastructure are based on tokens. After the successful authentication, a token is used to manage and monitor the resources. A security token is a string of 32 printable ASCII characters. We can distinguish between the two kinds of tokens:

    Infrastructure token Used for communication between OServer and GPUServer, defined by the auth_token parameter in the OServer and GPUServer configuration files. The infrastructure token must be the same as the token of one of the users with administrator privileges. Check the Security bootstrapping section to see how to set it correctly.
    User token Used for authorizing regular users and administrators.

    Security bootstrapping

    Within the very first start of OServer, it creates the security database with a single, already enabled superuser with the gpubox UserID, username GPUBox Administrator and password gpubox. The database is newly created this way each time the path to the security database is changed in the oserver_security_plugin configuration parameters or if the indicated file is deleted.

    After the database initialization, the gpubox user will get the same token as an infrastructure token for OServer and GPUServer delivered by installers. The default token is d6e459844365274877ca5949b144f076. This will allow for communication with OServer and GPUServer without any additional security configuration. In order to change the default security behavior, a new infrastructure user needs to be created and an infrastructure token has to be generated once again. You can follow the next few steps to change the default settings.

    1. Create a new infrastructure user.

    It is highly recommended, in terms of security, to create a new infrastructure user to replace the default gpubox in order to prevent unwanted access to an account with administrator credentials. When communication is established, it is a good practice to disable or even delete the default gpubox superuser.

    Create and enable the new superuser (that is meant to be the infrastructure user)
    $ agpubox user --add --userid=infuser -n='Infrastructure User'
    Enter the password: ******
    Retype the password: ******
    Created
    $ agpubox user --modify --userid=infuser --enable
    $ agpubox user --modify --userid=infuser --group=0

    The last statement --group=0 assigns the user to the 0 group, which means that he is granted administrator privileges.

    2. Generate a new infrastructure token

    To change the infrastructure token, you first have to log in as the new superuser, here as the infuser.
    $ agpubox t -u=infuser -o=https://203.0.113.1:8082
    Enter the password: ******
    Configuration file /home/john/.gpubox already exists, overwrite it? [y/n]y
    Login successful

    You can use the agpubox whoami command to verify if the login was successful.

    Home directory /home/john and Linux user’s ID does not have to match the GPUBox userid. In this example, we use infuser for GPUBox infrastructure user and john for the regular Linux user.

    Client's configuration file to which the token is downloaded is by default located in the $HOME/.gpubox file unless an environment variable GPUBOX_CLIENT indicates a different path and filename.
    Generate and display the new 32-byte long token:
    $ agpubox gentoken
    $ cat $HOME/.gpubox | cut -d'|' -f2
    yE1qamoJmIqVIH5S4LDrF5vzuKSnViZK

    3. Change the token in the configuration file of OServer and GPUServer

    Copy the newly generated token to the OServer and GPUServer configuration file as the auth_token parameter.
    auth_token = "yE1qamoJmIqVIH5S4LDrF5vzuKSnViZK”
    To apply the introduced changes, you should:

    4. Verify that OServer and GPUServer are running with the new infrastructure user

    While starting, OServer displays the infrastructure superuser information in the OSVC-IC-69A message. If user is not enabled or the token is invalid, then it displays the OSVC-IC-99A message.
    [2013-04-09 19:10:23.898074] <NOTICE  > OSVC-IC-69A Infrastructure token belongs to userid: infuser, username: Infrastructure User
          
    To check if the new user is applied to GPUServer, find the following messages (with the same origin i.e. the same session ID ) in OServer’s log:
    [2013-07-21 09:24:59.370788] <INFO    > ROSC-HH-500 [0000002] New session [140081675130624]: put:http://203.0.113.1:8081/gpu, IP: 203.0.113.10:41139
    [2013-07-21 09:24:59.371923] <INFO    > ROSC-AX-51A [0000002] Access granted to superuser infuser, Infrastructure User
          
    If GPUServer is not authorized, then you should see this message in GPUServer’s log:
    [2013-04-09 19:50:49.239165] <CRITICAL> GBSC-SC-91Z [10415] Fatal response from OServer, GPUServer is not authorized.
          
    and those in OServer:
    [2013-04-09 19:50:49.237710] <INFO    > ROSC-HH-500 [0000018] New session [140159861380864]: put:https://oserver/gpu, IP:203.0.113.10:59469
    [2013-04-09 19:50:49.238013] <WARNING > QS1C-VT-74A [0000018] Invalid token given
    [2013-04-09 19:50:49.238055] <INFO    > ROSC-CW-500 [0000018] HTTP Status Code 401
          

    5. Create the new superuser

    When new infrastructure user is created, it is a good time to create another custom administrator account to manage users and GPUs. You can repeat Step 1 to create a new superuser.

    6. Disable or delete the default gpubox user

    $ agpubox user --modify --userid=gpubox --disable
    or to delete:
    $ agpubox user --delete --userid=gpubox
    After pressing Enter administrator is asked for confirmation.

    To confirm, an administrator must enter yes. Y or y do not work as a confirmation.

    General Security Options

    Authentication and authorization in the GPUBox infrastructure is served by OServer and controlled by a security plugin. The name of the plugin and the configuration string is passed to OServer by the two parameters:
    Parameter Default value Description
    oserver_security_plugin security1 Name of the security plugin. If not given, security1 will be used.
    oserver_security_config empty Path to security database. OServer will not run properly unless this parameter is defined.

    The same security database indicated by the oserver_security_config parameter can be shared between many OServers.


    The OServer security mechanism in OServer is based on plugin. Current version of GPUBox provides only the security1 plugin.

    Manage Users

    In the following examples, we will use the agpubox command, but the same results can be achieved by using GPUBox Web Console.

    For more details regarding particular commands used in this section, please check agpubox user in the Command Reference.

    Create user

    Every user must be manually enabled. Please see section Modify user.

    Any newly created user has the default settings specified in the User's attributes section.

    In order to create the new user in the GPUBox infrastructure, issue the command with the following structure
    agpubox user|u --add|-a --userid|-u=<userid> --username|-n=<username> [--password|-p=<password>] 
    where:
    --userid|-u=<userid> Basic user indicator that will be used in any command involving a particular user. <userid> must be at least 5 character long, may contain only alphanumeric characters and is case-sensitive. The first character must be a letter.
    --username|-n=<username> For example, consider a name and surname of user. It must be given in quotation marks if it contains spaces. The username must be at least 5 characters long, may contain alphanumeric characters, and is case-sensitive. The first character must be a letter.
    --password|-p=<password> A user’s password is needed to obtain the security token. When not given, the administrator will be prompted to enter it. The password may contain any printable ASCII characters, is case-sensitive, and must be at least 6 characters long.

    The password given in the parameter will be visible in the bash command history, system process list, and other places depending on the system configuration. To avoid that, you can skip the -p|--password parameter and let the agpubox command ask you to enter it.

    Usage examples:

    Add a regular user account and enable it:
    $ agpubox user  --add --userid=user1 --username=”User 1”
    Enter the password: ******
    Retype password: ******
    Created
    $ agpubox u --modify -u=user1 --enable
    Add an administrator account and enable it:
    $ agpubox user  --add --userid=admin1 --username=”Administrator 1”
    Enter the password: ******
    Retype password: ******
    Created
    $ agpubox u --modify -u=admin1 --enable --group=0
    Note that the admin1 user was automatically assigned to group 1 and had to be manually moved to group 0, which provided him with administrator privileges:
    $ agpubox u --list
    +------+------+----------------------+------+------+---------- ... -+
    |UserID|GroupI|Username              |Enable|Max GP|Valid from ...  |
    +------+------+----------------------+------+------+-----------... -+
    |admin1|0     |Administrator 1       |yes   |4     |2013-04-10 ...  |
    |user1 |1     |User 1                |yes   |4     |2013-04-10 ...  |

    User's attributes

    Every user in the GPUBox infrastructure has several attributes that can be listed by the administrator by issuing the agpubox u|user -l|--list -u|--user=<userid>. An exemplary listing below shows the list of the attributes for the user with UserID mary:

    $ agpubox u -l -u=mary
    ID                 :2
    UserID             :mary
    Username           :Mary
    Valid from         :2013-05-25 22:26:56
    Valid to           :2999-12-13 00:59:59
    Created            :2013-05-25 22:26:56
    Attempted login     :0
    Last successful login :2013-05-28 19:26:58
    Last failed login  :2013-05-28 13:40:50
    Enabled            :yes
    Group ID           :1
    Max GPU            :8
    Comment            :Created from 203.0.113.152 by GPUBox administrator (gpubox)

    The following table contains the full list of the user's attributes with descriptions. The Parameter column contains the parameters dedicated to particular attributes that can be used with the agpubox user --modify command. It also specifies whether an attribute was required while creating the user.

    Attribute Required Parameter Description Default value
    ID Unique identification number.
    Generated automatically
    UserID -u|--userid=<userid> The unique user's identification string in the GPUBox infrastructure must be between 5 and 25 character long, may contain only alphanumeric characters, and is case-sensitive. The first character must be a letter. -
    Username -n|--username=<username> The most common usage of this attribute is putting the user's name and surname into it. The string should uniquely identify the user account and be easy to recognize later in logs and reporting tools.
    It must be given in quotation marks if it contains spaces. The username must be at least 5 characters long, may contain alphanumeric characters, and is case-sensitive. The first character must be a letter.
    -
    Password -p|--password=[username] A user password is needed to obtain the security token. If it was not given during the creation of the account, an administrator will be prompted to enter it. The password can be changed by the user with the gpubox p|passwd command. The password may contain any printable ASCII characters, is case-sensitive, and must be at least 6 characters long. -
    Valid from -f|--validfrom=<"YYYY-MM-DD hh:dd:mm"> Timestamp indicating when the user's account became valid. Timestamp from when the account was created
    Valid to -t|--validto=<"YYYY-MM-DD hh:dd:mm"> Timestamp indicating when the user's account is valid until. 2999-12-12 23:59:59
    Created Timestamp indicating when the user's account was created. Timestamp of when the account was created
    Attempted login Number of incorrectly entered passwords.
  • It is not visible to the user.
  • It is zeroized after every successful login.
  • After 6 consecutive failed password entry attempts, the account is disabled.
  • 0
    Last successful login The last date and time when the user successfully signed in. 0001-01-01 01:01:01
    Last failed login The last date and time when user inserted an incorrect password. 0001-01-01 01:01:01
    Enabled -e|enable or -d|disable Yes when user is enabled and the account is valid No otherwise.
    This attribute has to be manually changed to enable the user.
    No
    Group ID -g|--group=<groupid> A number indicating to which group the user belongs and what role he plays in the GPUBox infrastructure.
    0 - superuser
    1 - regular user
    1
    Max GPU -x|--maxgpu=<max_gpu_limit> The maximum number of the GPUs that the user can concurrently have allocated. 4
    Comment -c|--comment=<comment> Any comment string related to the user's account. The comment can be changed at anytime.
    If the comment contains spaces, then it must be given in quotation marks.
    When the user is created, it has a default comment referring to which administrator created the account. The default comment will contain:
  • IP address from which the user's account was created
  • the administrator's name
  • the administrator's userid

  • Example:
    Created from 203.0.113.151 by Bob (bob).

    Comments will be visible only to the administrators.

    Created from [CLIENT_IP] by [USERNAME] ([USERID])

    Modify user

    User's role, limits, and access to the GPUBox infrastructure can be controlled by modifying the User's attributes. The --modify parameter to the subcommand agpubox user allows the administrator to change up to eight user attributes in one command by using the appropriate parameters:
    agpubox user|u  --modify|-m --user|-u=<userid>
                    [--username|-n=<username>]
                    [--enable|-e|--disable|-d]
                    [--group|-g=<groupid>]
                    [--password|-p=<password>]
                    [--comment|-c=<comment>]
                    [--maxgpu|-x=<max_gpu_limit>]
                    [--validfrom|-f=<"YYYY-MM-DD hh:mm:ss">]
                    [--validto|-t=<"YYYY-MM-DD hh:mm:ss">]

    UserID cannot be changed. It has to be given in the -u|--userid parameter for the $ agpubox user --modify command every time in order to indicate the user that will be modified.

    The --enable|-e and --disable|-d parameters switch the user's ability to log in to GPUBox infrastructure. Both options are mutually exclusive. By default, every newly created user is disabled and must be enabled with --enable|-e parameter!

    The password given in the parameter will be visible in the bash command history, system process list, and other places depending on the system configuration. To avoid that, you can skip the -p|--password parameter and let the agpubox command ask you to enter it.

    Usage examples:

    Enable user with UserID bob:
    $ agpubox u -m -u=bob -e
    Enable user with UserID bob, change his password to passwd2 and set his maximum number of GPUs to 6:
    $ agpubox u -m -u=bob -e -p=passwd2 -x=6
    Set timestamp defining the activation of account for user with UserID bob to February 1st 2013, 5:00 PM:
    $ agpubox u -m -u=bob -f=”2013-02-01 17:00”
    Define the period from February 1st 2013, 5:00PM to February 7th 2013, 5:00PM when the account is active for user with UserID bob:
    $ agpubox u -m -u=bob -f=”2013-02-01 17:00” -t=”2013-02-07 17:00”

    Delete users

    To permanently delete a user from the security database, issue the command:
    agpubox user|u --delete|-d --userid|-u=
    For example, to delete user with UserID john the following command can be used:
    $ agpubox u --delete -u=john
    Are you sure you want to permanently delete admin1[yes/no]
    yes
    User deleted
    For confirmation, administrator has to enter yes. Y and y do not work as confirmation.

    User with GPUs currently allocated cannot be deleted. In order to delete such a user an administrator has to use the agpubox udrop subcommand to drop the allocations on behalf of the user.

    Logging and Reporting

    Administrator can retrieve information about user’s activity and GPU allocation in a few ways: The first three bullets are discussed in detail in the referenced chapters. Here we will focus on the security aspects of OServer and GPUServer log. In order to retrieve the following messages, the oserver_log_level configuration parameter must be set to TRACE.

    OServer log

    During start-up, OServer displays a few messages information regarding security.

    Security configuration

    For the security1 plugin, the oserver_security_config parameter indicates the security database.
    <INFO    > OSVC-IC-50D oserver_security_plugin = security1
    <INFO    > OSVC-IC-50K oserver_security_config = /usr/local/gpubox-oserver/data/oserver_security.db
    Security was initialized and the configuration string was successfully passed.
    <INFO    > OSVC-PS-500 Plugin 'security1' loaded.
    <INFO    > OSVC-PS-510 Security plugin received configuration.
    The security database was initialized from scratch.
    <NOTICE  > QSEC-RL-64A [FFFFFFF] Security database initialized with user "gpubox".
    The infrastructure token in auth_token was given:
    <INFO    > OSVC-IC-50H Infrastructure token presented
    The OSVC-IC-69A message displays to whom the infrastructure token belongs:
    <NOTICE  > OSVC-IC-69A Infrastructure token belongs to UserID: gpubox, username: GPUBox administrator

    Request begin

    Every new request to OServer is reported by:
    <INFO    > ROSC-HH-500 [0000001] New session [139768459052800]: put:https://oserver/gpu, IP: 203.0.113.11:55530
    where:
    [0000001] is request ID
    [139768459052800] is a thread ID, each new request gets a unique thread ID
    put:https://oserver/gpu the PUT method was used to upload the GPU configuration from GPUServer
    IP:203.0.113.11:55530 the remote IP and port of the requester

    By using the above request we can easily identify the GPUServer start sequence. This RESTful request uploads the GPU configuration from GPUServer.

    Who requested

    <INFO    > ROSC-AX-51B [000004A] Access granted to bob, Bob
    <INFO    > ROSC-AX-51A [000008D] Access granted to superuser gpubox, GPUBox administrator
    Every request from superuser is reported by ROSC-AX-51B for regular users or ROSC-AX-51A for superusers.

    Request end

    Every finished request is reported by the ROSC-CW-500 message:
    <INFO    > ROSC-HH-500 [0000042] New session [140385032337152]: get:https://oserver/auth/login/bob, IP: 203.0.113.150:45978
    <INFO    > ROSC-CW-500 [0000042] HTTP Status Code 200
    ...
    <INFO    > ROSC-HH-500 [0000043] New session [140385021847296]: get:https://oserver/auth/bob, IP: 203.0.113.150:45979
    <INFO    > ROSC-AX-510 [0000043] Access granted to bob, Bob
    <INFO    > ROSC-CW-500 [0000043] HTTP Status Code 400
    An invalid authorization is reported by the ROSC-CW-500 [0000043] HTTP Status Code 400 message. In the above example Bob was trying to use the agpubox command but access was denied. Bob has to be a superuser (i.e. has to be assigned to the 0 group) to use the agpubox command.

    GPUServer log

    Security configuration

    Infrastructure token in auth_token was given:
    <INFO    > GBSC-GI-51H [18126] Infrastructure token presented

    Request begin

    Only superusers with a valid infrastructure token can connect to the GPUServer RESTful interface.

    Every new request from OServer is reported by:

    <INFO    > GBRS-CH-50A [R:0000001:18129] Connection from:203.0.113.1:43553, GET:/process/list
    where:
    [R:0000001:18129] request ID and process id, R means RESTful request
    IP:203.0.113.1:43553 the remote IP and port of the requester
    GET:/process/list OServer used the GET method to retrieve the process list
    <INFO    > RGBC-WS-500 [R:0000001:18129] HTTP Status Code 200