GPUServer

GPUServer is responsible for virtualizing the physical GPU devices and provisioning them to the GPUBox Infrastructure. OServer authorizes and redirects the user's request to a specific GPUServer.

Starting GPUServer on Linux

Depending on the used type of installation, you can start GPUServer in one of the two ways:

  • As a system service

    # service gpuserver start

    If you run the installation with the root privileges, the daemon script and the designated gpubox user are by default installed automatically. See the GPUServer Installation section for more details.

  • As a user process

    When the GPUSERVER_CONF environment variable indicates the configuration file or the GPUServer uses the default <GPUSERVER_INSTALLATION_DIR>/etc/gpuserver.conf location, to start GPUServer, you can simply issue a command:
     $ gpuserver
    otherwise, you have to pass the configuration file using the -c parameter, for example:
     # gpuserver -c /usr/local/gpubox-gpuserver/etc/gpuserver.conf
    To daemonize GPUServer, use the -d option:
     $ gpuserver -d

Parameters

gpuserver parameters
gpuserver -c path_to_config_file -d -u user -v -h
-c path_to_config_file relative or absolute path to configuration file
-d daemonize GPUServer
-v show version
-u user start GPUServer as an indicated system user (root privileges required)
-h show help

Environment variables

The following environment variables can affect GPUServer:
GPUSERVER_CONF Path to the configuration file. The configuration file given as a command line parameter has precedence over this environment variable.
GPUBOX_IBDEV Variable is eligible for both GPUServer and Client.
If more than one InifniBand adapter is installed, then set the name of the required device. For example, GPUBOX_IBDEV=mlx4_0.
The gpuserver_infiniband_device parameter in the GPUServer configuration has precedence over this environment variable.
GPUBOX_IBPORTS The variable is eligible for both GPUServer and Client.
If the InifniBand adapter has more than one port, the variable sets which port is used by GPUServer and Client. The value of this environment variable must meet the following requirements:

  • First port has number 1.
  • More ports can be specified in array.
  • When more ports are specified, threads use the ports in a round-robin like algorithm.
  • For example:

  • GPUBOX_IBPORTS=1
  • GPUBOX_IBPORTS=[1,2]
  • The gpuserver_infiniband_ports parameter in the GPUServer configuration has precedence over this environment variable.

    Return codes

    0OK
    1 Error related to:
  • cannot find configuration file
  • configuration parameter parsing
  • missing required parameters
  • GPUServer with -d option could not switch to daemon mode/&l
  • More detailed messages are also displayed in log.
    2Forced to quit, the SIGQUIT signal was received
    3GPUServer caught an exception
    4Exception generated by signals such as SIGSEGV or SIGFPE

    Configuration

    All of the parameter values, except numbers, must be enclosed in quotation marks. Only the value of the gpuserver_bind_port parameter is a number.

    GPUServer’s configuration file is located under the GPUSERVER_CONFenvironment variable. By default it is <GPUSERVER_INSTALLATION_DIR>/etc/gpuserver.conf. The exemplary configuration file is presented below:

    gpuserver_bind_port=9393
    gpuserver_bind_ip="203.0.113.11"
    gpuserver_rest_bind="203.0.113.11:8080"
    gpuserver_rest_oserver_address="http://203.0.113.1:8081"
    auth_token="d6e459844365274877ca5949b144f076"
    gpuserver_log_path="/usr/local/gpubox-gpuserver/log/gpuserver.log"
    gpuserver_log_level="TRACE"
    gpuserver_install_dir="/usr/local/gpubox-gpuserver"
    gpuserver_gpus="all"
    gpuserver_infiniband_enabled="no"
    gpuserver_infiniband_device="mlx4_0"
    

    Functions of particular parameters are described in the table below.

    Parameter ( - required) Default value Description
    gpuserver_bind_port 9393 Number of the TCP port used by Client to connect to GPUServer.
    gpuserver_bind_ip none Required address that must be bound on a specific interface (cannot be 0.0.0.0).
    gpuserver_rest_bind 127.0.0.1:8080 Secondary server bind interface and port. GPUServer's RESTful interface is listening on this port.
    gpuserver_rest_oserver_address https://127.0.0.1:8082 HTTP/HTTPS address of OServer. For example:http://203.0.113.1:8081 or https://203.0.113.1:8082
    auth_token d6e45984436527
    4877ca5949b144f076
    The authorization token used to connect to OServer. Must be the same as the one set in the OServer configuration.
    gpuserver_log_path STDOUT Path to the GPUServer log file. If the path to the file is not given, then the parameter will be set to STDOUT and messages will be logged to the standard output.
    gpuserver_log_level TRACE Parameter referring to the logging level. It can be chosen from one of the following:TRACE,DEBUG,INFO,NOTICE,WARNING,ERROR,CRITICAL, where TRACE is the most detailed and CRITICAL gives only the most important entries.
    gpuserver_gpus all Number of GPUs used by GPUServer. The default value all means that every compatible GPU found in the machine will be used.
    gpuserver_infiniband_enabled no When set to yes, it enables the InfiniBand communication.
    gpuserver_infiniband_device none Set the InfiniBand device that the GPUServer will use. By default, the GPUServer always uses the first available device. The parameter can be useful when more than one InfiniBand adapter is installed. The parameter has precedence over the GPUBOX_IBDEV environment variable.
    gpuserver_infiniband_ports 1 Set the InfiniBand port that the GPUServer will use. By default, the GPUServer always uses the first port. When InfiniBand device has more than single port, GPUBox can utilize all of them in round-robin like algorithm. Administrator has to specify:
    gpuserver_infiniband_ports=[1,2]
    then each new thread will use next port i.e. 1,2,1,2,...
    The parameter has precedence over the GPUBOX_IBPORTS environment variable.

    Use variables with configuration parameters

    Any non-array string value can be replaced by the environment variable.

    variable="$environment-variable"    
    

    As an example, GPUServer's IP addresses can be replaced by the variable.
    Let us set the environment variable in the /etc/rc.local file.

    # echo "export IB0_ADDR=\"`ip a s dev ib0 | grep -oP 'inet\s+\K[^/]+'`:8081,8082s\"" >> /etc/rc.local
    # echo "export GPUSERVER_REST_BIND="$IB0_ADDR:8080" >> /etc/rc.local
    

    The variables are available in OServer's start script:

    gpuserver_bind_ip="$GPUSERVER_BIND"
    gpuserver_rest_bind="$GPUSERVER_REST_BIND"
    

    GPUServer's behavior when configuration parameters are invalid

    The following table describes GPUServer's behavior when parameter's syntax is correct but semantics are invalid.

    Parameter Description
    gpuserver_bind_port Only values from range of 1..65535 are valid.
    [2013-07-21 13:49:17.006179] <ERROR   > GBSC-GI-81B [P:680] Invalid port number for parameter 'gpuserver_bind_port': 65536
    GPUServer is terminated.
    gpuserver_bind_ip Specific interface address is required, 0.0.0.0 is invalid.
    [2013-07-21 14:21:39.360384] <ERROR   > GBSC-SG-80C [P:12046] Config parameter 'gpuserver_bind_ip' cannot be 0.0.0.0, current value :0.0.0.0
    [2013-07-21 14:21:39.360436] <ERROR   > GBSC-GI-82I [P:12046] 'gpuserver_rest_bind' has invalid format, must be bound on specific IP address
    [2013-07-21 14:21:39.360446] <CRITICAL> GBSC-SG-99J [P:12046] GPUServer cannot be started
    GPUServer is terminated.
    gpuserver_rest_bind

    When the port is busy, the interface does not exists or the entire string is incorrect, the OSVC-MN-84A message will explain the details.

    [2013-07-21 14:33:46.027928] <ERROR   > OSVC-MN-84A  set_ports_option: cannot bind to 203.0.113.10:8082s: Address already in use
    [2013-07-21 14:33:46.028030] <CRITICAL> GBSC-SR-91A [R:17148] Cannot start RESTful service

    or, when the IP address is invalid:

    [2013-07-21 14:25:00.186375] <ERROR   > GBSC-GI-82I [P:12815] 'gpuserver_rest_bind' has invalid format, must be bound on specific IP address
    [2013-07-21 14:25:00.186387] <CRITICAL> GBSC-SG-99J [P:12815] GPUServer cannot be started
    ...
    [2013-07-21 15:25:00.187524] <CRITICAL> GBSC-SG-98X [P:16051] Cannot continue without RESTful service
    [2013-07-21 15:31:22.747656] <DEBUG   > GBSC-SR-30A [R:16052] Starting RESTful service on:203.0.113.401:8080
    [2013-07-21 15:31:22.750092] <ERROR   > OSVC-MN-84A  set_ports_option: cannot bind to 203.0.113.401:8080s: Cannot assign requested address
    [2013-07-21 15:31:22.750192] <CRITICAL> GBSC-SR-91A [R:16052] Cannot start RESTful service
    ...
    [2013-07-21 15:31:23.750524] <CRITICAL> GBSC-SG-98X [P:16051] Cannot continue without RESTful service
    GPUServer is terminated.
    gpuserver_rest_oserver_address If there is no OServer working under an indicated address, GPUServer will keep on trying to send its GPUs configuration
    [2013-07-30 09:43:39.960629]  GBSC-SC-71A [C:5932] Cannot send initial GPUs configuration, retrying every 5s ...
    GPUServer is terminated.
    gpuserver_log_path

    In case of an invalid path, the log is always redirected to the standard output.

    [2013-07-21 16:26:43.106917] <WARNING > GBSC-SL-71A [P:4073] Cannot open output file /user/local/gpubox-gpuserver/data/gpuserver.log logging continue to standard output
    
    GPUServer continues processing.
    gpuserver_gpus

    GPUServer uses the default value.

    [2013-07-21 16:36:08.531314] <INFO    > GBSC-GI-52F [P:9447] Type of 'gpuserver_gpus' not supported , all GPUs will be used
    GPUServer continues processing.
    gpuserver_infiniband_device If the InfiniBand device is invalid, GPUServer will continue the communication over the TCP protocol.
    [2013-07-21 16:33:55.145340] <CRITICAL> IBCC-TC-96B [8476] Cannot find IB device (mthca5)
    GPUServer continues processing.
    gpuserver_infiniband_ports

    In case of an invalid value, the first port will be used.

    [2013-07-21 16:36:08.531252] <WARNING > GBSC-GI-72K [P:9447] Type of 'gpuserver_infiniband_ports' not supported, first port will be used
    GPUServer continues processing.
    gpuserver_infiniband_enabled

    GPUServer uses the default value.

    [2013-07-30 10:04:49.403343]  GBSC-GI-71I [P:7431] Configuration parameter 'gpuserver_infiniband_enabled' is invalid, default value used: yes
    GPUServer continues processing.
    gpuserver_log_level

    GPUServer uses the default value.

    [2013-02-05 07:19:14.302740] <INFO    > GBSC-SL-50C [P:14407] Logging started
    [2013-02-05 07:19:14.302749] <NOTICE  > GBSC-SL-68A [P:14407] Invalid log level, 'TRACE' will be used instead
    [2013-02-05 07:19:14.302756] <INFO    > GBSC-SL-50A [P:14407] gpuserver_log_path  = STDOUT
    [2013-02-05 07:19:14.302762] <INFO    > GBSC-SL-50B [P:14407] gpuserver_log_level = TRACE
    
    GPUServer continues processing.
    auth_token

    GPUServer uses the value passed in the configuration file.

    [2013-07-30 09:41:18.491173] <CRITICAL> GBSC-SC-91Z [C:5863] Fatal response from OServer, GPUServer is unauthorized
    GPUServer continues processing.

    Start sequence

    While GPUServer is starting it:
    • starts primary server
    • reads configuration
    • forks configuration process
      • sends the GPU configuration to OServer
    • forks secondary server

    Messages at start

    At start, GPUServer displays the values of all of the the configuration parameters, except auth_token

    [2013-04-16 13:01:48.234944] <INFO    > GBSC-CF-51A [P:19242] Configuration file proceeds from environment variable GPUSERVER_CONF: /usr/local/gpubox-gpuserver/etc/gpuserver.conf
    [2013-04-16 13:01:48.235519] <WARNING > GBSC-SL-70A [P:19242] Parameter 'gpuserver_log_path ' not found in configuration file, default value used: STDOUT
    [2013-04-16 13:01:48.235555] <INFO    > GBSC-SL-50C [P:19242] Logging started
    [2013-04-16 13:01:48.235584] <INFO    > GBSC-SL-50A [P:19242] gpuserver_log_path  = STDOUT
    [2013-04-16 13:01:48.235610] <INFO    > GBSC-SL-50B [P:19242] gpuserver_log_level = TRACE
    [2013-04-16 13:01:49.235624] <INFO    > GBSC-CF-51B [P:19242] Configuration file: /tmp/gpuserver.log
    [2013-04-16 13:01:48.235639] <INFO    > GBSC-GI-000 [P:19242] Version: 0.8
    [2013-04-16 13:01:48.235673] <INFO    > GBSC-GI-51A [P:19242] GPUServer's hostname = GPUServer1
    [2013-04-16 13:01:48.235700] <INFO    > GBSC-GI-51B [P:19242] gpuserver_bind_ip = 203.0.113.10
    [2013-04-16 13:01:48.235725] <INFO    > GBSC-GI-51C [P:19242] gpuserver_bind_port = 9393
    [2013-04-16 13:01:48.235753] <INFO    > GBSC-GI-51D [P:19242] gpuserver_rest_bind = 203.0.113.10:8080
    [2013-04-16 13:01:48.235779] <INFO    > GBSC-GI-51F [P:19242] gpuserver_rest_oserver_address = https://203.0.113.1:8082/
    [2013-04-16 13:01:48.235804] <INFO    > GBSC-GI-51E [P:19242] gpuserver_infiniband_enabled = 0
    [2013-04-16 13:01:48.235835] <INFO    > GBSC-GI-52B [P:19242] All GPUs will be used
    [2013-04-16 13:01:48.235964] <NOTICE  > GBSC-SG-61A [P:19242] GPUServer is ready
    [2013-04-16 13:01:48.310201] <NOTICE  > GBSC-SR-90A [P:19242] Rest server bound on:203.0.113.10:8080, PID: 31573
    [2013-04-16 13:01:48.310311] <NOTICE  > GBSC-SC-60A [C:19244] Found GPU device ID:0, name: GeForce GTX 690
    [2013-04-16 13:01:48.310386] <NOTICE  > GBSC-SC-60A [C:19244] Found GPU device ID:1, name: GeForce GTX 690
    [2013-04-16 13:01:48.310451] <NOTICE  > GBSC-SC-60A [C:19244] Found GPU device ID:2, name: GeForce GTX 690
    [2013-04-16 13:01:48.310678] <NOTICE  > GBSC-SC-60A [C:19244] Found GPU device ID:3, name: GeForce GTX 690
    [2013-04-16 13:01:48.391046] <WARNING > GBSC-SC-71A [C:19244] Cannot send initial GPUs’ configuration, retrying every 5s ...
        

    and information such as:

    GBSC-GI-000 [P:19242] Version: 0.8 GPUServer's hostname = GPUServer1
    

    In an exemplary listing above, GPUServer cannot connect to OServer, which is probably shut down, not ready, or not reachable under the 203.0.113.1 address. GPUServer will continue trying to connect to OServer every 5 seconds. Within the established connection, GPUServer notifies about two important things:

    [2013-04-16 13:34:13.628566] <NOTICE  > GBSC-SC-61A [C:2648] Initial GPU configuration sent to OServer
    [2013-04-16 13:34:13.644977] <INFO    > GBSC-CX-50A [P:2647] Subprocess 2648 ended
    

    GPUServer successfully sent the initial GPU configuration to OServer. In response, OServer initializes the heartbeat process. GPUServer notifies about the acceptance of the heartbeat connection in the GBSC-SG-50A and GBSC-SG-52A messages:

    [2013-04-16 13:34:18.878871] <INFO    > GBSC-SG-50A [P:19242:0000001] Connection accepted from:203.0.113.1:53257
    [2013-04-16 13:34:18.878936] <INFO    > GBSC-SG-52A [P:19242:0000001] Heartbeat to 203.0.113.1 started
          

    The Connection ID must be the same in both messages

    Stopping GPUServer on Linux

    You can shutdown GPUServer in three ways:

  • Normal
    GPUServer receives the SIGINT(2) or SIGTERM(15) signal and waits until the Clients' requests are done. Every running process displays:
    <WARNING  > GSRV-SH-76C [P:21334] Stopping GPUServer...
    and finally
    <NOTICE   > GSRV-SH-66C [P:21334] GPUServer stopped.
    If processes are still active, then GPUServer prevents the termination of the connections with the Client and displays:
    GSRV-OS-76B [14045] Cannot shutdown GPUServer; it still has running processes: 10

    GPUServer does not check if users have GPU allocated. When GPUServer is successfully stopped, users will see the GPUs in the STOPPED status; they can be dropped by the user anytime and they are not active during processing.

  • Force
    GPUServer receives the SIGQUIT(3) signal and Clients' requests are finished immediately.

    Any user activities on this GPUServer will be terminated immediately, even if the user's program runs in the multi-GPU mode and uses the GPUs from another GPUServer. Most probably, the user's program will crash if GPUServer is closed this way.

    User is notified about the GPUServer termination by the message:

    terminate called after throwing an instance of 'TcpException'
    what():  get foreign port failed; Transport endpoint is not connected              
    
  • Kill
    Administrator sends the SIGKILL(9) signal. GPUServer does not have any chance to receive another signal and the process is terminated by the operating system.
  • Commands to stop

    Depending on how GPUServer was started, there are a few methods to terminate it.

  • Started from the command line as a non-daemon (option -d was not applied): Simply press Ctrl+C to send the SIGINT(2) signal and terminate GPUServer after the Client's last request is finished.

  • Started from the command line as a daemon (option -d was used). GPUServer as a daemon is disconnected from the input/output devices, such as the terminal or keyboard. The only way is to send signal from the command line.

    • Normal stop
      $ pkill -2 gpuserver
      or
      $ pkill -15 gpuserver
    • Force stop
      $ pkill -3 gpuserver
    • Kill stop
      $ pkill -9 gpuserver
  • Started as a service.

    • Normal mode
      # service gpuserver stop
    • Force mode
      # service gpuserver force-stop

  • Stop sequence

    While GPUServer is stopping it:
    • receives one of the termination signals - SIGINT(-2), SIGQUIT(-3), SIGTERM(-15)
    • terminates subservers if Client's processes are not running (except for the case when GPUServer received the SIGQUIT(-3) signal, in this case subservers are terminated anyway)
    • terminates secondary servers
    • disconnects from OServer (terminates the heartbeat signal)
    • terminates primary server

    Messages at stop

  • Normal
  • [2013-04-24 20:10:12.912604] <WARNING > GSRV-SH-76D [P:21334] Received Interrupt(2)
    [2013-04-24 20:10:12.912604] <WARNING > GSRV-SH-76D [R:21335] Received Interrupt(2)
    [2013-04-24 20:10:12.912837] <WARNING > GSRV-SH-76C [P:21334] Stopping GPUserver...
    [2013-04-24 20:10:12.912837] <WARNING > GSRV-SH-76C [R:21335] Stopping GPUserver...
    [2013-04-24 20:10:12.912919] <DEBUG   > GBSC-DT-32A [P:21334] Heartbeat termination signal sent to OServer
    [2013-04-24 20:10:16.180799] <NOTICE  > GSRV-SH-66C [R:21335] GPUServer stopped. 
    [2013-04-24 20:10:17.502957] <NOTICE  > GSRV-WP-61A [R:21335] All children of current process terminated
    [2013-04-24 20:10:17.503756] <INFO    > GBSC-CX-50A [P:21334] Subprocess 21335 ended   
    [2013-04-24 20:10:17.503794] <NOTICE  > GSRV-SH-66C [P:21334] GPUServer stopped 
    [2013-04-24 20:10:17.503803] <NOTICE  > GSRV-WP-61A [P:21334] All children of current process terminated  
          
  • Force
  • [2013-07-21 09:25:09.405433] <WARNING > GSRV-SH-76D [P:25094] Received Quit(3)
    [2013-07-21 09:25:09.405468] <WARNING > GSRV-SH-77A [P:25094] GPUServer forced to quit
    [2013-07-21 09:25:09.405497] <WARNING > GSRV-SH-76D [R:25095] Received Quit(3)
    [2013-07-21 09:25:09.405523] <WARNING > GSRV-SH-77A [R:25095] GPUServer forced to quit
    [2013-07-21 09:25:09.405674] <WARNING > GSRV-SH-77C [P:25094] Stopping GPUserver... 
    [2013-07-21 09:25:09.405730] <DEBUG   > GBSC-DT-32A [P:25094] Heartbeat termination signal sent to OServer
    [2013-07-21 09:25:09.405806] <WARNING > GSRV-SH-77C [R:25095] Stopping GPUserver... 
    [2013-07-21 09:25:14.405958] <NOTICE  > GSRV-SH-67C [R:25095] GPUServer stopped
    [2013-07-21 09:25:14.405989] <NOTICE  > GSRV-WP-61A [R:25095] All children of current process terminated
    [2013-07-21 09:25:14.406783] <INFO    > GBSC-CX-50A [P:25094] Subprocess 25095 ended
    [2013-07-21 09:25:14.406820] <NOTICE  > GSRV-SH-67C [P:25094] GPUServer stopped
    [2013-07-21 09:25:14.406829] <NOTICE  > GSRV-WP-61A [P:25094] All children of current process terminated
        

    At shutdown, GPUServer is disconnected from OServer and this fact is communicated by the following OServer's messages:

    [2013-07-21 09:28:42.408877] <WARNING > OSVC-HB-73F Heartbeat connection to GPUServer 203.0.113.11 is closed
    [2013-07-21 09:28:42.408902] <NOTICE  > OSVC-HB-63H GPUServer is being removed from hearbeat's pool, IP:203.0.113.11
    [2013-07-21 09:28:42.777980] <TRACE   > DBHC-GR-05A [140081717090048] GPU devices from IP 203.0.113.11 changed status to STOPPED
    [2013-07-21 09:28:42.778296] <INFO    > DACC-WA-50A Accounting saved for user: gpubox, GPUBox administrator
    [2013-07-21 09:28:42.778340] <INFO    > DACC-WA-50A Accounting saved for user: gpubox, GPUBox administrator
    [2013-07-21 09:28:42.778360] <DEBUG   > DBHC-WA-16B [140081717090048] Saved 2 lease records for GPU from IP 203.0.113.11
    [2013-07-21 09:28:42.778370] <NOTICE  > OSVC-HB-63J Heartbeat: GPUServer with IP:203.0.113.11 set to STOPPED      
            

    Kill mode

    GPUServer does not show any message when receiving the SIGKILL signal, but the fact is signaled by OServer with the following messages:

    [2013-07-21 09:30:23.804586] <ERROR   > OSVC-HB-83G Heartbeat connection lost to GPUServer, IP:203.0.113.11
    [2013-07-21 09:30:23.804625] <NOTICE  > OSVC-HB-63H GPUServer is being removed from hearbeat's pool, IP:203.0.113.11
    [2013-07-21 09:30:24.198713] <TRACE   > DBHC-GR-05A [140081717090048] GPU devices from IP 203.0.113.11 changed status to BROKEN
    [2013-07-21 09:30:24.198994] <INFO    > DACC-WA-50A Accounting saved for user: gpubox, GPUBox administrator
    [2013-07-21 09:30:24.199036] <INFO    > DACC-WA-50A Accounting saved for user: gpubox, GPUBox administrator
    [2013-07-21 09:30:24.199055] <DEBUG   > DBHC-WA-16B [140081717090048] Saved 2 lease records for GPU from IP 203.0.113.11
    [2013-07-21 09:30:24.199064] <NOTICE  > OSVC-HB-63J Heartbeat: GPUServer with IP:203.0.113.11 set to BROKEN
            

    In the above example in Kill mode, GPUServer was terminated by the SIGKILL(9) signal and did not have any chance to disconnect properly from OServer. Statuses of all the GPUs from this GPUServer are switched to BROKEN, but the users' allocations are preserved and will be recovered after GPUServer from the 203.0.113.10 IP address restarts.

    Restarting GPUServer

    To restart GPUServer, you have to simply shut it down and start it up again, applying the rules described in the Starting and Stopping GPUServer sections.

    For GPUServer started as a service, you can do it in one step:

    Normal mode

    # service gpuserver restart

    Force mode

    # service gpuserver force-restart

    GPUServer phases

    GPUServer can be found in one of the following phases:

  • Initializing
  • Ready
  • Start Client's request
  • Processing
  • Stop Client's request
  • Stopping
  • Initializing

    At this stage, GPUServer:

  • Starts the RESTful interface to communicate with OServer (messageGBSC-SR-90A gives the process ID)
  • Discovers GPU devices and sends the configuration to OServer (message GBSC-SC-61A)
  • Negotiates the heartbeat process with OServer (message GBSC-SG-52A)

  • In the exemplary listing in the Starting GPUServer section, GPUServer is communicating its status with the GBSC-SC-71A message. This message may be displayed for a few reasons:

  • OServer is not started
  • OServer is not yet fully initialized and cannot accept the request from GPUServer
  • OServer has been stopped (in this case, the GBSC-SC-71A message should occur together with GBSC-HB-82R)
  • Connection to OServer is lost due to the network failure (signaled by the GBSC-HB-82R message)
  • Ready

    The GBSC-SG-61A and GBSC-SG-52A messages inform that the GPUServer is ready to accept the requests from OServer and Clients. For the reestablished heartbeat signal, GPUServer displays only the GBSC-SG-52A message.

    Two processes are running with fully initialized GPUServer: primary (in the example below: PID:12433) and secondary (in the example below: PID:12450, PPID:12433).

    $ ps -Af | grep gpuserver
    gpubox   12433     1  0 14:49 ?        00:00:01 /usr/local/gpubox-gpuserver/bin/gpuserver -d -c /usr/local/gpubox-gpuserver/etc/gpuserver.conf
    gpubox   12450 12433  0 14:49 ?        00:00:01 /usr/local/gpubox-gpuserver/bin/gpuserver -d -c /usr/local/gpubox-gpuserver/etc/gpuserver.conf
    

    Primary process is responsible for Client's communication, while secondary is the RESTful server responsible for serving the OServer's requests.

    Start Client's request

    GPUServer proceeds with the new Client's request in two phases:

  • Phase I - creates a subserver by forking primary process.
  • Phase II - in subserver, every thread with access to GPU creates separate connections.
  • Subserver starts with the next available port after the one defined by the gpuserver_bind_port parameter.

    An exemplary listing with new Client's request:

    [2013-07-21 09:40:17.837752] <INFO    > GBSC-SG-50A [P:29158:0000002] Connection accepted from:203.0.113.150:36436
    [2013-07-21 09:40:17.837949] <INFO    > GBSC-SG-53A [S:29555:0000002] Subprocess 29555 created
    [2013-07-21 09:40:18.127317] <INFO    > GBST-LP-50A [S:29555:0000002] Plugin 'cuda' loaded
    [2013-07-21 09:40:18.130869] <WARNING > GBST-SP-71B [S:29555:0000002] Upgrade to InfiniBand failed: InfiniBand device test failed on Client
    [2013-07-21 09:40:18.130890] <INFO    > GBST-SP-51C [S:29555:0000002] PhaseII:connection accepted from 203.0.113.150:55804
    [2013-07-21 09:40:31.155953] <WARNING > GBST-SP-71B [S:29555:0000002] Upgrade to InfiniBand failed: InfiniBand device test failed on Client
    [2013-07-21 09:40:31.155974] <INFO    > GBST-SP-51C [S:29555:0000002] PhaseII:connection accepted from 203.0.113.150:55809
    [2013-07-21 09:40:31.677609] <WARNING > GBST-SP-71B [S:29555:0000002] Upgrade to InfiniBand failed: InfiniBand device test failed on Client
    [2013-07-21 09:40:31.677635] <INFO    > GBST-SP-51C [S:29555:0000002] PhaseII:connection accepted from 203.0.113.150:55810
    
    1. The primary process accepted the connection from the Client: 203.0.113.150
    2. GPUServer created new subserver with process ID 29555
    3. Subserver loaded the 'cuda' plugin to fulfill the request
    4. Tried to switch to the InfiniBand connection, but the Client was not prepared and the library was not loaded. Communication will continue via the TCP protocol. A successful switch to InfiniBand GPUServer sends the BST-SP-51A ... Upgraded to InfiniBand message
    5. Every single thread that uses the GPU on the Client's side will create a new communication channel, which is reported by the GBST-SP-51C message

    Processing

    While processing the Client's requests, GPUServer has more than two processes running: the primary server, the secondary server, and the subserver from every new Client's process.

    You can watch the running processes and their statistics by executing the agpubox listprocess command or opening the GPUServers / Running processes section in the GPUBox Web Console. Those processes only reflect the system's processes and threads and are kept internally in the GPUServer's memory. Sometimes, when GPUServer is not closed correctly, the processes can be still visible, whereas their physical counterparts are already gone. Such processes can be cleaned up with the agpubox rprocess command.

    Stop Client's request

    [2013-03-16 15:32:56.356287] <INFO    > GBST-ST-52A [S:29555:0000002] Connection close, task ended.
    [2013-03-16 15:32:56.356252] <NOTICE  > GBST-CH-68C [S:29555:0000002] Client's connection terminated
    [2013-03-16 15:32:56.356642] <INFO    > GBSC-SG-53B [S:29555:0000002] Process: 29555 finished
    [2013-03-16 15:32:56.357578] <INFO    > GBSC-CX-50A [P:29158] Subprocess 29555 ended
        

    The GBST-ST-52A message is displayed when a thread on the Client's side is terminated. At the Client's process termination (i.e. when user closes the rendering application), GPUServer displays information that the subserver is ended in the GBSC-CX-50A message.

    Despite the already closed Client's program, threads and other related processes may need some time to be fully terminated due to the clean-up task. For the InfiniBand connection, this can take a few seconds and all processes are visible via the agpubox listprocess command or from the GPUBox Web Console.

    Stopping

    GPUServer at this stage:

  • Checks if any Client's request is still running
  • Sends the disconnection signal to OServer
  • Terminates all processes, such as the subservers, the primary server, and the secondary server

  • There are three methods of shutting down of GPUServer.

    InfiniBand communication

    As an option, the data between the Client and GPUServer can be transferred within native InfiniBand protocol.

    InfiniBand is a switched fabric computer network communications link. The most featured capability of InfiniBand is support for native Remote Direct Memory Access (RDMA). RDMA supports the zero-copy mechanism; data is directly transferred to or from the application memory. It eliminates the need for copying the data between application memory and the data buffers in the operating system. InfiniBand has a low CPU overhead and is a high-speed, low-latency, and scalable communication method, which suits it for GPUBox requirements.
    However, InfiniBand communication implemented in GPUBox still needs the TCP/IP network to establish the connection between GPUServer and Client.

    Requirements

    Both Client and GPUServer must support the InfiniBand communication, i.e. to successfully establish a connection they need:

    InfiniBand card It is a PCIe card known also as the HCA (Host Channel Adapter). There are various adapters available on the market, but we recommend using those with as low latency and high throughput as possible.
    InfiniBand driver

    For the Linux operating system, the software driver can be obtained in a few ways.

  • Download from the Mellanox website
  • Install from the repositories provided by the Linux operating system
  • Download OpenFabrics Enterprise Distribution (OFED™) from OpenFabrics Alliance website
  • For the Windows operating system, the software driver can be downloaded from:.

  • Mellanox OFED for Windows (WinOF) from the Mellanox website
  • OpenFabrics Enterprise Distribution (OFED™) for Windows from OpenFabrics Alliance website
  • ibverbs library Library is a part of the InfiniBand driver and must be located on a system's library search path.
    libInfiniBand-gpubox.so library

    Linux

    By default, in GPUServer and Client for Linux, library is installed in the <GPUSERVER_INSTALLATION_DIR>/lib64 and <CLIENT_INSTALLATION_DIR>/lib64 directories.
    InfiniBand-gpubox.dll library

    Windows

    By default, in GPUBox Client for Windows, library is installed in the %WINDIR%\system32 directory; in most cases, the path is C:\Windows\System32

    Configuration

    Infiniband communication is set for each GPUServer separately. To enable the InfiniBand communication, GPUServer must have the gpuserver_infiniband_enabled parameter set to yes; every other value disables the InfiniBand communication. InfiniBand can also be enabled dynamically via the agpubox command. To enable InfiniBand communication, for example, on the GPU1 GPUServer, issue:

    $ agpubox gsconfig GPU1 gpuserver_infiniband_enabled "yes"

    and to disable it, issue:

    $ agpubox gsconfig GPU1 gpuserver_infiniband_enabled "no"

    By default, the InfiniBand communication is disabled.

    Parameters can also be changed from the GPUBox Web Console.

    By default, GPUBox will use the first available InfiniBand card; GPUServer and the GPUBox Client firstly uses card with ordinality 0.
    If GPUServer or the Client has more than one InfiniBand card install­ed, then the administrator or user can choose the desired card by setting the GPUBOX_IBDEV environment variable to indicate the proper name. In GPUServer, you can also set the desired device by the gpuserver_infiniband_device parameter in GPUServer configuration. The parameter has precedence over the environment variable.

    The proper name of the InfiniBand card can be obtained from ibstatus command; the following command provides a list of all available HCA adapters:

    $ ibstatus | grep device | cut -f3 -d' ' | sort -u
    'mlx4_0'
    'mlx4_1'
    

    Verify InfiniBand status

    You can verify the status and the number of available InfiniBand devices in the system by sending the command:

    $ ibstatus
    Infiniband device 'mlx4_0' port 1 status:
    	default gid:	 fe80:0000:0000:0000:0014:0500:0000:0001
    	base lid:	 0x9
    	sm lid:		 0x4
    	state:		 4: ACTIVE
    	phys state:	 5: LinkUp
    	rate:		 40 Gb/sec (4X QDR)
    	link_layer:	 InfiniBand
    
    Infiniband device 'mlx4_0' port 2 status:
    	default gid:	 fe80:0000:0000:0000:0000:0000:0000:0000
    	base lid:	 0x0
    	sm lid:		 0x0
    	state:		 1: DOWN
    	phys state:	 3: Disabled
    	rate:		 10 Gb/sec (4X)
    	link_layer:	 InfiniBand
    

    Here, we have only one device with one active, 40 Gb/sec link.

    When GPUServer and the Client are ready to establish the InfiniBand communication, within each new thread, GPUServer displays the message:

    [2013-05-22 10:26:59.346150]  GBST-SP-51A [G:00000A6:6633] Upgraded to InfiniBand, using device: mlx4_0
    

    otherwise, if connection fails, it displays the message gives the reason of failure.

    [2013-05-22 14:21:55.096532]  GBST-SP-71B [G:00000C2:6143] Upgrade to InfiniBand failed: Client couldn't load InfiniBand library.
    

    The most common reasons for failures are:

  • InfiniBand disabled in GPUServer config
  • Client couldn't load InfiniBand library
  • Server couldn't load InfiniBand library
  • InfiniBand device test failed on Client
  • InfiniBand device test failed on server
  • InfiniBand initialization failed on the Client
  • InfiniBand initialization failed on GPUServer
  • If any problem with establishing the connection over InfiniBand occurs, the Client will display the same messages without the message ID.

    The administrator can verify if the connection was successfully established based on the InfiniBand communication via the agpubox listprocess command or from the GPUBox Web Console (the Net column).

    $ agpubox lp
    +----------+-----------+---+----+---+-----------+---------+
    |GPUServer |Type       |...|Net |...|Mem        |Dev      |
    +----------+-----------+---+----+---+-----------+---------+
    |GPU1      |main       |...|TCP |...|0B         |0        |
    |GPU1      |thread     |...|IB  |...|243.8MB    |0        |
    |GPU1      |thread     |...|IB  |...|243.8MB    |1        |
    |GPU1      |thread     |...|IB  |...|243.8MB    |0        |
    |GPU1      |thread     |...|IB  |...|243.8MB    |1        |
    |GPU1      |thread     |...|IB  |...|243.8MB    |1        |
    |GPU1      |thread     |...|IB  |...|243.8MB    |1        |
    |GPU1      |thread     |...|IB  |...|243.8MB    |1        |
    |GPU1      |thread     |...|IB  |...|243.8MB    |0        |
    |GPU1      |thread     |...|IB  |...|243.8MB    |0        |
    +----------+-----------+---+----+---+-----------+---------+

    The main process is responsible for the creation of threads and does not interfere in data transfer or processing; therefore, it always stays within the TCP connection.

    You can obtain similar data from GPUBox Web Console as well.

    InfiniBand in VM

    The latest hardware and software supports the InfiniBand virtualization. The last and previous generation of the HCA adapters can operate in 2 virtualization modes.

    Device passthrough It's a PCI-passthrough technology that gives the control of a physical device to a virtual machine guest, thereby giving it full and direct access to the PCI device. The PCI device has a near-native performance; however, it is isolated and exclusively used by a VM guest, so it cannot be shared by other VM systems. In order to assign the device to a VM guest, the hardware needs to support the VT-d technology, i.e. VT-d must be supported by the processor, BIOS, and the device itself. Moreover the VM hypervisor must understand the VT-d technology. Intel calls the technology Virtualization Technology as Directed I/O (VT-d), while AMD calls it I/O Memory Management Unit (IOMMU).
    SR-IOV The next step in virtualization is Single-Root I/O Virtualization (SR-IOV). That technology allows the single physical device to be seen as multiple separate PCIe devices. Additionally, the device is not only seen as multiple devices, but it also exports a number of functions. Technology of SR-IOV introduces two types of functions: physical functions (PFs) and virtual functions (VFs). There is always one PF, while the driver can generate multiple VFs.
    In order to use the SR-IOV, the hardware and software must support VT-d. Additionally, BIOS and the device driver must support SR-IOV.

    Operations

    Add GPUServer to Infrastructure

    In order to add GPUServer, simply install, configure, and start GPUServer. New GPUServer with the devices specified in the configuration will be automatically registered to OServer.

    Archiving Log

    The log can be archived while GPUServer is running via tools like logrotate. The only requirement is to send the SIGHUP(1) signal afterwards. For example:

    # service gpuserver reload
    
    or
    $ pkill -1 gpuserver
    

    As an example, the following logrotate configuration file gpuserver-logrotate will be rotated once every 10MB and will be done 12 times.­

    gpuserver.log {
      rotate 12
      size 10M
      compress
      create 0644 gpubox gpubox
      postrotate
        /usr/bin/killall -HUP gpuserver
      endscript  
    }
    

    The following command can be run by cron periodically:

    logrotate -v -s ./gpuserver_state ./gpuserver-logrotate    
    

    Change system name

    In order to change the name of the system on which GPUServer is running, follow these steps:

  • Stop GPUServer
  • Having the root privileges change the hostname
  • Remove all of the GPUs from infrastructure that are related to the GPUServer
  • Restart GPUServer
  • Change IP address

    The following steps are necessary if GPUServer is bound to the same IP address (the gpuserver_bind_ip parameter) that the administrator wants to change. If so, follow these steps:

  • Stop GPUServer
  • Change the IP address
  • Remove all the GPUs from infrastructure that is related to the GPUServer
  • Restart GPUServer
  • GPUServer partitioning

    If the GPU devices need to be split between more than one GPUServer. The GPU devices can be split and assigned to different GPUServers by setting the gpuserver_gpus parameter.
    For example, we can split 4 devices between two GPUServers, i.e. first GPUServer has the parameter set as gpuserver_gpus = [0,1] while the other one has gpuserver_gpus = [2,3].

    Recovering GPU

    In a case when the GPUs were being used and GPUServer has failed or was restarted:

    Processes Running processes are terminated and the user's process is likely terminated as well.
    The current GPUBox implementation does not provide the mechanisms for recovering from such a situation. GPUServer and user processes have to be restarted.
    Allocations When GPUServer is down, all related GPUs have the BROKEN status or the STOPPED status if GPUServer was stopped properly.
    Users will see the BROKEN or STOPPED statues of GPUs.
    Users can drop devices with such a status.
    After GPUServer is restarted, all of the allocations are recovered by OServer and the previous usage mode is restored.

    Remove GPU from Infrastructure

    Sometimes, it is necessary to completely remove the GPU from the GPUBox infrastructure.

    The change is irreversible and the GPU will be deleted from OServer's database. The only way to once again provide the GPU is to restart a particular GPUServer.

    In order to remove the GPU, the administrator has to get rid of all of the processes and allocations regarding the GPU. Refer to the Remove GPUServer from Infrastructure chapter about setting the devices to OFF status.
    The next step is to issue the $ agpubox drop <ID> command, where <ID> is a global GPU's ID.

    Remove GPUServer from Infrastructure

    In order to remove GPUServer, none of user's process have to be running. Basically, an administrator has a few possible scenarios for removing GPUServer from the infrastructure.

    Force to quit Administrator sends the SIGQUIT(3) signal by issuing the $ pkill -3 gpuserver or # service gpuserver force-stop command. All of the activities on GPUServer are terminated and GPUServer's processes quit immediately.
    An even more intrusive and not recommended action is to send the SIGKILL(9) signal; in this case, GPUServer does not have a chance to do any other action and the GPUs will be seen by OServer as BROKEN.
    Normal quit Issue the command agpubox to check if there are any processes running with a particular GPUServer:
    $ agpubox listprocess <GPUSERVER_NAME>
    
    If you receive the 0 processes message, send the SIGTERM(15) $ pkill -15 gpuserver or SIGINT(2) $ pkill -2 gpuserver signal, or issue the # service gpuserver stop command. GPUServer will not stop if any processes are still running and it will display the appropriate message:
    [2013-05-26 12:42:13.546389] <WARNING > GSRV-SH-76B [24508] Cannot shutdown GPUServer, it still has running processes: 2
    
    Set the devices to the OFF status You can switch the devices into the OFF status, which will prevent them from being allocated and wait until the users drop the allocations.
    When the GPU is set to OFF status and GPUs are still allocated, users see them as OFF-PENDING until the allocations are active.
    To verify if there are any allocations related to this GPUServer, issue the $ agpubox gpus or $ agpubox list command.
    Once all of the allocations are dropped, you can shut the GPUServer down by sending the SIGTERM(15) or SIGINT(2)signal, or also by issuing the # service gpuserver stop command.

    We recommend this method of removing GPUServer, as it is the least invasive and transparent method for users.