Using Windows HPC Server 2008 with GPGPU hardware

In Release Candidate 2 of HPC Server 2008 a new feature has been added that will allow users to run GPGPU programs through the Windows HPC Server 2008 Job Scheduler.

In order to understand how GPGPU programs can be run on Windows HPC Server 2008 you first need to understand how sessions work in Windows. In Windows Server 2008 services are isolated to run in session 0 as shown in diagram 1

Diagram 1 – default job creation is in Session 0

There are several tools in windows that can be used to better understand and monitor session usage. In Windows Task Manager you can add a column so that you can see what session each process is running in. To run Windows Task Manager right click on the system tray and click on Task Manager. When Task manager starts click on the View menu, then click on Select Columns and you will see the dialog below where you can add a column to show what the Session ID is for all processes. You will also probably want to select “Show Processes from all users” in Task Manager to see all of the processes running on your server.

There is also a command line tool named qwinsta.exe that will list out all the sessions on a computer. You may use qwinsta.exe on either a local or remote server to view all of the sessions and their owners.

Starting in Release Candidate 2 of Windows HPC Server 2008 users are allowed to specify that their jobs be started in either the console session or in a terminal server session. Users may specify this by using the HPC_ATTACHTOCONSOLE and HPC_ATTACHTOSESSION environment variables when they submit their jobs. Diagram 2 illustrates how HPC_ATTACHTOCONSOLE allows a user to specify that their GPGPU program be started in the console session.

Diagram 2 – starting a job in the console session

Allowing a job to be started in the console session is necessary if a user is using their graphics processing unit (GPU) for acceleration. On Windows Server 2008 the GPU is accessed from the video driver which is only loaded in the console session, so programs which rely on access to the video driver must be run in the console session.

A locally logged on console session must first be created or jobs submitted using HPC_ATTACHTOCONSOLE=TRUE will fail. The owner of the console session must be the same as the user who the jobs will be run as.

Environment Variable Name: Possible Values

HPC_ATTACHTOCONSOLE

TRUE The Job Scheduler will attempt to start the submitted job in the console session. If a console session exists that has the same owner as the user who the job is run as then the job will be started in the console session. If the job cannot be started in the console session, either because there isn’t a logged on console session or the console session owner isn’t the same as the user who the job is running as, then the job will fail

TRY The Job Scheduler will attempt to start the submitted job in the console session. If a console session exists that has the same owner as the user who the job is run as then the job will be started in the console session. If the job cannot be started in the console session, either because there isn’t a logged on console session or the console session owner isn’t the same as the user who the job is being run as, then the job will not be run in the console session but will still be run.

HPC_ATTACHTOSESSION

TRUE The Job Scheduler will attempt to start the submitted job in a user’s terminal server session. If a terminal server session exists that has the same owner as the user who the job is run as then the job will be started in that terminal server session. If the job cannot be started in a terminal server session because there isn’t a terminal server session owned by the user who the job is running as, then the job will fail

TRY The Job Scheduler will attempt to start the submitted job in the console session. If a console session exists that has the same owner as the user who the job is run as then the job will be started in the console session. If the job cannot be started in a terminal server session, then the job will not be run in a terminal server session but will still be run.

A typical command line to submit a job using HPC_ATTACHTOCONSOLE is:

“Job submit /env:HPC_ATTACHTOCONSOLE=TRUE mygpgpu.exe”

When HPC_ATTACHTOCONSOLE or HPC_ATTACHTOSESSION is specified it will be applied to all ranks in an MPI Job.

Steps to use GPGPU on Windows HPC Server 2008

Now that you understand sessions and the new HPC_ATTACHTOCONSOLE feature in Windows HPC Server 2008 we’ll review the steps necessary to get GPGPU programs working.

1. Setup Windows HPC Server 2008 and HPC Pack 2008 on your cluster

2. Install your GPGPU enabled video drivers on your compute nodes

3. Create a logged on local console session on your GPU enabled compute nodes with the same user that GPGPU jobs will be run as

4. (optional) create a Job Scheduler group for your nodes which have GPUs

5. Submit your GPGPU jobs to the job scheduler using HPC_ATTACHTOCONSOLE and using the same user account as the owner of the console session. If you created the Job Scheduler GPGPU group then you can also specify that the job only be scheduled on nodes in that group

Additional Tips

To create the Console Session automatically you can modify your machines registry so that when your compute nodes boot they automatically login a user and create an active console session. The KB article “How to turn on automatic logon in Windows XP” gives more details and is available at http://support.microsoft.com/kb/315231

The information in this KB article does apply to Windows HPC Server 2008 too. Do note that some domains may have group policies that prevent auto login.

Of course if you do use Auto Login on your servers to create an active Console Session you will want to also lock the workstation so other users won’t be able to access the session. You may easily lock a workstation in a cmd file by using the following command: “rundll32.exe user32.dll LockWorkStation”

If some of your compute nodes don’t have GPU hardware or you have different types of GPU hardware you can use the new Job Scheduler Groups feature of HPC Server 2008 to make sure that your submitted GPGPU jobs go to the compute nodes that contain GPU hardware. From the HPC Cluster Manager select the Node Management pane, click on Nodes, By Group, then in the action pane select Add Group. You can then create a new group to contain your GPGPU compute Nodes

You can right click on your Compute Nodes that contain GPGPU hardware and select Groups and add these compute nodes to the GPGPU Nodes group.

Now when you create a new job you can specify that the job only be scheduled on compute nodes that belong to the GPGPU nodes group. You can do this in the Resource Selection tab of the New Job dialog:

You can also use the command line to specify the node group when you submit a job:

“job submit /env:HPC_ATTACHTOCONSOLE=TRUE /nodegroup:”GPGPU Nodes “ mygpgpu.exe”

This will request that your job only be run on machines in the node group “GPGPU Nodes”

Another useful tool for managing devices in clusters is devcon.exe. It can be useful for managing video devices remotely via command line in your clusters. More information on devcon.exe is available in this KB Article:

http://support.microsoft.com/kb/311272