Running KFX and EXSIM on OneCompute
Introduction
The purpose of this document is to provide KFX and EXSIM users the know-how to efficiently use the OneCompute service. OneCompute is DNV’s general purpose interface to cloud computing, which provides the following benefits:
- Easier and faster handling of studies with a high number of scenarios
- Brings the power and resources of Microsoft Azure, Microsoft’s cloud computing platform, to your local machine and removes the hassle of hosting servers and licences. You only pay for what you use.
- Seamless integration with KFX and EXSIM
Basic terminology
Client
The client program oc or oc.exe is the tool to interact with the computing resources.
It can be invoked as $KB/oc or simply oc if available in the system path.
If no command is specified the client is launched in interactive mode. If a command is specified (e.g. oc Set-project), that command is executed and oc exits. The latter may be useful for scripting.
Job
A Job is a collection of KFX or EXSIM scenarios to be computed and administered together. The command Create-Job JobA creates a job named jobA. Create-Job is also used to switch between jobs.
JobId
A JobId is a GUID to uniquely identify an instance of a job. An example of a JobId is 5679f60f-5856-463d-b0d0-3c3c1efc1505. For clarity, jobs can be referenced in oc commands using a JobRef (see below) in lieu of the full JobId.
JobRef - Job Reference
A JobRef takes the form ProjectName-JobName-1 and is an alias for the GUID. The last digit in the JobRef is used to distinguish between instances of the same job. When a job is started via the Start-Job command this instance is given a JobId and a JobRef. Both JobId and JobRef can be used in most commands.
Example output from the Get-JobId command:
The JobId of JobRef < Demo-jobA-8 > = < 5679f60f-5856-463d-b0d0-3c3c1efc1505 >
Project
In OneCompute Jobs are organised into Projects, where a Project is a collection of Jobs contained in a directory on the user’s local computer. The command Set-Project is used to configure and select such directories. Note that subsequent commands will apply exclusively to the selected Project. The same command (Set-Project) is also used to switch between projects.
Worker
A virtual computer created specifically for a single simulation/case that is assigned a unique WorkerId (GUID). For clarity, simulations are typically referenced using the associated JobRef.
First time setup of OneCompute for KFX and EXSIM
Follow the instructions below carefully.
Registration for OneCompute
Users of OneCompute require a Veracity account. Go to The OneCompute Portal, log in using your Veracity credentials and accept the user agreement. If you don’t already have a Veracity account you can create one there. Once you have accepted the agreement, please notify your contact in DNV that you would like OneCompute access to be added to your account.
Once access is granted, the next step is to create or specify a local project folder that oc may use.
First time start-up of oc
Firstly, open a KFX and EXSIM enabled terminal, for instance from the KFX licence client window (see figure below).
Launch oc by typing ’oc’ and pressing enter in the newly opened command window:
oc
Once running, oc will output the various options that are available for defining the project folder:
------>Welcome to OneComputIT - Running KFX and EXSIM in the cloud<----------- To get started please specify a folder to store cases, Jobs and results by running: oc Set-Project <ProjectName> This will create a folder <ProjectName> oc Set-Project <FullPathToFolder> The last folder will be the name of the Project oc Set-Project without options will use current dir as ProjectFolder If still unsure a good choice could be: oc Set-Project /home/COMPUTIT/rols/KFX-Work/OneCompute OneCompute will then be the name of the Project
Mandatory and optional parameters to a command
The commands follow a Verb-Noun pattern as in Powershell. The commands have <Mandatory> and [Optional] parameters. For instance, the command Create-Job have <Name> in angle brackets, and must be provided:
$KB/oc Get-Help Create-Job
----------------------------------------------------------------- Create-Job - <Name> Create Job Alias: change-job Alias: cj ----------------------------------------------------------------- Description: Create a new Job of Name. -----cmd=get-help -> _done -------- ? to Get-Help
Set-Project (sp)
The Set-Project command will configure oc to use the specified directory, whose name will now be defined as the ProjectName.
In the below example, run in interactive mode, the full path to the desired directory is used.
Set-Project /home/COMPUTIT/rols/ldev/OneComputePlantCFD-Documentation/TestProject
This command gives the following output:
ProjectName=TestProject Store cases to run in folder /home/COMPUTIT/rols/ldev/OneComputePlantCFD-Documentation/TestProject/Cases Jobs are stored locally /home/COMPUTIT/rols/ldev/OneComputePlantCFD-Documentation/TestProject/Jobs Results are downloaded to /home/COMPUTIT/rols/ldev/OneComputePlantCFD-Documentation/TestProject/Results ----------------------------------------------------------- when _done adding cases continue with: Start-Case <casename> or Create-job, Add-Cases, Upload-job, Start-Job, and when done Get-Results ----------------------------------------------------------- Please use Get-Help, Get-Help <command>, Get-Info and Get-Alias frequently! Happy OneComputing!!
Get-Help
With the Project created, a good first step is to run the Get-Help command, which will provide a list of the core commands available in oc.
Get-Help
gives the output:
-------------------------------------------------------------------- ProjectName - OneCompute ProjectDir - C:\Users\rols\KFX-WORK\OneCompute JobName - T03 CurrentDir - C:\home\rols\ldev\Prosjekt\DebugPoolF -------------------------------------------------------------------- The following commands may be used Set-Project - [path] Set path as ProjectDir and init. Create-Job - <Name> Create Job. Add-Cases - [Cases Folder] Add Cases from Cases Folder to Job and prepare for upload. Upload-Job - Upload job to cloud storage. Start-Job - [-d <days>] [-s] [-p <poolId>] Submit a new instance of job for execution.. Stop-Job - [JobRef] [-q] [Name] sends stop to (all) or specific worker. Get-Jobs - [-All] [-JobId] Get a list of running Jobs (JobRefs). Get-Status - [JobRef] [Name/-All] [-d/-t] Progress of workers in a JobRef. Send-Control - [JobRef] <Name/-All> sends kfx Control command to worker. Sync-Results - [JobRef] [-r] [Name] [filter] Download results (default filter="*.r3d *.kfx.gz"). Get-Help - Short help menu, option <command> detailed help about command. Get-FullHelp - List All Commands. Get-Alias - Display a list of alias for the commands. Quit - Quit the oc program. -----cmd=get-help -> _done -------- ? to Get-Help <OneCompute-T03-13>|</OneCompute>
Note that the last line of the output lists the JobRef and current folder, where the JobRef is ’OneCompute-T03-13’, and ’OneCompute’ is the current folder:
<OneCompute-T03-13>|</OneCompute>
Most commands in oc refer to the JobRef and hence it’s important to take note of this.
Calculations in OneCompute can be run via hybrid or cloud desktop approaches.
Supported scenarios and workflows
Hybrid
In the hybrid approach, scenarios are defined on the desktop, but the actual calculations are submitted and run on OneCompute. The results can then be dowloaded and post-processed locally.
Cloud Desktop
The cloud desktop approach essentially offloads the entire desktop to the cloud via DNV Remote Services
This can be desirable when temporary use of a GPU-enabled desktop occurs, as is often the case when operating KFX and EXSIM.
Defining KFX and EXSIM cases
When defining inputs and outputs in cases, relative paths are preferred (absolute paths should be avoided) to avoid issues associated with transferring files between machines. The following ways of specifying KFX and EXSIM scenarios are supported on OneCompute:
EXSIM
wex
The standard EXSIM input file format (.wex file), given:
- One folder contains one case only (whilst this format supports the running of many cases in a single directory, this is currently limited to one case per directory in OneCompute)
- All required files to execute the case are contained in the same folder
- Only one wex file in that folder (where multiple wex files are present, the user will be requested to select one from the list)
- No scn, json or fsc files in that folder
KFX
fsc
A case specified with fsc may be excecuted (.fsc file), given:
- One folder contains one case only
- All required files to execute the case are contained in the same folder
- Only one fsc file in that folder
- No scn, json or wex files in that folder
json
OneCompute is ideal for Phast-CFD scenarios, which are specified in the json format.
xml files
xml files with spawn commands are supported for all types of KFX scenario specification.
scn
Scenario files, both single- and multi-case, may be executed (.scn files).
Single-case scenario (.scn) file calculations
If there is just a single scenario to run, the calculation may be submitted to OneCompute by:
oc start-case CaseName
This will create a job named CaseName including one scenario named CaseName, and it will be uploaded and started in one go - this can serve as a quick way to start a simulation. A job monitor displaying the output as the simulation progresses will then be started. If the job monitor is killed it does not stop the calculation.
Multi-case scenario (.scn) file calculations
KFX on OneCompute is most beneficial for jobs that include multiple cases. Depending on the complexity of the scenarios, the sweet spot is around 5 cases per job. First, copy the cases to run into the cases folder of the project folder. Run command create-job JobName and add-cases to add all cases from cases folder to the jobs/JobName folder (alternatively, if it is desired to add cases from a different location, this can be specified as an option to the add-cases command). Now run upload-job and start-job. While the job is running, you can use the command Get-JobStatus to show the status of the running simulations and get an estimated time to completion.
Tips and tricks
List commands or aliases
Many commands have an associated alias. For example, the alias fro Create-Job i simply cj. You can access the complete list by running the Get-Alias command.
List all available commands
The complete help menu can be accessed by running the Get-FullHelp command, which is run as follows in interactive mode:
Get-FullHelp
----------------------------------------------------------------- ProjectName - TestProject ProjectDir - /home/COMPUTIT/rols/ldev/OneComputePlantCFD-Documentation/TestProject JobName - Unknown ----------------------------------------------------------------- The following commands may be used Add-Cases - [Cases Folder] Add Cases from Cases Folder to Job and prepare for upload. Cancel-Job - [JobRef] Abort all workers in the Job. Copy-File - Copy a file/folder within cloud storage. Create-Job - <Name> Create Job. Create-RestartJob - [-r/-t] Creates a new Job from current JobRef to easily restart simulation. Delete-JobRef - [-y] <JobRef*> Deletes JobRef matching option and corresponding Results. Download-Job - Download current Job to local storage. Get-Alias - Display a list of alias for the commands. Get-AzCopy - Get a command to download with azcopy. Get-CaseChildItem - <CaseName/-All> asks worker of CaseName to give a list of files in working dir. Get-ChildItem - [-r] [dir] list files in blob storage. Get-Debug - [JobId] Download debug log of either current job or with JobId. Get-File - <fileInBlob> Download a file/folder from blob storage. Get-FullHelp - List All Commands. Get-Help - Short help menu, option <command> detailed help about command. Get-Info - Get info about pool and cloud storage. Get-JobId - [JobRef] Get JobId and WorkerIds of JobRef. Get-Jobs - [-All] [-JobId] Get a list of running Jobs (JobRefs). Get-LocalChildItem - [dir] list files in blob storage. Get-MyBlob - [-w] [-d #daysValid] [-n fileName] Get a read only SasToken. Get-Results - [-r] [JobRef] [Name] [filter] Download results (default filter="*.r3d *.kfx.gz"). Get-Status - [JobRef] [Name/-All] [-d/-t] Progress of workers in a JobRef. Push-File - <CaseName/-All> <fileName> ask worker to push file to blob. Quit - Quit the oc program. Remove-Item - [filter] Remove a file/folder in cloud storage. Send-Control - [JobRef] <Name/-All> sends kfx Control command to worker. Send-File - <CaseName/-All> Sends a file to the worker of CaseName.. Set-JobRef - <JobRef> Change the current JobRef. Set-KFXVersion - [KFXVersion] change KFXVersion to run. Set-LocalLocation - Change current directory in local storage. Set-Location - [-l] Change current directory in cloud storage or [-l] local. Set-MaxDaysOfJob - Set the default number of days for a job to expire. Set-Platform - [test/develop] [UserName] Change platform.onecompute.DNV.com. Set-Pool - [poolId] change to another pool of computers. Set-Project - [path] Set path as ProjectDir and init. Set-WorkerImage - [workerImage] change Worker to run. Show-Messages - Read short messages from job.. Start-Agent - Start an Agent. Start-Case - <case> Prepare Submit and start case in one go. Start-Job - [-d <days>] [-s] [-p <poolId>] Submit a new instance of job for execution.. Start-JobLocal - Create a new instance of job and execute locally. Start-JobMonitor - Monitor a running job. Stop-Job - [JobRef] [-q] [Name] sends stop to (all) or specific worker. Sync-Results - [JobRef] [-r] [Name] [filter] Download results (default filter="*.r3d *.kfx.gz"). Unzip-File - <fileName> [outDir] Unzip a file (Linux). Upload-Cases - Add cases and Upload to job cloud storage. Upload-File - Upload a file/folder to blob storage. Upload-Job - Upload job to cloud storage. -----cmd=get-fullhelp -> _done -------- ? to Get-Help
Running three simulations in a single job
An example is given below for JobName=LiquidPool, which has three scenarios defined in json format (N-OCTANE_Circle_3_1.5_F.json, N-DECANE_Rectangle_3_2_1.5_F.json, N-NONANE_Rectangle_5_2_1.5_F.json ). Follow the steps below:
Create-Job
oc cj LiquidPool
This command sets JobName to LiquidPool, and creates a folder named LiquidPool in the Jobs directory.
Add-Cases
oc ac
This command copies cases from the Cases directory to the Jobs/LiquidPool directory. The Job <LiquidPool> is now ready to be uploaded to the cloud storage by the command Upload-Job
Upload-Job
oc upload
This command will provide the following output:
<Test-LiquidPool-0>|</Test> -- Uploading cases from /home/COMPUTIT/larsl/Projects/Test/Jobs/LiquidPool -> BlobStorage:/Test/Jobs/LiquidPool Uploaded: N-OCTANE_Circle_3_1.5_F.json Uploaded: N-DECANE_Rectangle_3_2_1.5_F.json Uploaded: N-NONANE_Rectangle_5_2_1.5_F.json -----cmd=upload-job -> _done -------- ? to Get-Help <Test-LiquidPool-0>|</Test>
The job is then submitted to the queue for calculation by the command Start-Job
Start-Job
oc sj
This command will provide the following output:
<Test-LiquidPool-0>|</Test> Start-Job ----------------------------------------------------------------- Creating new Job Test-LiquidPool-0 of Job LiquidPool with 3 cases to Compute ----------------------------------------------------------------- Worker 75078ccf-5fde-4023-9c70-bc9314471bd4 will compute case NONANE_Circle_5_3_D.p2k Worker f0c7e34a-d861-430c-88f4-421cc6b111cd will compute case OCTANE_Circle_4_1.5_E.p2k Worker 8e27da8d-72ac-4cc9-bb22-991d51bf59b8 will compute case OCTANE_Rectangle_5_4_1.5_D.p2k ----------------------------------------------------------------- Waiting for Job Test-LiquidPool-0 to be registered Waiting for Job Test-LiquidPool-0 to be registered Waiting for Job Test-LiquidPool-0 to be registered Submitted Job Test-LiquidPool-0 jobId 9ecb7393-d2c2-4ee0-a9aa-ec72c011d1fc Connecting to Job Test-LiquidPool-0 -----cmd=start-job -> _done -------- ? to Get-Help
Displaying Job status (single Job)
Get-Status
oc gs
This command lists the status of cases within the selected Job along with the status of the overall Job on cloud storage:
<Test-LiquidPool-0>|</Test> Get-status ------------------------------------------------------------------------------------------- | Job Reference | JobId | Status| | Test-LiquidPool-0 | 9ecb7393-d2c2-4ee0-a9aa-ec72c011d1fc | Completed| ------------------------------------------------------------------------------------------- | Case Name | WorkerId | Message| | NONANE_Circle_5_3_D | 75078ccf-5fde-4023-9c70-bc9314471bd4 | | | OCTANE_Rectangle_5_4_1.5_D | 8e27da8d-72ac-4cc9-bb22-991d51bf59b8 | | | OCTANE_Circle_4_1.5_E | f0c7e34a-d861-430c-88f4-421cc6b111cd | | ------------------------------------------------------------------------------------------- -----cmd=get-job -> _done -------- ? to Get-Help
Displaying Job status (multiple Jobs)
Get-Jobs
oc gj
This command lists the all active (pending/running) Jobs. A complete list including all Jobs can be obtained by running the command with the -All option.
<Test-LiquidPool-0>|</Test>|</home/COMPUTIT/larsl/Projects/Testing/Jobs/LiquidPool>Get-Jobs -All |---+------------------------------+-----------------------+----------+-----------+-----| | - | JobRef | StartTime | CpuHours | Status | % | |---+------------------------------+-----------------------+----------+-----------+-----| | 1 | One_Compute-LiquidPooltest-1 | 4/15/2021 8:36:02 AM | 0.07 | Aborted | 100 | | 2 | One_Compute-LiquidPooltest-2 | 4/15/2021 8:40:28 AM | 3.65 | Completed | 100 | | 3 | Test-LiquidPool-0 | 4/15/2021 12:55:20 PM | 4.8 | Completed | 100 | |---+------------------------------+-----------------------+----------+-----------+-----| -----cmd=get-jobs -> _done -------- ? to Get-Help
Downloading the results of a Job
Use Sync-Results.
oc sr
Results are downloaded from cloud storage to the local Results folder. The results are still stored in the cloud storage.
Uploading or Downloading specific files
Use upload-file:
oc upload-file geometry_module.kfx
Use get-file:
oc gf jet_fire_bmg_exit.r3d
Note that this command will search for all file(s) with the specified name and download them to the local Results folder.
Sharing results with others
Use Get-MyBlob:
oc gmb
Get-AzCopy:
oc gaz
Setting an Active Job
Use Set-JobRef <JobRef> (from list of Jobs shown by the command Get-Jobs):
oc sjr onecompute_104-jet_fire_multi_scn-1
Check that JobRef in the command prompt has changed, for example, from <Unknown>|</onecompute_104> to <onecompute_104-jet_fire_multi_scn-1>|</onecompute_104>
Killing or Stopping a running Job
Use Cancel-Job to kill a Job; no results are then stored.
oc cancel-job
Use Stop-Job if a controlled stop is wanted; all simulations are stopped and results are stored (currently KFX only).
oc stop-job
Controlling active simulations
Use Send-Control to change control variables for a single case or a Job. In this example, the mximum Courant number is set to 15:
oc Send-Control <CaseName> Coumax=15