Changes between Version 4 and Version 5 of DRM4G/ResourceConfiguration


Ignore:
Timestamp:
Dec 1, 2016 3:07:55 PM (5 years ago)
Author:
minondoa
Comment:

--

Legend:

Unmodified
Added
Removed
Modified

  • DRM4G/ResourceConfiguration

    v4 v5  
    22
    33The configuration file `resources.conf` is used to describe computing resources. When you start DRM4G, resources.conf file is copied under `~/.drm4g/etc` directory by default if it does not exist or under whatever directory specified with **DRM4G_DIR**. The file can be edited directly or by executing the `drm4g resource edit` command.
     4
    45
    56== Configuration format ==
    … …  
    78The configuration resource file consists of sections, each led by a [section] header, followed by `key = value` entries. Lines beginning with `#` are ignored. Permitted sections are `[DEFAULT]` and `[resource_name]`.
    89
     10
    911== DEFAULT section ==
    1012
    1113The DEFAULT section provides default values for all other resource sections.
    1214
     15
    1316== Resource section ==
    1417
    1518Each resource section has to begin with the line `[resource_name]` followed by `key = value` entries.
     19
    1620
    1721=== Configuration keys common to all resources: ===
    … …  
    2630    * `username`:        Name of the user that will be used to log on to the front-end.
    2731 
    28     * `frontend`:        Hostname or ip address of either the cluster or grid user interface you'll be connected to. The syntax is "host:port", by default the port used is 22.
     32    * `frontend`:        Hostname or ip address of either the cluster or grid user interface you'll be connected to. The syntax is "''host:port''" and by default the port used is 22.
    2933
    3034    * `private_key`:     Path to the identity file needed to log on to the front-end.
    3135
    32     * `public key`:      Path to the public identity file needed to log on to the front-end. (OPTIONAL, by default the '''private_key''''s value will be taken, to which '''.pub''' will be added)
     36    * `public key`:      Path to the public identity file needed to log on to the front-end.
     37        * '''OPTIONAL''': by default the '''private_key''''s value will be taken, to which '''.pub''' will be added)
    3338
    3439    * `scratch`:         Directory used to store temporary files for jobs during their execution, by default, it is `$HOME/.drm4g/jobs`
    … …  
    4348        - `slurm`:         SLURM cluster.
    4449        - `slurm_res`:     [http://www.bsc.es/marenostrum-support-services/res RES(Red Española de Supercomputación)] resources.
     50        - `fedcloud`:      it will indicate the DRM4G that this resource will be used to create ''VMs'' (Virtual Machines).
    4551
    46 [[NoteBox(tip, Note that for `communicator` you have two options when it comes to accessing a resource through the ssh protocol. If you don't know which one you prefer choose `ssh`.)]]
     52[[NoteBox(tip, Note that for `communicator` you have two options when it comes to accessing a resource through the ssh protocol. If you don't know which one you prefer use `ssh`.)]]
     53
    4754
    4855=== Keys for non-grid resources such as HPC resources: ===
    4956
    50     * `queue`:             Queue available on the resource. If there are several queues, you have to use a "," as follows "queue  = short,medium,long".
     57    * `queue`:             Queue available on the resource. If there are several queues, you have to use a "," as follows "''queue''  = ''short,medium,long''".
    5158
    5259    * `max_jobs_in_queue`: Max number of jobs in the queue.
    … …  
    5865    * `project`:           It specifies the project variable and is for TORQUE/PBS,  Grid Engine and LSF clusters.
    5966
     67
    6068=== Keys for grid resources: ===
    6169 
    6270    * `vo`:                Virtual Organization (VO) name.
    6371
    64     * `host_filter`:       A host list for the VO. Each host is separated by a ",". Here is an example: "host_filter = prod-ce-01.pd.infn.it, creamce2.gina.sara.nl".
     72    * `host_filter`:       A host list for the VO. Each host is separated by a ",". Here is an example: "''host_filter = prod-ce-01.pd.infn.it, creamce2.gina.sara.nl''".
    6573
    66     * `bdii`:              It indicates the BDII host to be used. The syntax is "bdii:port". If you do not specify this variable, the `LCG_GFAL_INFOSYS` environment variable defined on the grid user interface will be used by default.                     
     74    * `bdii`:              It indicates the BDII host to be used. The syntax is "''bdii'':''port''". If you do not specify this variable, the `LCG_GFAL_INFOSYS` environment variable defined on the grid user interface will be used by default.                     
    6775
    6876    * `myproxy_server`:    Server to store grid credentials. If you do not specify this variable, the `MYPROXY_SERVER` environment variable defined on the grid user interface will be used by default.
     77
     78
     79=== Keys for !FedCloud resources: ===
     80
     81'''!FedCloud''' resources are a bit different than the rest. They are not considered to be ''hosts'' so they won't be listed when using the command `drm4g host` and therefore won't be used to execute jobs. Their only function is to connect to a machine with a cloud proxy certificate ('''X.509 certificates''') capable of giving you access to cloud resources.
     82
     83    * `vm_communicator`:  or  authentication type for the created VMs : 
     84        - `local`:       The resource will be accessed directly.
     85        - `ssh`:         The resource will be accessed through ssh's protocol via Paramiko's API.
     86        - `op_ssh`:      The resource will be accessed through OpenSSH's CLI.
     87
     88    * `vm_user`:         Name of the user that will be used to log on to the creates VMs.
     89
     90    * `bdii`:            It indicates the ''BDII'' host to be used. The syntax is "''bdii'':''port''".
     91        * You can just use '''''lcg-bdii.cern.ch:2170'''''
     92
     93    * `myproxy_server`:  Server to store cloud credentials. If you do not specify this variable, the `MYPROXY_SERVER` environment variable defined on the grid user interface will be used by default.
     94
     95    * `nodes`:           It indicates how many VMs you wish to create with the specified configuration
     96
     97    * `volume`:          It's possible to create some extra storage and add it to the VM. With this you can specify how many extra GBs of storage you want.
     98
     99
     100The values of the next configuration keys can be customized at your discretion. A new cloud configuration file has been added to the DRM4G called "'''cloudsetup.json'''" for this reason. This resource keys reference the information saved in this cloud configuration file.
     101
     102    * `cloud`:           Name that describes the site from which the image, that will be used to create the VM, will be acquired.
     103
     104    * `virtual_image`:   It indicates which one of the system images available you will be using
     105
     106    * `flavour`:         It indicates the template of your new VM
     107        *  specification
     108
     109Where and how to get the correct values for your cloud configuration file as well as a more in depth explanation of some of these configuration keys can be found in the section [./EGIFedCloud How to configure an EGI FedCloud VM].
    69110
    70111== Examples ==
    … …  
    126167{{{#!comment
    127168#for this maybe another section will be better, since we'll have to explain how to set up the cloud_json and how it works
    128 [cesnet]
     169}}}
     170!FedCloud virtual organization
     171
     172{{{
     173[cesnet_metacloud]
    129174enable         = true
    130175communicator   = ssh
    131176username       = user
    132 vo_user        = drm4g_admin
     177vm_communicator= op_ssh
     178vm_user        = drm4g_admin
    133179frontend       = ui.macc.unican.es
    134180private_key    = ~/.ssh/id_rsa