Changes between Version 3 and Version 4 of DRM4G/ResourceConfiguration/EGIFedCloud


Ignore:
Timestamp:
Dec 1, 2016 8:34:34 PM (5 years ago)
Author:
minondoa
Comment:

--

Legend:

Unmodified
Added
Removed
Modified

  • DRM4G/ResourceConfiguration/EGIFedCloud

    v3 v4  
    9595
    9696
    97 
    98 1. Configure the `esrVO` resource. If the grid user interface has defined `LCG_GFAL_INFOSYS` and `MYPROXY_SERVER` variables, you do not have to indicate `bdii` and `myproxy_server` keys in your configuration:
     97== How to manage VMs ==
     98
     99
     1001. Customize your `cloudsetup.json` to match your requirements. You can search for the available images in [https://appdb.egi.eu/store/vappliance/ EGI's Applications Database Cloud Marketplace].
     101 - Search for the image you want. [[br]]
     102  [[Image(searchSystemImage.png,40%)]]
     103 - Select the one you prefer. [[br]]
     104  [[Image(selectSystemImage.png,40%)]]
     105 - In the page that has opened up, select one of the sites that offer that same image. [[br]]
     106  [[Image(selectSite.png,40%)]]
     107 - A list of flavours will appear. Choose one and click on "get IDs". [[br]]
     108  [[Image(selectFlavour.png,40%)]]
     109 - After a pop-up window will appear with the information you'll have to write on the cloud setup file. [[br]]
     110  [[Image(systemImageInfo.png,40%)]]
     111  * '''Site endpoint''' corresponds to `endpoint`
     112  * '''Template ID''' corresponds to `<hardware_template>`
     113  * '''OCCI ID''' corresponds to `<system_image_name>`
     114{{{#!js
     115{
     116  "fedcloud" : {
     117    "vo" : "fedcloud.egi.eu",
     118    "clouds" : {
     119      "EGI FedCloud - CESNET-METACLOUD" : {
     120        "endpoint" : "https://carach5.ics.muni.cz:11443",
     121        "apps" : {
     122          "Ubuntu-14.04" : "http://occi.carach5.ics.muni.cz/occi/infrastructure/os_tpl#uuid_egi_ubuntu_server_14_04_lts_fedcloud_warg_131"
     123        },
     124        "flavours" : {
     125          "Medium" : "http://fedcloud.egi.eu/occi/compute/flavour/1.0#medium"
     126        }
     127      }
     128    }
     129  }
     130}
     131}}}
     1321. Configure a "'''fedcloud'''" resource in `resources.conf`.
     133 * The way to inform the ''DRM4G'' that a defined resource is going to be used to create VMs, is to set its `lrms` key to `fedcloud`.
     134 * This way, the ''DRM4G'' won't consider this resource as a ''host'' and so '''it won't send it jobs to execute'''.
     135  * The VMs created by this resource are what will be considered to be its ''hosts''. That entails that '''the VMs will not be listed by the command ''' `drm4g resource list`
     136{{{#!sh
     137[user@mycomputer~]$ drm4g resource edit
     138[CESNET_Metacloud]
     139enable         = true
     140communicator   = ssh
     141vm_communicator= op_ssh
     142private_key    = ~/.ssh/id_rsa
     143username       = user
     144vm_user        = drm4g_admin
     145frontend       = ui.meteo.unican.es
     146lrms           = fedcloud
     147cloud          = EGI FedCloud - CESNET-METACLOUD
     148myproxy_server = myproxy1.egee.cesnet.cz
     149flavour        = Medium
     150virtual_image  = Ubuntu-14.04
     151nodes          = 1
     152volume         = 10
     153max_jobs_running = 5
     154}}}
     1551. Start the DRM4G and check if the resource has been created successfully :
    99156{{{
    100157#!sh
    101 [user@mycomputer~]$ drm4g resource edit
    102 [esrVO]
    103 enable         = true
    104 communicator   = local
    105 username       = user
    106 frontend       = ui.meteo.unican.es
    107 lrms           = cream
    108 vo             = esr
    109 bdii           = bdii.grid.sara.nl:2170
    110 myproxy_server = px.grid.sara.nl
    111 }}}
    112 1. List and check if resource has been created successfully :
    113 {{{
    114 #!sh
     158[user@mycomputer~]$ drm4g start
     159Starting DRM4G ....
     160  OK
    115161[user@mycomputer~]$ drm4g resource list
    116162RESOURCE            STATE               
    117 esrVO               enabled
    118 }}}
    119 1. List the CEs available on the `esr` VO:
    120 {{{
    121 #!sh
    122 [user@mycomputer~]$ drm4g host list
    123 HID ARCH       JOBS(R/T) LRMS       HOST
    124 0   x86_64           0/0 cream-pbs  esrVO::cream.afroditi.hellasgrid.gr
    125 1   x86_64           0/0 cream-pbs  esrVO::ce1.ipgp.fr   
    126 2   x86_64           0/0 cream-pbs  esrVO::cr1.ipp.acad.bg
    127 3   x86_64           0/0 cream-pbs  esrVO::sbgce2.in2p3.fr
    128 4   x86_64           0/0 cream-pbs  esrVO::ce0.bordeaux.inra.fr
    129 5   x86_64           0/0 cream-pbs  esrVO::cce.ihep.ac.cn
    130 6   x86_64           0/0 cream-pbs  esrVO::ce02.ngcc.acad.bg
    131 7   x86_64           0/0 cream-pbs  esrVO::ce01.macc.unican.es
    132 8   x86_64           0/0 cream-pbs  esrVO::cygnus.grid.rug.nl
    133 9   x86_64           0/0 cream-pbs  esrVO::t2ce06.physics.ox.ac.uk
    134 10  x86_64           0/0 cream-lsf  esrVO::ce1.ts.infn.it
    135 11  x86_64           0/0 cream-lsf  esrVO::gridce1.pi.infn.it
    136 12  x86_64           0/0 cream-lsf  esrVO::gridce3.pi.infn.it
    137 13  x86_64           0/0 cream-pbs  esrVO::cream02.grid.uoi.gr
    138 14  x86_64           0/0 cream-pbs  esrVO::lapp-ce02.in2p3.fr
    139 15  x86_64           0/0 cream-pbs  esrVO::grid002.jet.efda.org
    140 16  x86_64           0/0 cream-lsf  esrVO::gridce4.pi.infn.it
    141 17  x86_64           0/0 cream-lsf  esrVO::gridce0.pi.infn.it
    142 18  x86_64           0/0 cream-lsf  esrVO::gridce2.pi.infn.it
    143 19  x86_64           0/0 cream-pbs  esrVO::t2ce06.physics.ox.ac.uk
    144 20  x86_64           0/0 cream-pbs  esrVO::grid0.fe.infn.it
    145 21  x86_64           0/0 cream-pbs  esrVO::ce0.m3pec.u-bordeaux1.fr
    146 22  x86_64           0/0 cream-pbs  esrVO::juk.nikhef.nl 
    147 23  x86_64           0/0 cream-pbs  esrVO::gridce.ilc.cnr.it
    148 24  x86_64           0/0 cream-lsf  esrVO::cert-37.pd.infn.it
    149 25  x86_64           0/0 cream-pbs  esrVO::cream-ce-2.ba.infn.it
    150 26  x86_64           0/0 cream-sge  esrVO::cccreamceli09.in2p3.fr
    151 27  x86_64           0/0 cream-sge  esrVO::cccreamceli10.in2p3.fr
    152 28  x86_64           0/0 cream-pbs  esrVO::gazon.nikhef.nl
    153 29  x86_64           0/0 cream-pbs  esrVO::klomp.nikhef.nl
    154 30  x86_64           0/0 cream-pbs  esrVO::cream-ce-3.ba.infn.it
    155 31  x86_64           0/0 cream-pbs  esrVO::cream-ce-4.ba.infn.it
    156 32  x86_64           0/0 cream-pbs  esrVO::creamce.gina.sara.nl
    157 33  x86_64           0/0 cream-lsf  esrVO::prod-ce-01.pd.infn.it
    158 34  x86_64           0/0 cream-pbs  esrVO::creamce2.gina.sara.nl
    159 35  x86_64           0/0 cream-pbs  esrVO::creamce3.gina.sara.nl
    160 36  x86_64           0/0 cream-slur esrVO::ce3.ui.savba.sk
    161 37  x86_64           0/0 cream-pbs  esrVO::glite-cream.scai.fraunhofer.de
    162 38  x86_64           0/0 cream-pbs  esrVO::cream-ce02.marie.hellasgrid.gr
    163 39  x86_64           0/0 cream-pbs  esrVO::cream-ce01.marie.hellasgrid.gr
    164 40  x86_64           0/0 cream-pbs  esrVO::fal-pygrid-44.lancs.ac.uk
    165 41  x86_64           0/0 cream-pbs  esrVO::hepgrid6.ph.liv.ac.uk
    166 42  x86_64           0/0 cream-pbs  esrVO::cream-ce01.ariagni.hellasgrid.gr
    167 43  x86_64           0/0 cream-pbs  esrVO::snf-189278.vm.okeanos.grnet.gr
    168 44  x86_64           0/0 cream-pbs  esrVO::snf-458754.vm.okeanos.grnet.gr
    169 45  x86_64           0/0 cream-pbs  esrVO::hepgrid5.ph.liv.ac.uk
    170 46  x86_64           0/0 cream-pbs  esrVO::cream01.kallisto.hellasgrid.gr
    171 47  x86_64           0/0 cream-pbs  esrVO::hepgrid10.ph.liv.ac.uk
    172 48  x86_64           0/0 cream-pbs  esrVO::hepgrid97.ph.liv.ac.uk
    173 }}}
    174 1. Create an identity for 7 days:
    175 {{{
    176 #!sh
    177 [user@mycomputer~]$ drm4g id esrVO init
     163CESNET_Metacloud    enabled
     164}}}
     1651. Create your remote proxy certificate
     166{{{#!sh
     167[user@mycomputer~]$ drm4g id CESNET_Metacloud init
     168--> Add '/home/user/.ssh/id_rsa' into ssh-agent for 168 hours
     169 Lifetime set to 7 days, 0:00:00
     170--> Copying '/home/user/.ssh/id_rsa' to ~/.ssh/authorized_keys file on 'ui.meteo.unican.es'
     171The copy of the public key /home/user/.ssh/id_rsa.pub has been succesfull
     172 WARNING: It is assumed that the grid certificate has been already configured
     173--> Creating '~/.drm4g/security' directory to store the proxy ...
     174Connected (version 2.0, client OpenSSH_5.3)
     175Authentication (publickey) successful!
    178176--> Create a local proxy credential ...
    179177Insert your Grid password:
    180 Your identity: /DC=es/DC=irisgrid/O=unican/CN=user
    181 Creating proxy ........................................................................................... Done
     178Your identity: <user_information>
     179Creating proxy ................................................................................... Done
    182180Proxy Verify OK
    183 Your proxy is valid until: Thu Feb 26 21:37:19 2015
    184 Your identity: /DC=es/DC=irisgrid/O=unican/CN=user
    185 Creating proxy .................................................................... Done
     181Your proxy is valid until: Thu Dec  8 12:00:00 2016
     182Your identity: <user_information>
     183Creating proxy .......................................................................................................... Done
    186184Proxy Verify OK
    187 A proxy valid for 168 hours (7.0 days) for user /DC=es/DC=irisgrid/O=unican/CN=user now exists on px.grid.sara.nl.
    188 }}}
    189 1. Check the timeleft of your identity:
    190 {{{
    191 #!sh
    192 [user@mycomputer~]$ drm4g id esrVO info
    193 --> Grid credentials
    194 subject  : /DC=es/DC=irisgrid/O=unican/CN=user/CN=proxy/CN=proxy
    195 issuer   : /DC=es/DC=irisgrid/O=unican/CN=user/CN=proxy
    196 identity : /DC=es/DC=irisgrid/O=unican/CN=user
    197 type     : full legacy globus proxy
    198 strength : 2048 bits
    199 path     : /home/user/.drm4g/security/px.grid.sara.nl
    200 timeleft : 167:57:52  (7.0 days)
    201 }}}
    202 
    203 That's it! Now, you can summit jobs to the `esr` VO. Keep in mind that you will have to renew your identity depending on the `proxy-lifetime` used.
     185A proxy valid for 168 hours (7.0 days) for user <user_information> now exists on myproxy1.egee.cesnet.cz.
     186
     187
     188Warning: your certificate and proxy will expire Thu Dec  8 12:00:00 2016
     189 which is within the requested lifetime of the proxy
     190
     191}}}
     1921. Create the VM (Virtual Machine) :
     193 * This command will create as many VMs as you have defined in "''__resources.conf__''" '''each time you execute it'''
     194{{{#!sh
     195[user@mycomputer~]$ drm4g resource create
     196'ssh-agent' is running but without any keys
     197Connected (version 2.0, client OpenSSH_5.3)
     198Authentication (publickey) successful!
     199Creating new resource
     200The proxy '~/.drm4g/security/x509up.fedcloud.egi.eu' has probably expired
     201Renewing proxy certificate
     202The proxy certificate will be operational for 24 hours
     203    Resource 'https://carach5.ics.muni.cz:11443/compute/79896' has been successfully created
     204Waiting until resource is active
     205Getting resource's IP direction
     206    Public IP: XXX.XXX.XXX.XX
     207}}}
     2081. Verify that the VM has been created and is been listed as a ''host'' :
     209{{{#!sh
     210[user@mycomputer~]$ drm4g host
     211HID ARCH       JOBS(R/T) LRMS       HOSTNAME           
     2121   x86_64           0/0 fork       CESNET_Metacloud_XXX.XXX.XXX.XX
     213
     214}}}
     2151. To send it jobs, just submit some as you'd normally do :
     216{{{#!sh
     217[user@mycomputer~]$ drm4g job submit path/to/job.job --ntasks 10
     218ARRAY ID: 0
     219
     220TASK JOB
     2210    0   
     2221    1   
     2232    2   
     2243    3   
     2254    4   
     2265    5   
     2276    6   
     2287    7   
     2298    8   
     2309    9   
     231
     232
     233
     234}}}
     2351. And then to check that it is actually doing them :
     236{{{#!sh
     237[user@mycomputer~]$ drm4g job list
     238JID DM   EM   START    END      EXEC    XFER    EXIT NAME            HOST                     
     2390   done ---- 12:07:43 12:08:14 0:00:12 0:00:07 0    date.job        CESNET_Metacloud_147.228.242.99/fork
     2401   done ---- 12:07:43 12:08:14 0:00:12 0:00:07 0    date.job        CESNET_Metacloud_147.228.242.99/fork
     2412   done ---- 12:07:43 12:08:15 0:00:21 0:00:07 0    date.job        CESNET_Metacloud_147.228.242.99/fork
     2423   done ---- 12:07:43 12:08:14 0:00:21 0:00:06 0    date.job        CESNET_Metacloud_147.228.242.99/fork
     2434   done ---- 12:07:43 12:08:15 0:00:21 0:00:07 0    date.job        CESNET_Metacloud_147.228.242.99/fork
     2445   wrap pend 12:07:43 --:--:-- 0:00:04 0:00:01 --   date.job        CESNET_Metacloud_147.228.242.99/fork
     2456   wrap pend 12:07:43 --:--:-- 0:00:05 0:00:00 --   date.job        CESNET_Metacloud_147.228.242.99/fork
     2467   wrap pend 12:07:43 --:--:-- 0:00:04 0:00:01 --   date.job        CESNET_Metacloud_147.228.242.99/fork
     2478   wrap ---- 12:07:43 --:--:-- 0:00:01 0:00:00 --   date.job        CESNET_Metacloud_147.228.242.99/fork
     2489   wrap ---- 12:07:43 --:--:-- 0:00:00 0:00:01 --   date.job        CESNET_Metacloud_147.228.242.99/fork
     249
     250}}}
     2511. To list resources, you'd usually use the command `drm4g resource list`, but that doens't show the created VMs. If you want to see all of them listed, including the created VMs, in case you want to manually connect to it for example, you can use the command option "'''--all'''"
     252{{{#!sh
     253[user@mycomputer~]$ drm4g resource list --all
     254Resources:
     255    CESNET_Metacloud
     256        communicator:  ssh
     257        username:      user
     258        frontend:      ui.macc.unican.es
     259        private key:   /home/user/.ssh/id_rsa
     260        lrms:          fedcloud
     261    CESNET_Metacloud_147.228.242.99
     262        communicator:  op_ssh
     263        username:      drm4g_admin
     264        frontend:      147.228.242.99
     265        private key:   /home/user/.ssh/id_rsa
     266        lrms:          fork
     267}}}
     268
     2691. Finally, once you're done with your VM, destroy it
     270 * This command will destroy all of the VMs you've created so far
     271{{{#!sh
     272[user@mycomputer~]$ drm4g resource destroy
     273Deleting resource https://carach5.ics.muni.cz:11443/compute/79896
     274Connected (version 2.0, client OpenSSH_5.3)
     275Authentication (publickey) successful!
     276    Resource 'https://carach5.ics.muni.cz:11443/compute/79896' has been successfully deleted
     277}}}
     278
     279
     280
     281
     282That's it! Now, you can summit jobs to '''VMs''' created using the '''EGI Federated Cloud VO'''.