wiki:DRM4G/ResourceConfiguration/EGIFedCloud

How to configure an EGI FedCloud VM

For configuring an EGI Federated Cloud Virtual Machine, follow the steps below:

First, you'll need to have access to cloud resources. To do that you'll have to follow the instructions given to access the federated services.

Then, once you've correctly installed the DRM4G, in your $DRM4G_DIR/etc directory you'll find a file called cloudsetup.json. In it you'll have to define the information needed to create your Virtual Machine(VM).

Configuration format

  • It's a JSON file, so it's syntax includes the following:
    • Data is represented in name/value pairs.
    • Curly braces hold objects and each name is followed by ':'(colon), the name/value pairs are separated by "," (commas).

By default it comes with examples using sites that have been proven to work. The list will be updated with upcoming releases.

This is an example of how the "cloudsetup.json" should look like:

{
    "rocci" : {
        "vo" : "fedcloud.egi.eu",
        "cloud_providers" : {
            "EGI FedCloud - CESNET-METACLOUD" : {
                "endpoint" : "https://carach5.ics.muni.cz:11443",
                "apps" : { 
                    "Ubuntu-14.04" : "http://occi.carach5.ics.muni.cz/occi/infrastructure/os_tpl#uuid_egi_ubuntu_server_14_04_lts_fedcloud_warg_131"
                },
                "flavours" : {   
                    "Small"  : "http://fedcloud.egi.eu/occi/compute/flavour/1.0#small",
                    "Medium" : "http://fedcloud.egi.eu/occi/compute/flavour/1.0#medium"
                }
            }
        }
    }
}

Cloud configuration keys

  • rocci: Indicates the name of the Virtual Organization (VO) being used. In the future more may be added.
    • vo: it's used for authorization, to get the proxy certificate
      • fedcloud.egi.eu: shouldn't be changed for FedCloud sites
    • cloud_providers: it will usually contain a list of sites with the VM configurations.
      • <site_name>: name that represents the site where your VM will be hosted
        • endpoint: end of a communication channel, URL of their server
        • apps: list of system images
          • <system_image_name>: OCCI ID that represents the system image you'll be using
        • flavours: list of template IDs
          • <hardware_template>: template ID that defines hardware specifications, sizes for RAM, disk and number of CPU cores

How to link the resource configuration file and the cloud setup file

There are a couple of configuration keys in "resources.conf" that reference values found in this JSON.

  • From the resource configuration file "resources.conf" you have to indicate which "cloud_providers", "apps" and "flavours" you want to use from all of the ones you have defined in the cloud setup file.
    • It is done with the "cloud_provider", "virtual_image" and "flavour" keys from the resource configuration file respectively.
  • You can modify the name (not the value) of the "<site_name>", "<system_image_name>" and "<hardware_template>", so be sure to write the same value on the resource configuration file as the name on the cloud setup file.
    • The value in "resources.conf" has to match with the name in "cloudsetup.josn"
      [CESNET_Metacloud]
      cloud_provider    = EGI FedCloud - CESNET-METACLOUD
      flavour           = Small
      virtual_image     = Ubuntu-14.04
      
      "cloud_providers" : {
          "EGI FedCloud - CESNET-METACLOUD" : {
              "endpoint" : "https://carach5.ics.muni.cz:11443",
              "apps" : { 
                  "Ubuntu-14.04" : "http://occi.carach5.ics.muni.cz/occi/infrastructure/os_tpl#uuid_egi_ubuntu_server_14_04_lts_fedcloud_warg_131"
               },
              "flavours" : {   
                  "Small"  : "http://fedcloud.egi.eu/occi/compute/flavour/1.0#small",
                  "Medium" : "http://fedcloud.egi.eu/occi/compute/flavour/1.0#medium"
              }
          } 
      }
      

How to manage VMs

  1. Customize your cloudsetup.json to match your requirements. You can search for the available system images in EGI's Applications Database Cloud Marketplace.
    • Search for the system image you want.
    • Select the one you prefer.
    • In the page that has opened up, select one of the sites that offer that same image.
    • A list of flavours will appear. Choose one and click on "get IDs".
    • After a pop-up window will appear with the information you'll have to write on the cloud setup file.
      • Site endpoint corresponds to endpoint
      • Template ID corresponds to <hardware_template>
      • OCCI ID corresponds to <system_image_name>
        { 
          "rocci" : {
            "vo" : "fedcloud.egi.eu",
            "cloud_providers" : {
              "EGI FedCloud - CESNET-METACLOUD" : {
                "endpoint" : "https://carach5.ics.muni.cz:11443",
                "apps" : { 
                  "Ubuntu-14.04" : "http://occi.carach5.ics.muni.cz/occi/infrastructure/os_tpl#uuid_egi_ubuntu_server_14_04_lts_fedcloud_warg_131"
                },
                "flavours" : {
                  "Medium" : "http://fedcloud.egi.eu/occi/compute/flavour/1.0#medium"
                }
              }
            }
          }
        }
        
  2. Configure a "rocci" resource in resources.conf.
    • The way to inform the DRM4G that a defined resource is going to be used to create VMs, is to set its lrms key to rocci.
    • This way, the DRM4G won't consider this resource as a host and so it won't send it jobs to execute.
      • The VMs created by this resource are what will be considered to be its hosts. That entails that the VMs will not be listed by the command drm4g resource list
        [user@mycomputer~]$ drm4g resource edit
        [CESNET_Metacloud]
        enable         = true
        communicator   = ssh
        vm_communicator= op_ssh
        private_key    = ~/.ssh/id_rsa
        username       = user
        vm_user        = drm4g_admin
        frontend       = ui.meteo.unican.es
        lrms           = rocci
        cloud_provider = EGI FedCloud - CESNET-METACLOUD
        myproxy_server = myproxy1.egee.cesnet.cz
        flavour        = Medium
        virtual_image  = Ubuntu-14.04
        instances      = 1
        volume         = 10
        max_jobs_running = 5
        
  3. Start the DRM4G and check if the resource has been created successfully :
    [user@mycomputer~]$ drm4g start
    Starting DRM4G .... 
      OK
    [user@mycomputer~]$ drm4g resource list
    RESOURCE            STATE               
    CESNET_Metacloud    enabled
    
  4. Create your remote proxy certificate
    [user@mycomputer~]$ drm4g id CESNET_Metacloud init
    --> Add '/home/user/.ssh/id_rsa' into ssh-agent for 168 hours
     Lifetime set to 7 days, 0:00:00
    --> Copying '/home/user/.ssh/id_rsa' to ~/.ssh/authorized_keys file on 'ui.meteo.unican.es'
    The copy of the public key /home/user/.ssh/id_rsa.pub has been succesfull
     WARNING: It is assumed that the grid certificate has been already configured
    --> Creating '~/.drm4g/security' directory to store the proxy ... 
    Connected (version 2.0, client OpenSSH_5.3)
    Authentication (publickey) successful!
    --> Create a local proxy credential ... 
    Insert your Grid password: 
    Your identity: <user_information>
    Creating proxy ................................................................................... Done
    Proxy Verify OK
    Your proxy is valid until: Thu Dec  8 12:00:00 2016
    Your identity: <user_information>
    Creating proxy .......................................................................................................... Done
    Proxy Verify OK
    A proxy valid for 168 hours (7.0 days) for user <user_information> now exists on myproxy1.egee.cesnet.cz.
    
    
    Warning: your certificate and proxy will expire Thu Dec  8 12:00:00 2016
     which is within the requested lifetime of the proxy
    
    
  5. Create the VM (Virtual Machine) :
    • This command will create as many VMs as you have defined in "resources.conf" each time you execute it
      [user@mycomputer~]$ drm4g resource create
      'ssh-agent' is running but without any keys
      Connected (version 2.0, client OpenSSH_5.3)
      Authentication (publickey) successful!
      Creating new resource
      The proxy '~/.drm4g/security/x509up.fedcloud.egi.eu' has probably expired
      Renewing proxy certificate
      The proxy certificate will be operational for 24 hours
          Resource 'https://carach5.ics.muni.cz:11443/compute/79896' has been successfully created
      Waiting until resource is active
      Getting resource's IP direction
          Public IP: XXX.XXX.XXX.XX
      
  6. Verify that the VM has been created and is been listed as a host :
    [user@mycomputer~]$ drm4g host
    HID ARCH       JOBS(R/T) LRMS       HOSTNAME            
    1   x86_64           0/0 fork       CESNET_Metacloud_XXX.XXX.XXX.XX
    
    
  7. To send it jobs, just submit some as you'd normally do:
    [user@mycomputer~]$ drm4g job submit path/to/job.job --ntasks 10
    ARRAY ID: 0
    
    TASK JOB 
    0    0   
    1    1   
    2    2   
    3    3   
    4    4   
    5    5   
    6    6   
    7    7   
    8    8   
    9    9   
    
  8. And then to check that it is actually doing them:
    [user@mycomputer~]$ drm4g job list
    JID DM   EM   START    END      EXEC    XFER    EXIT NAME            HOST                      
    0   done ---- 12:07:43 12:08:14 0:00:12 0:00:07 0    date.job        CESNET_Metacloud_XXX.XXX.XXX.XX/fork 
    1   done ---- 12:07:43 12:08:14 0:00:12 0:00:07 0    date.job        CESNET_Metacloud_XXX.XXX.XXX.XX/fork 
    2   done ---- 12:07:43 12:08:15 0:00:21 0:00:07 0    date.job        CESNET_Metacloud_XXX.XXX.XXX.XX/fork 
    3   done ---- 12:07:43 12:08:14 0:00:21 0:00:06 0    date.job        CESNET_Metacloud_XXX.XXX.XXX.XX/fork 
    4   done ---- 12:07:43 12:08:15 0:00:21 0:00:07 0    date.job        CESNET_Metacloud_XXX.XXX.XXX.XX/fork 
    5   wrap pend 12:07:43 --:--:-- 0:00:04 0:00:01 --   date.job        CESNET_Metacloud_XXX.XXX.XXX.XX/fork 
    6   wrap pend 12:07:43 --:--:-- 0:00:05 0:00:00 --   date.job        CESNET_Metacloud_XXX.XXX.XXX.XX/fork 
    7   wrap pend 12:07:43 --:--:-- 0:00:04 0:00:01 --   date.job        CESNET_Metacloud_XXX.XXX.XXX.XX/fork 
    8   wrap ---- 12:07:43 --:--:-- 0:00:01 0:00:00 --   date.job        CESNET_Metacloud_XXX.XXX.XXX.XX/fork 
    9   wrap ---- 12:07:43 --:--:-- 0:00:00 0:00:01 --   date.job        CESNET_Metacloud_XXX.XXX.XXX.XX/fork 
    
    
  9. To list resources, you'd usually use the command drm4g resource list, but that doesn't show the created VMs. If you want to see all of them listed, including the created VMs, in case you want to manually connect to it for example, you can use the command option "--all":
    [user@mycomputer~]$ drm4g resource list --all
    Resources:
        CESNET_Metacloud
            communicator:  ssh
            username:      user
            frontend:      ui.meteo.unican.es
            private key:   /home/user/.ssh/id_rsa
            lrms:          rocci
        CESNET_Metacloud_XXX.XXX.XXX.XX
            communicator:  op_ssh
            username:      drm4g_admin
            frontend:      XXX.XXX.XXX.XX
            private key:   /home/user/.ssh/id_rsa
            lrms:          fork
    
  1. Finally, once you're done with your VM, destroy it.
    • This command will destroy all of the VMs you've created so far:
      [user@mycomputer~]$ drm4g resource destroy
      Deleting resource https://carach5.ics.muni.cz:11443/compute/79896
      Connected (version 2.0, client OpenSSH_5.3)
      Authentication (publickey) successful!
          Resource 'https://carach5.ics.muni.cz:11443/compute/79896' has been successfully deleted
      

That's it! Now, you can summit jobs to VMs created using the EGI Federated Cloud VO.

Last modified 5 years ago Last modified on Dec 23, 2016 9:17:25 PM

Attachments (5)

Download all attachments as: .zip