Version 10 (modified by minondoa, 6 years ago) (diff) |
---|
How to configure an EGI FedCloud VM
For configuring an EGI Federated Cloud Virtual Machine, follow the steps below:
First, you'll need to have access to cloud resources. To do that you'll have to follow the instructions given to access the federated services.
Then, once you've correctly installed the DRM4G, in your $DRM4G_DIR/etc directory you'll find a file called cloudsetup.json. In it you'll have to define the information needed to create your Virtual Machine(VM).
Configuration format
- It's a JSON file, so it's syntax includes the following:
- Data is represented in name/value pairs.
- Curly braces hold objects and each name is followed by ':'(colon), the name/value pairs are separated by "," (commas).
By default it comes with examples using sites that have been proven to work. The list will be updated with upcoming releases.
This is an example of how the "cloudsetup.json" should look like:
{ "rocci" : { "vo" : "fedcloud.egi.eu", "cloud_providers" : { "EGI FedCloud - CESNET-METACLOUD" : { "endpoint" : "https://carach5.ics.muni.cz:11443", "apps" : { "Ubuntu-14.04" : "http://occi.carach5.ics.muni.cz/occi/infrastructure/os_tpl#uuid_egi_ubuntu_server_14_04_lts_fedcloud_warg_131" }, "flavours" : { "Small" : "http://fedcloud.egi.eu/occi/compute/flavour/1.0#small", "Medium" : "http://fedcloud.egi.eu/occi/compute/flavour/1.0#medium" } } } } }
Cloud configuration keys
- rocci: Indicates the name of the Virtual Organization (VO) being used. In the future more may be added.
- vo: it's used for authorization, to get the proxy certificate
- fedcloud.egi.eu: shouldn't be changed for FedCloud sites
- cloud_providers: it will usually contain a list of sites with the VM configurations.
- <site_name>: name that represents the site where your VM will be hosted
- endpoint: end of a communication channel, URL of their server
- apps: list of system images
- <system_image_name>: OCCI ID that represents the system image you'll be using
- flavours: list of template IDs
- <hardware_template>: template ID that defines hardware specifications, sizes for RAM, disk and number of CPU cores
- <site_name>: name that represents the site where your VM will be hosted
- vo: it's used for authorization, to get the proxy certificate
How to link the resource configuration file and the cloud setup file
There are a couple of configuration keys in "resources.conf" that reference values found in this JSON.
- From the resource configuration file "resources.conf" you have to indicate which "cloud_providers", "apps" and "flavours" you want to use from all of the ones you have defined in the cloud setup file.
- It is done with the "cloud_provider", "virtual_image" and "flavour" keys from the resource configuration file respectively.
- You can modify the name (not the value) of the "<site_name>", "<system_image_name>" and "<hardware_template>", so be sure to write the same value on the resource configuration file as the name on the cloud setup file.
- The value in "resources.conf" has to match with the name in "cloudsetup.josn"
[CESNET_Metacloud] cloud_provider = EGI FedCloud - CESNET-METACLOUD flavour = Small virtual_image = Ubuntu-14.04
"cloud_providers" : { "EGI FedCloud - CESNET-METACLOUD" : { "endpoint" : "https://carach5.ics.muni.cz:11443", "apps" : { "Ubuntu-14.04" : "http://occi.carach5.ics.muni.cz/occi/infrastructure/os_tpl#uuid_egi_ubuntu_server_14_04_lts_fedcloud_warg_131" }, "flavours" : { "Small" : "http://fedcloud.egi.eu/occi/compute/flavour/1.0#small", "Medium" : "http://fedcloud.egi.eu/occi/compute/flavour/1.0#medium" } } }
- The value in "resources.conf" has to match with the name in "cloudsetup.josn"
How to manage VMs
- Customize your cloudsetup.json to match your requirements. You can search for the available system images in EGI's Applications Database Cloud Marketplace.
- Search for the system image you want.
- Select the one you prefer.
- In the page that has opened up, select one of the sites that offer that same image.
- A list of flavours will appear. Choose one and click on "get IDs".
- After a pop-up window will appear with the information you'll have to write on the cloud setup file.
- Site endpoint corresponds to endpoint
- Template ID corresponds to <hardware_template>
- OCCI ID corresponds to <system_image_name>
{ "rocci" : { "vo" : "fedcloud.egi.eu", "cloud_providers" : { "EGI FedCloud - CESNET-METACLOUD" : { "endpoint" : "https://carach5.ics.muni.cz:11443", "apps" : { "Ubuntu-14.04" : "http://occi.carach5.ics.muni.cz/occi/infrastructure/os_tpl#uuid_egi_ubuntu_server_14_04_lts_fedcloud_warg_131" }, "flavours" : { "Medium" : "http://fedcloud.egi.eu/occi/compute/flavour/1.0#medium" } } } } }
- Search for the system image you want.
- Configure a "rocci" resource in resources.conf.
- The way to inform the DRM4G that a defined resource is going to be used to create VMs, is to set its lrms key to rocci.
- This way, the DRM4G won't consider this resource as a host and so it won't send it jobs to execute.
- The VMs created by this resource are what will be considered to be its hosts. That entails that the VMs will not be listed by the command drm4g resource list
[user@mycomputer~]$ drm4g resource edit [CESNET_Metacloud] enable = true communicator = ssh vm_communicator= op_ssh private_key = ~/.ssh/id_rsa username = user vm_user = drm4g_admin frontend = ui.meteo.unican.es lrms = rocci cloud_provider = EGI FedCloud - CESNET-METACLOUD myproxy_server = myproxy1.egee.cesnet.cz flavour = Medium virtual_image = Ubuntu-14.04 instances = 1 volume = 10 max_jobs_running = 5
- The VMs created by this resource are what will be considered to be its hosts. That entails that the VMs will not be listed by the command drm4g resource list
- Start the DRM4G and check if the resource has been created successfully :
[user@mycomputer~]$ drm4g start Starting DRM4G .... OK [user@mycomputer~]$ drm4g resource list RESOURCE STATE CESNET_Metacloud enabled
- Create your remote proxy certificate
[user@mycomputer~]$ drm4g id CESNET_Metacloud init --> Add '/home/user/.ssh/id_rsa' into ssh-agent for 168 hours Lifetime set to 7 days, 0:00:00 --> Copying '/home/user/.ssh/id_rsa' to ~/.ssh/authorized_keys file on 'ui.meteo.unican.es' The copy of the public key /home/user/.ssh/id_rsa.pub has been succesfull WARNING: It is assumed that the grid certificate has been already configured --> Creating '~/.drm4g/security' directory to store the proxy ... Connected (version 2.0, client OpenSSH_5.3) Authentication (publickey) successful! --> Create a local proxy credential ... Insert your Grid password: Your identity: <user_information> Creating proxy ................................................................................... Done Proxy Verify OK Your proxy is valid until: Thu Dec 8 12:00:00 2016 Your identity: <user_information> Creating proxy .......................................................................................................... Done Proxy Verify OK A proxy valid for 168 hours (7.0 days) for user <user_information> now exists on myproxy1.egee.cesnet.cz. Warning: your certificate and proxy will expire Thu Dec 8 12:00:00 2016 which is within the requested lifetime of the proxy
- Create the VM (Virtual Machine) :
- This command will create as many VMs as you have defined in "resources.conf" each time you execute it
[user@mycomputer~]$ drm4g resource create 'ssh-agent' is running but without any keys Connected (version 2.0, client OpenSSH_5.3) Authentication (publickey) successful! Creating new resource The proxy '~/.drm4g/security/x509up.fedcloud.egi.eu' has probably expired Renewing proxy certificate The proxy certificate will be operational for 24 hours Resource 'https://carach5.ics.muni.cz:11443/compute/79896' has been successfully created Waiting until resource is active Getting resource's IP direction Public IP: XXX.XXX.XXX.XX
- This command will create as many VMs as you have defined in "resources.conf" each time you execute it
- Verify that the VM has been created and is been listed as a host :
[user@mycomputer~]$ drm4g host HID ARCH JOBS(R/T) LRMS HOSTNAME 1 x86_64 0/0 fork CESNET_Metacloud_XXX.XXX.XXX.XX
- To send it jobs, just submit some as you'd normally do:
[user@mycomputer~]$ drm4g job submit path/to/job.job --ntasks 10 ARRAY ID: 0 TASK JOB 0 0 1 1 2 2 3 3 4 4 5 5 6 6 7 7 8 8 9 9
- And then to check that it is actually doing them:
[user@mycomputer~]$ drm4g job list JID DM EM START END EXEC XFER EXIT NAME HOST 0 done ---- 12:07:43 12:08:14 0:00:12 0:00:07 0 date.job CESNET_Metacloud_XXX.XXX.XXX.XX/fork 1 done ---- 12:07:43 12:08:14 0:00:12 0:00:07 0 date.job CESNET_Metacloud_XXX.XXX.XXX.XX/fork 2 done ---- 12:07:43 12:08:15 0:00:21 0:00:07 0 date.job CESNET_Metacloud_XXX.XXX.XXX.XX/fork 3 done ---- 12:07:43 12:08:14 0:00:21 0:00:06 0 date.job CESNET_Metacloud_XXX.XXX.XXX.XX/fork 4 done ---- 12:07:43 12:08:15 0:00:21 0:00:07 0 date.job CESNET_Metacloud_XXX.XXX.XXX.XX/fork 5 wrap pend 12:07:43 --:--:-- 0:00:04 0:00:01 -- date.job CESNET_Metacloud_XXX.XXX.XXX.XX/fork 6 wrap pend 12:07:43 --:--:-- 0:00:05 0:00:00 -- date.job CESNET_Metacloud_XXX.XXX.XXX.XX/fork 7 wrap pend 12:07:43 --:--:-- 0:00:04 0:00:01 -- date.job CESNET_Metacloud_XXX.XXX.XXX.XX/fork 8 wrap ---- 12:07:43 --:--:-- 0:00:01 0:00:00 -- date.job CESNET_Metacloud_XXX.XXX.XXX.XX/fork 9 wrap ---- 12:07:43 --:--:-- 0:00:00 0:00:01 -- date.job CESNET_Metacloud_XXX.XXX.XXX.XX/fork
- To list resources, you'd usually use the command drm4g resource list, but that doesn't show the created VMs. If you want to see all of them listed, including the created VMs, in case you want to manually connect to it for example, you can use the command option "--all":
[user@mycomputer~]$ drm4g resource list --all Resources: CESNET_Metacloud communicator: ssh username: user frontend: ui.meteo.unican.es private key: /home/user/.ssh/id_rsa lrms: rocci CESNET_Metacloud_XXX.XXX.XXX.XX communicator: op_ssh username: drm4g_admin frontend: XXX.XXX.XXX.XX private key: /home/user/.ssh/id_rsa lrms: fork
- Finally, once you're done with your VM, destroy it.
- This command will destroy all of the VMs you've created so far:
[user@mycomputer~]$ drm4g resource destroy Deleting resource https://carach5.ics.muni.cz:11443/compute/79896 Connected (version 2.0, client OpenSSH_5.3) Authentication (publickey) successful! Resource 'https://carach5.ics.muni.cz:11443/compute/79896' has been successfully deleted
- This command will destroy all of the VMs you've created so far:
That's it! Now, you can summit jobs to VMs created using the EGI Federated Cloud VO.
Attachments (5)
- searchSystemImage.png (6.8 KB) - added by minondoa 6 years ago.
- selectFlavour.png (43.4 KB) - added by minondoa 6 years ago.
- selectSite.png (14.2 KB) - added by minondoa 6 years ago.
- selectSystemImage.png (39.6 KB) - added by minondoa 6 years ago.
- systemImageInfo.png (22.9 KB) - added by minondoa 6 years ago.
Download all attachments as: .zip