Agent¶

Requirements¶

This service requires Java runtime 1.7+, rsync, and python 2.6+.

Configure Agent machine¶

1. Configure Agent machine to automatically trust worker machines (SSH first handshake). First make sure you have a directory called .ssh in your home directory, if not you can create it executing mkdir ~/.ssh. Then create a file inside that directory called config (file should be ~/.ssh/config)

   Host *
       StrictHostKeyChecking no
       ForwardAgent yes
   

2. Typically we would want to retrieve data from other Agents. In this way we can make use of intermediate results that are available at remote sites or we can exploit data locality by retrieving the copy of a file that is closest to us. For this we should be able to ssh with no password between Agents.

   • Generate sshkey in each agent by executing ssh-keygen. Press enter to all questions and do not establish a passphrase.
   • Add public ssh key to all other Agents and data storage sites from which Agents may retrieve data. Copy the content of the file ~/.ssh/id_rsa.pub and place insert it in the file ~/.ssh/authorized_keys of all other Agents. If ~/.ssh/authorized_keys does not exist, you can create it and make sure that the permissions are 0600 (chmod 0600 ~/.ssh/authorized_keys).

Edit comet.properties file¶

This file must be exactly the SAME as the one created in the Task Manager’s comet.properties file. As we described before, it contains information required by CometCloud. The property IsolatedProxy tells the Agent where to request tasks. As we mentioned in Customizing Additional CometCloud Ports section, additional properties are required if we want to customize the CometCloud ports.

MasterClass=tassl.application.cometcloud.WorkflowMaster
WorkerClass=tassl.application.cometcloud.AppWorker
TaskClass=tassl.application.cometcloud.WorkflowTaskTuple

RoutingKeys=TaskId

TaskMonitoringPeriod=50000

IsolatedProxy=machine2.domain.com

Note

Remember that if you change comet.properties in the Task Manager/WorkflowMaster, you need to copy it to every Agent and restart the Agent.

Edit agent.properties file¶

This file contains the information required to configure the different services of an Agent. We are going to explain the content of this file by sections. More information about this section configuration files can be found in AGENT section.

• Configure Task Service. The following properties define the machine where the Agent is running (publicIpAgent) and the port of the Task Service (portAgent). Two optional parameters are IpAgentForWorkers and portAgentForWorkers to specify a local IP and port that can be used for the workers to interact with the Task Service. The property MaxNoTaskRetries allows Agents to send workers to sleep when there are no more tasks of the type that the worker is requesting. Agents can awake workers if any change occurs, such a rescheduling.

  publicIpAgent=machine3.domain.com
  portAgent=8880
  
  IpAgentForWorkers=192.0.1.2
  portAgentForWorkers=6666
  
  #Number of times a worker uses a query unsuccessfully.
  MaxNoTaskRetries=100
  
  logFile=Agent.log
  

• File Server configuration. This server is used to transfer files automatically to the resources when the workers need them. It uses rsync for that. The server will be listening on the port FileServerPort and it will be able to handle FileServerMaxThreads concurrent requests. We also need to define the staging directory (StageFileDir) where input files and intermediate results can be temporarily stored.

  StartFileServer=true
  FileServerPort=6668
  FileServerMaxThreads=10
  StageFileDir=/tmp/stage/
  

• Local Resource Management Service configuration. This service is used to dynamically provision resources. It will be listening in the port MgmtPort and it also defines a monitoring interval of MonitorInterval that is used to monitor status of resources. An Agent needs to know which resource manager (i.e. autonomic scheduler) has to register. This is specified by the property CentralManagerServer, as we can see it has the same values as the one defined in the Workflow Manager configuration Section

  StartManager=true
  MgmtPort=9889
  CentralManagerServer=machine1.domain.com:7778
  MonitorInterval=30
  

• Resource configuration. In this property we define the type of resources this agent is going to be managing. Currently we support two types, namely cloud and cluster. A cloud requires provisioning virtual machines (VMs) before workers can start, while in a cluster we assume that the machines are available (e.g., a pilot job has booked machines) and we can simply start workers.

  • Cluster definition. clusterDell is the name of the file that has the specific details of this cluster.

  Resources=cluster:clusterDell
  

  • Cloud definition. cloudOSSierra is the name of the file that has the specific details of this cloud.

  Resources=cloud:cloudOSSierra
  

• Metrics service configuration. Next we specify the information regarding the metrics service, which will be used to store runtime information regarding this agent (e.g., application execution times, failure rates, etc.). More information about the metrics service can be found in the Metrics Service configuration Section. The use of the metrics service is optional (disabled by default) as the system could operate using static information. To enable the use of the metrics service we define the following properties: UseMeticsService equals true, AgentMetricsAddress equals to the IP or name hosting the metrics service, AgentMetricsPort equals to the port where the metrics service is listening, and DBName indicates the name of the database that will store the information of this agent.

  UseMeticsService=true
  AgentMetricsAddress=metrics.domain.com
  AgentMetricsPort=8891
  DBName=agentmetricsSpring
  

Configuring a Cluster (clusterDell)¶

This configuration file shows an example of how to configure a cluster. We are going to explain the content of this file by sections.

• General information about the site. It has the Name and Zone, which identifies the site and it is used to enforce the constraints established in the input data sources (see Workflow Definition). Currently, we assume that the machines of the cluster are reserved to us (e.g., pilot job), hence QueueType and QueueName are not in use. In the future they will be used to enable dynamic reservation of machines.

  Name=siteDell02
  Zone=zoneB
  
  QueueType=Torque
  QueueName=regular
  

• Resource information. In this section we specify attributes of the site:

  Overhead=5
  WorkerLimit=192.168.2.156:1;192.168.2.157:1
  Cost=192.168.2.156:0.6;192.168.2.157:0.12
  BenchmarkScore=192.168.2.156:30000;192.168.2.157:30000
  
  workerPortRange=7777:7788
  

  • Overhead. Overhead to provision a machine. This value might change over time if the Metrics service is enabled.
  • WorkerLimit. Number of workers per machine. It is recommended to use only one per machine, more than one is experimentally supported and might have bugs. In the example we have two machines and we can provision one worker in each one.
  • Cost of a machine per unit of time (i.e. hour). We can specify a different cost for each machine. In this case the machine 192.168.2.156 costs 0.6 dollars/SU per hour and the machine 192.168.2.157 costs 0.12 dollars/SUs per hour.
  • BenchmarkScore. Benchmark score of each machine. This is used to estimate the performance of each machine. Any benchmark can be used for this, e.g., the Whetstone Score (UnixBench) for a t2.medium instance in AWS is 8200. This value is only used when there is no real data regarding to a specific application. Once the agent has real data this score is ignored. Note that in order for the agent to obtain real data from application executions the Metrics service must be enabled.
  • workerPortRange, are the ports to be used inside the provisioned machines to start the workers.

  Note

  Agent machine must be able to contact the workers in workerPortRange range of ports (i.e. ports in machines hosting workers must be OPEN).

• Application information. We specify a list of applications this resource can handle, and the CometCloud worker class that contains the logic. The name of this applications are the ones that need to be used in the Application field of the workflow definition (see Defining Workflows).

  SupportedApps=montage,mapreduce
  montage=tassl.application.cometcloud.sampleMontage.AppWorkerMontage
  mapreduce=tassl.application.cometcloud.sample.AppWorker
  

• Internal Resource Information. This information is regarding to the worker machines. Therefore make sure that the paths refer to those inside those machines.

  SoftwareDirWorker=/cac/soft/cometcloud/
  UserWorker=med208
  WorkingDir=/tmp/work/
  

  • SoftwareDirWorker. Software directory in worker machine. Directory where the software is expected to be in the resource for each application. For example, the software of montage is expected to be in the path SoftwareDirWorker/montage/. Inside that directory we must have any binary required by our application plus the dist and lib directories with the CometCloud jar files as well as the specific worker application jar.
  • UserWorker. User in worker machine. This is the user that we will use to ssh into the worker machine. It must be configure to enable ssh with no password.
  • WorkingDir. Working directory in worker machine. Directory inside the worker machine that will be used to transfer input files and write any data generated during the execution of the application.

  Note

  Remember agents must be able to ssh with no password to the worker machines (i.e. ssh med208@192.168.2.156 should not ask for a password)

Configuring a Cloud (cloudOSSierra)¶

This configuration file shows an example of how to configure a cloud. We are going to explain the content of this file by sections.

• General information about the site.

  Name=siteSierra
  Zone=zoneA
  key=/N/u/jdiaz/OS-grizzly/ec2/jdiaznova.pem
  CloudScript=../../scripts/cloud.py
  ProviderType=openstack_ec2
  Region=nova
  ProviderConfigFile=/N/u/jdiaz/OS-grizzly/ec2/eucarc
  

  • Name and Zone, which identifies the site and it is used to enforce the constraints established in the input data sources (see Workflow Definition).

  • key. This key is used to interact with the virtual machines (VMs) of the cloud. Typically, when using cloud technology you need to specify the a keypair that will allow ssh to the VM.

  • CloudScript. Path to the plugin that enables provisioning and deprovisioning cloud resources. Currently we provide with one called cloud.py that supports some of the major cloud platforms.

  • ProviderType. Since the CloudScript supports multiple platforms, we need to specify which one we will be using. The following table summarizes the current options.

    ProviderType	Platform	API-used
    openstack_ec2	OpenStack EC2	boto
    openstack_nova	OpenStack Nova	novaclient
    nimbus_ec2	Nimbus EC2	boto
    aws_ec2	AWS EC2	boto

  • Region. Cloud providers usually require to specify a region in which you will be operating. Some examples of current platforms are described in the following table.

    Provider	Region
    FutureSystems	nova
    Chameleon (Alamo)	regionOne
    AWS (US West -Oregon)	us-west-2

  • ProviderConfigFile. This file contains the configuration provided for your cloud provider. Some examples are provided in Cloud configuration Files section.

  Note

  The key file must have the same name as the keypair in the cloud (you can see your cloud keypairs in the portal-security or using the command line, e.g., euca-describe-keypairs, nova keypait-list). The extension (.pem in this case) is not important. Also make sure that the permission of the key file are correct (e.g., chmod 0600 jdiaznova.pem).

• Resource information. In this section we specify attributes of the site:

      Overhead=20
  VMLimits=m1.small:2;m1.medium:5;m1.large:3
      WorkerLimit=m1.small:1;m1.medium:1;m1.large:1
      Cost=m1.small:0.06;m1.medium:0.12;m1.large:0.24
      BenchmarkScore=t2.small:4050;t2.medium:8200;t2.large:16000
      CostDataIn=0.01
      CostDataOut=0.12
      workerPortRange=7777:7888
  

  • Overhead. It is the overhead to provision a machine. This value might change over time if the Metrics service is enabled.
  • VMLimits. Determine the type of virtual machine (VM) supported and the number of each type that is available. Format is <type>:<number>;<type>:<number>. In our example, this site will be able to launch a maximum of 2 VMs of m1.small type, 5 VMs of m1.medium type, and 3 VMs of m1.large type.
  • WorkerLimit. Number of workers per type of VM. It is recommended to use only one per machine, more than one is experimentally supported and might have bugs. In the example we have three types of VMs and we can provision one worker in each one.
  • Cost. Cost of a VM per unit of time (i.e. hour). We can specify a different cost for each machine. In this case VMs of type m1.small cost 0.06 dollars per hour, m1.medium VMs cost 0.12 dollars per hour, and m1.large Vms cost 0.24 dollars per hour.
  • BenchmarkScore. Benchmark score of each machine. This is used to estimate the performance of each machine. Any benchmark can be used for this, e.g., the Whetstone Score (UnixBench) for a t2.medium instance in AWS is 8200. This value is only used when there is no real data regarding to a specific application. Once the agent has real data this score is ignored. Note that in order for the agent to obtain real data from application executions the Metrics service must be enabled.
  • CostDataIn. Cost of transferring data inside the site. Cost per GB of data.
  • CostDataOut. Cost of transferring data outside the site. Cost per GB of data.
  • workerPortRange, are the ports to be used inside the provisioned machines to start the workers. Format is <initialPort>:<lastPort>

  Note

  Agent machine must be able to contact the workers in the workerPortRange range of ports (i.e. ports in VMs hosting workers must be OPEN). This is typically done in changing options in the security groups of your cloud.

• Application information. We specify which application this resource can handle, and the CometCloud worker class that contains the logic. The name of this applications are the ones that need to be used in the Application field of the workflow definition (see Defining Workflows).

  SupportedApps=montage,mapreduce
  montage=tassl.application.cometcloud.sampleMontage.AppWorkerMontage
  mapreduce=tassl.application.cometcloud.sample.AppWorker
  
  defaultImageId=ami-0000003a
  mapreduceImageId=ami-0000003b
  

  • defaultImageId. This is the id of the VM image that will be deployed by default as a worker machine. This VM image has to be prepared beforehand with the proper software and libraries. More information is provided in Cloud VM image configuration section.
  • We can also specify different VMs images for different applications. The format is <application>ImageId. In our example, we have a specific image for the mapreduce application (mapreduceImageId), while the montage application uses the default one.
  • Using the right VM image ID, if you use ec2 you need to obtain the id using euca-describe-images, and if you use nova you need to obtain it using nova image-list.

• Internal Resource Information. This information is regarding to the worker machines. Therefore make sure that the paths refer to those inside those machines.

  SoftwareDirWorker=/home/ubuntu/
  UserWorker=ubuntu
  WorkingDir=/home/ubuntu/
  

  • Software directory in worker machine. Directory where the software is expected to be in the resource for each application. For example, the software of montage is expected to be in the path SoftwareDirWorker/montage/. Inside that directory we must have any binary required by our application plus the dist and lib directories with the CometCloud jar files as well as the specific worker application jar.
  • User in worker machine. This is the user that we will use to ssh into the worker machine.
  • Working directory in worker machine

  Note

  Remember agents must be able to ssh with no password to the worker VMs. This is typically achieved automatically in cloud infrastructures using the previous keypair. (e.g., ssh -i jdiazkey.pem ubuntu@54.187.182.145)

Starting the Service¶

Once the configuration is ready, you can start the agent by executing the script startAgent.sh. This script contains the following code that defines the CLASSPATH and executes the appropriated java class.

export CLASSPATH=../../dist/*:../../lib/*
java -Xms256m -Xmx1024m -cp $CLASSPATH tassl.application.cometcloud.AgentLite -propertyFile comet.properties -propertyFile agent.properties

Note

Agents register with the Autonomic Scheduler, hence you need to make sure that the Workflow Manager and Autonomic Scheduler are running before you start an Agent.