Configuring Task Manager (or WorkflowMaster)

The task manager generates tasks for each stage of the workflow and inserts them in the CometCloud space, which allows different sites query and retrieve the tasks they need. It offers an API for users to programmatically define the way tasks are generated. The Task Manager a service built using CometCloud master/worker programming interface. Hence, it requires deploying CometCloud core components, such as CometCloud space, request handler, and proxy. CometCloud can be deployed across multiple machines, providing scalability for the management, query, and delivery of tasks. For simplicity in this section we deploy all CometCloud services within one single machine. More information about how to configure and deploy CometCloud can be found in the CometCloud website

This service requires several configuration files: master.properties, comet.properties, nodeFile, exceptionFile, RequestHandlerList, portFile, chord.properties, and squid.properties. Next we present configuration where all services are going to be deployed in the machine machine2.domain.com. The Task Manager will be listening for instructions from the Workflow Manager and Autonomic Scheduler in the port defined by workflowMasterPort. More information about this section configuration files can be found in MASTER section. For simplicity we keep scripts and examples of the configuration files in the directory simple_run/master/

Requirements

This service requires Java runtime 1.7+.

Edit master.properties file

This file contains basic information about the Task Manager.

workflowMasterPort=7777
logFile=master.log
logFileRH=requesthandler.log

Edit comet.properties file

This file contains information required by CometCloud. Only TaskMonitoringPeriod and IsolatedProxy require to be modified. TaskMonitoringPeriod indicates an interval of time in milliseconds that tells CometCloud when to look for missing tasks and reinsert them into the system. This can be disabled by indicating 0 as value. The IsolatedProxy indicates the machine that has the CometCloud proxy and hence enables access to query and retrieve tasks from the CometCloud Space. This will be used by the Agents. By default the port used is 12342. This can be modified as described in the Customizing CometCloud ports Section.

MasterClass=tassl.application.cometcloud.WorkflowMaster
WorkerClass=tassl.application.cometcloud.AppWorker
TaskClass=tassl.application.cometcloud.WorkflowTaskTuple

RoutingKeys=TaskId

TaskMonitoringPeriod=50000

IsolatedProxy=machine2.domain.com

Note

Remember to send the comet.properties file to the Agents every time it is modified.

Edit nodeFie file

This file tells CometCloud where to start the services. The number 2 tells CometCloud that it has to start two peers (i.e. Request Handler and a Master).

machine2.domain.com:2

Edit portFie file

This file is used by CometCloud to give ports to the different peers. It has to have as many ports as peers in CometCloud. In this case we only need two.

5555
5556

Edit exceptionFile file

This file is to set the roles of each CometCloud peer and their ports.

machine2.domain.com:5555
comet.NodeType=MASTER
machine2.domain.com:5556
comet.NodeType=REQUEST_HANDLER

Edit RequestHandlerList file

machine2.domain.com

Note

Files chord.properties and squid.properties should not be modified.

Customizing Additional CometCloud Ports (Optional)

CometCloud makes use of additional ports to manage its different services. This step is not mandatory, but it provides information about port usage that can be useful.

  • Ports defined in the comet.properties file.

    • IsolatedProxyPort. This port is used by Agents to contact CometCloud proxy and request tasks. Its default value is 12342.
    • REQUEST_HANDLER_PORT. This port is used by the CometCloud proxy to contact a request handler and communicate Agents’ task requests. Its default value is 12343.
    • TCP_CONTROL_PORT. This port is used by Agents to communicate completed tasks and their result to the Task Manager. Its default value is 12340.
    • COMET_CONTROL_PORT. This port is used by CometCloud to control its services. Its default value is 12330.

  • Overlay Ports need to be changed in the starting command of the Proxy and Task Manager, startProxy.sh (simply indicate port at the end) and startWorkflowMaster.sh (-overlayPort option), respectively.

    • Overlay Server Port. This port is used by CometCloud to start the CometCloud space and assign roles to different peers. Its default value is 4444.

Starting the Service

Once the configuration is ready, you can start the Task Manager and CometCloud by executing the script startAll.sh. This script contains the following code.

#!/bin/bash
./startOverlayServer.sh &
./startProxy.sh &
./startWorkflowMaster.sh

  • ./startOverlayServer.sh starts the CometCloud overlay, which is required to create the CometCloud space. This script contains the following code.

    export CLASSPATH=../../dist/*:../../lib/*
java -Xms256m -Xmx1024m -cp $CLASSPATH tassl.automate.overlay.OverlayControlServer 4444

  • ./startProxy.sh starts the CometCloud proxy, which enables access to query and retrieve tasks from the CometCloud Space. This script contains the following code.

    export CLASSPATH=../../dist/*:../../lib/*
java -Xms256m -Xmx1024m -cp $CLASSPATH tassl.application.node.isolate.RequestHandlerProxyWfl -propertyFile comet.properties

  • ./startWorkflowMaster.sh starts the Task Manager as a CometCloud application.

    export CLASSPATH=../../dist/*:../../lib/*
java -Xms256m -Xmx1024m -cp $CLASSPATH tassl.application.cometcloud.WorkflowStarter -nodeFile nodeFile -portFile portFile -exceptionFile exceptionFile -propertyFile chord.properties -propertyFile squid.properties -propertyFile comet.properties -overlayPort 4444 -propertyFile master.properties

Note

These services can start regardless the status of other services of our federation.