ARC SDK
|
Class for submitting jobs. More...
#include <arc/compute/Submitter.h>
Public Member Functions | |
Submitter (const UserConfig &uc) | |
Create a Submitter object. More... | |
void | addConsumer (EntityConsumer< Job > &addConsumer_consumer) |
Add a Job consumer. More... | |
void | removeConsumer (EntityConsumer< Job > &removeConsumer_consumer) |
Remove a previous added consumer object. More... | |
SubmissionStatus | Submit (const Endpoint &endpoint, const JobDescription &desc) |
Submit job to endpoint. More... | |
SubmissionStatus | Submit (const Endpoint &endpoint, const JobDescription &desc, Job &job) |
Submit job to endpoint. More... | |
SubmissionStatus | Submit (const Endpoint &endpoint, const std::list< JobDescription > &descs) |
Submit jobs to endpoint. More... | |
SubmissionStatus | Submit (const Endpoint &endpoint, const std::list< JobDescription > &descs, std::list< Job > &jobs) |
Submit jobs to endpoint. More... | |
SubmissionStatus | Submit (const std::list< Endpoint > &endpoint, const std::list< JobDescription > &descs) |
Submit jobs to any endpoints. More... | |
SubmissionStatus | Submit (const std::list< Endpoint > &endpoint, const std::list< JobDescription > &descs, std::list< Job > &jobs) |
Submit jobs to any endpoints. More... | |
SubmissionStatus | Submit (const ExecutionTarget &et, const JobDescription &desc) |
Submit job to ExecutionTarget (computing service) More... | |
SubmissionStatus | Submit (const ExecutionTarget &et, const JobDescription &desc, Job &job) |
Submit job to ExecutionTarget (computing service) More... | |
SubmissionStatus | Submit (const ExecutionTarget &et, const std::list< JobDescription > &descs) |
Submit jobs to ExecutionTarget (computing service) More... | |
SubmissionStatus | Submit (const ExecutionTarget &et, const std::list< JobDescription > &descs, std::list< Job > &jobs) |
Submit jobs to ExecutionTarget (computing service) More... | |
SubmissionStatus | BrokeredSubmit (const std::list< std::string > &endpoints, const std::list< JobDescription > &descs, const std::list< std::string > &requestedSubmissionInterfaces=std::list< std::string >()) |
Submit jobs to matching and ranked computing services. More... | |
SubmissionStatus | BrokeredSubmit (const std::list< std::string > &endpoints, const std::list< JobDescription > &descs, std::list< Job > &jobs, const std::list< std::string > &requestedSubmissionInterfaces=std::list< std::string >()) |
Submit jobs to matching and ranked computing services. More... | |
SubmissionStatus | BrokeredSubmit (const std::list< Endpoint > &endpoints, const std::list< JobDescription > &descs, const std::list< std::string > &requestedSubmissionInterfaces=std::list< std::string >()) |
Submit jobs to matching and ranked computing services. More... | |
SubmissionStatus | BrokeredSubmit (const std::list< Endpoint > &endpoints, const std::list< JobDescription > &descs, std::list< Job > &jobs, const std::list< std::string > &requestedSubmissionInterfaces=std::list< std::string >()) |
Submit jobs to matching and ranked computing services. More... | |
const std::list< const JobDescription * > & | GetDescriptionsNotSubmitted () const |
Get job descriptions not submitted. More... | |
void | ClearNotSubmittedDescriptions () |
Clear list of not submitted job descriptions. More... | |
const EndpointStatusMap & | GetEndpointQueryingStatuses () const |
Get status map for queried endpoints. More... | |
void | ClearEndpointQueryingStatuses () |
Clear endpoint status querying map. More... | |
const std::map< Endpoint, EndpointSubmissionStatus > & | GetEndpointSubmissionStatuses () const |
Get submission status map. More... | |
void | ClearEndpointSubmissionStatuses () |
Clear submission status map. More... | |
void | ClearAllStatuses () |
Clear all status maps. More... | |
void | ClearAll () |
Clear all. More... | |
Class for submitting jobs.
As the name indicates this class is used for submitting jobs. It has a number of different submit methods which can be used directly for different job submission purposes. The class it self can be considered as a frontend to the SubmitterPlugin class, an abstract class which is extended by specialised plugins providing the actual coupling to a particular type of computing service. As a frontend, this class also takes care of loading the specialised plugins and choosing the right plugin for a given computing service, however that can also be done manually with the SubmitterPluginLoader class. In order to use the Submitter class a reference to a UserConfig object must be provided, it should exist throughout the lifetime of the created Submitter object, and the UserConfig object should contain configuration details such as the path to user credentials.
Generally there are two types of submit methods. One which doesn't accept a reference to a Job or list of Job objects, and one which does. This is because the Submitter class is able to pass submitted Job objects to consumer objects. Registering a consumer object is done using the addConsumer method passing a reference to an EntityConsumer<Job> object. An example of such a consumer is the JobSupervisor class. Multiple consumers can be registered for the same Submitter object. Every submit method will then pass submitted Job objects to the registered consumer objects. A registered consumer can be removed using the removeConsumer method.
For submitting a Grid job one should use one of the BrokeredSubmit methods. They accept a list of job descriptions and a list of information system endpoints for which computing services are discovered and matched to the job descriptions. Jobs are then submitted to the matching services in the order ranked by the algorithm specified in the UserConfig object.
Another way of submitting a job is by using the Submit methods. These methods accepts submission endpoints or ExecutionTarget objects. Using these methods will not do any client side checks whether the computing service resources pointed to by the submission endpoint (or ExecutionTarget) really matches the specified job description(s).
Common for both ways of submitting jobs is that they both return a SubmissionStatus object indicating the outcome of the submission attemp(s). If the returned status object indicates failures, further examination can be carried out by using the GetDescriptionsNotSubmitted, GetEndpointQueryingStatuses and/or GetEndpointSubmissionStatuses methods. Note that on each invocation of any of the submit methods the state from a previous submission attemp will be cleared, thus the just mentioned methods should be used just after an attempted submission fails.
An example of submitting a Grid job using C++ is shown below:
This code can be compiled with
Same example using Python shown below:
And same example using Java:
|
inline |
Create a Submitter object.
Stores a reference to the passed UserConfig object which will be used for obtaining among others location of user credentials.
|
inline |
Add a Job consumer.
Add a consumer object which will be called every time a job is submitted.
Example use of consumer concept:
[in] | addConsumer_consumer | consumer object receiving newly submitted jobs. |
SubmissionStatus Arc::Submitter::BrokeredSubmit | ( | const std::list< std::string > & | endpoints, |
const std::list< JobDescription > & | descs, | ||
const std::list< std::string > & | requestedSubmissionInterfaces = std::list< std::string >() |
||
) |
Submit jobs to matching and ranked computing services.
The passed job descriptions will be submitted to any of the matching computing services in ranked order, which have been discovered using the provided information endpoints.
First all previously set statuses will be cleared by invoking the ClearAll method. Then resource discovery is invoked using the ComputingServiceRetriever class to query the provided information endpoints, then for each JobDescription object the discovered computing services is matched against the job description, then matching computing services is ranked according to the broker algorithm (specified in the UserConfig object of the Submitter). Then if any requested submission interfaces has been specified in the optional parameter 'requestedSubmissionInterfaces', then computing services which doesn't have a matching submission interface will be ignored. Lastly the job description is tried submitted to the computing services in the ranked order, and upon the first successful submission a corresponding Job object is propagated to the registered consumers, and then the next job description is processed. If a job description cannot be submitted a pointer to it will be added to an internal list, which afterwards can be obtained using the GetDescriptionsNotSubmitted method. If any problems was encountered during submission, more details can be obtained from the returned SubmissionStatus object, or by using the GetDescriptionsNotSubmitted, GetEndpointQueryingStatuses and GetEndpointSubmissionStatuses.
[in] | endpoints | the information endpoints which will be used to initiate resource discovery. |
[in] | descs | the JobDescription objects describing the jobs to be submitted. |
[in] | requestedSubmissionInterfaces | an optional list of submission interfaces to use for submission. |
SubmissionStatus Arc::Submitter::BrokeredSubmit | ( | const std::list< std::string > & | endpoints, |
const std::list< JobDescription > & | descs, | ||
std::list< Job > & | jobs, | ||
const std::list< std::string > & | requestedSubmissionInterfaces = std::list< std::string >() |
||
) |
Submit jobs to matching and ranked computing services.
Identical to BrokeredSubmit except that submitted jobs are added to the referenced list of Job objects.
[in] | endpoints | the information endpoints which will be used to initiate resource discovery. |
[in] | descs | the JobDescription objects describing the jobs to be submitted. |
[in] | jobs | reference to a list of Job obects for which to add submitted jobs. |
[in] | requestedSubmissionInterfaces | an optional list of submission interfaces to use for submission. |
SubmissionStatus Arc::Submitter::BrokeredSubmit | ( | const std::list< Endpoint > & | endpoints, |
const std::list< JobDescription > & | descs, | ||
const std::list< std::string > & | requestedSubmissionInterfaces = std::list< std::string >() |
||
) |
Submit jobs to matching and ranked computing services.
Identical to BrokeredSubmit except that the endpoints are not strings but Endpoint objects, which can be used to provide detailed information about endpoints making resource discovery more performant.
[in] | endpoints | the information endpoints which will be used to initiate resource discovery. |
[in] | descs | the JobDescription objects describing the jobs to be submitted. |
[in] | requestedSubmissionInterfaces | an optional list of submission interfaces to use for submission. |
SubmissionStatus Arc::Submitter::BrokeredSubmit | ( | const std::list< Endpoint > & | endpoints, |
const std::list< JobDescription > & | descs, | ||
std::list< Job > & | jobs, | ||
const std::list< std::string > & | requestedSubmissionInterfaces = std::list< std::string >() |
||
) |
Submit jobs to matching and ranked computing services.
Identical to BrokeredSubmit except that submitted jobs are added to the referenced list of Job objects and that the endpoints are not strings but Endpoint objects, which can be used to provide detailed information about endpoints making resource discovery more performant.
[in] | endpoints | the information endpoints which will be used to initiate resource discovery. |
[in] | descs | the JobDescription objects describing the jobs to be submitted. |
[in] | jobs | reference to a list of Job obects for which to add submitted jobs. |
[in] | requestedSubmissionInterfaces | an optional list of submission interfaces to use for submission. |
|
inline |
Clear all.
Convenience method which calls ClearNotSubmittedDescriptions and ClearEndpointQueryingStatuses and ClearEndpointSubmissionStatuses.
|
inline |
Clear all status maps.
Convenience method which calls ClearEndpointQueryingStatuses and ClearEndpointSubmissionStatuses.
|
inline |
Clear endpoint status querying map.
|
inline |
Clear submission status map.
|
inline |
Clear list of not submitted job descriptions.
|
inline |
Get job descriptions not submitted.
|
inline |
Get status map for queried endpoints.
The returned map contains EndpointQueryingStatus objects of all the information endpoints which were queried during resource discovery.
|
inline |
Get submission status map.
The returned map contains EndpointSubmissionStatus objects for all the submission endpoints which were tried for job submission.
void Arc::Submitter::removeConsumer | ( | EntityConsumer< Job > & | removeConsumer_consumer | ) |
Remove a previous added consumer object.
[in] | removeConsumer_consumer | consumer object which should be removed. |
|
inline |
Submit job to endpoint.
Submit a job described by the passed JobDescription object to the specified submission endpoint of a computing service. The method will load the specialised submitter plugin which corresponds to the specified interface name. If no such plugin is found submission is unsuccessful. If however the the interface name is unspecified (empty), then all available submitter plugins will be tried. If submission is successful, the submitted job will be added to the registered consumer object. If unsuccessful, more details can be obtained from the returned SubmissionStatus object, or by using the GetDescriptionsNotSubmitted, GetEndpointQueryingStatuses and GetEndpointSubmissionStatuses.
[in] | endpoint | the endpoint to which job should be submitted. |
[in] | desc | the JobDescription object describing the job to be submitted. |
SubmissionStatus Arc::Submitter::Submit | ( | const Endpoint & | endpoint, |
const JobDescription & | desc, | ||
Job & | job | ||
) |
Submit job to endpoint.
Identical to Submit(const Endpoint&, const JobDescription&), with the addition that the Job object passed as reference will also be filled with job information if submission was successful.
[out] | job | a reference to a Job object which will be filled with job details if submission was successful. |
SubmissionStatus Arc::Submitter::Submit | ( | const Endpoint & | endpoint, |
const std::list< JobDescription > & | descs | ||
) |
Submit jobs to endpoint.
Identical to Submit(const Endpoint&, const JobDescription&), except that this method submits multiple jobs to same endpoint. Submitted jobs will be added to the registered consumer.
SubmissionStatus Arc::Submitter::Submit | ( | const Endpoint & | endpoint, |
const std::list< JobDescription > & | descs, | ||
std::list< Job > & | jobs | ||
) |
Submit jobs to endpoint.
Identical to Submit(const Endpoint&, const JobDescription&), with the addition that the list of Job objects passed reference will filled with the submitted jobs, and that multiple jobs are submitted to same endpoint.
SubmissionStatus Arc::Submitter::Submit | ( | const std::list< Endpoint > & | endpoint, |
const std::list< JobDescription > & | descs | ||
) |
Submit jobs to any endpoints.
Submit multiple jobs to a list of submission endpoints to computing services. For each JobDescription object submission is tried against the list of submission endpoints in order. If submission to a endpoint fails the next in the list is tried - no ranking of endpoints will be done. Also note that a job is only submitted once, and not to multiple computing services. Submitted Job objects is passed to the registered consumer objects.
SubmissionStatus Arc::Submitter::Submit | ( | const std::list< Endpoint > & | endpoint, |
const std::list< JobDescription > & | descs, | ||
std::list< Job > & | jobs | ||
) |
Submit jobs to any endpoints.
Identical to Submit(const Endpoint&, const std::list<JobDescription>&, std::list<Job>&) with the addition that submitted jobs are also added to the passed list of Job objects.
|
inline |
Submit job to ExecutionTarget (computing service)
Submit a job to a computing service, represented by a ExecutionTarget object. This is useful when resource discovery is carried out manually, not using the BrokeredSubmit methods, but using the ComputingServiceRetriever class. The submitter plugin corresponding to the interface name will be loaded. If that plugin cannot be loaded, submission will be unsuccessful. When loaded the ExecutionTarget and JobDescription object will be passed to the SubmitterPlugin::Submit method of the loaded plugin and the status of that method is returned.
[in] | et | the ExecutionTarget to which job should be submitted. |
[in] | desc | the JobDescription object describing the job to be submitted. |
SubmissionStatus Arc::Submitter::Submit | ( | const ExecutionTarget & | et, |
const JobDescription & | desc, | ||
Job & | job | ||
) |
Submit job to ExecutionTarget (computing service)
Identical to Submit(const ExecutionTarget&, const JobDescription&), with the addition that the Job object passed as reference will also be filled with job information if submission was successful.
[in] | et | the ExecutionTarget to which job should be submitted. |
[in] | desc | the JobDescription object describing the job to be submitted. |
SubmissionStatus Arc::Submitter::Submit | ( | const ExecutionTarget & | et, |
const std::list< JobDescription > & | descs | ||
) |
Submit jobs to ExecutionTarget (computing service)
Identical to Submit(const ExecutionTarget&, const JobDescription&), except that this method submits multiple jobs to the same computing service. Submitted jobs will be added to the registered consumer.
[in] | et | the ExecutionTarget to which job should be submitted. |
[in] | desc | the JobDescription object describing the job to be submitted. |
SubmissionStatus Arc::Submitter::Submit | ( | const ExecutionTarget & | et, |
const std::list< JobDescription > & | descs, | ||
std::list< Job > & | jobs | ||
) |
Submit jobs to ExecutionTarget (computing service)
Identical to Submit(const ExecutionTarget&, const JobDescription&) with the addition that this method submits multiple jobs to the same computing service, and that submitted jobs are also added to the passed list of Job objects.
[in] | et | the ExecutionTarget to which job should be submitted. |
[in] | desc | the JobDescription object describing the job to be submitted. |