IVOA

Universal Worker Service Pattern
Version 2.0

IVOA WG Internal Draft 2011 June 21

This version:
http://www.ivoa.net/Documents/...
Latest version:
http://www.ivoa.net/Documents/latest/...
Previous versions:
http://www.ivoa.net/Documents/...
http://www.ivoa.net/Documents/...
Working Group:
http://www.ivoa.net/twiki/bin/view/IVOA/IvoaGridAndWebServices
Author(s):
J. Normand, J-C. Malapert

Abstract

The Universal Worker Service (UWS) pattern defines how to manage asynchronous execution of jobs on a service. Any application of the pattern defines a family of related services with a common service contract.

Status of this Document

This is a Working Draft. The first release of this document was 2011 June 21.

A list of current IVOA Recommendations and other technical documents can be found at http://www.ivoa.net/Documents/.

Acknowledgments

Contents

1. Introduction

2. Universal Worker Service Interface

2.1 Objects

As illustrated in figure 1, a UWS service consists logically in a set of objects that may be read and written to in order to control jobs.

2.1.1 Jobs

The Jobs object contains all the jobs in the UWS.

2.1.2 Job

The Job object contains all the information relative to one job. Each Job contains:

2.1.3 Phase

The job is treated as a state machine. The Phase object names the state. The values for Phase are:

PENDING
The job is accepted by the service but not yet committed for execution by the client. This is the state into which a job enters when it is first created.
QUEUED
The job is committed to execution by the client but the service has not yet assigned it to a processor.
EXECUTING
The job has been assigned to a processor.
COMPLETED
The execution of the job is over.
ERROR
The job failed to complete.
ABORTED
The job has been aborted by the user.
SUSPENDED
The job has been suspended by the system during execution.
UNKNOWN
The job is in an unknown state.

A successful job will normally progress through the PENDING, QUEUED, EXECUTING, COMPLETED phases in that order. At any time before the COMPLETED phase a job may be ABORTED, SUSPENDED or may suffer an ERROR. If the UWS reports an UNKNOWN phase, then all the client can do is re-query the phase until a known phase is reported.

When a job has been SUSPENDED, the UWS will automaticaly resume the job into the EXECUTING phase without any intervention. The client may abord the job.

2.1.4 Quote

The Quote object represents the instant when the job is likely to complete.

2.1.5 Owner

The Owner object represents the identifier of the creator of the job.

When an authentication mechanism is used in the UWS, the implementation should set the Owner object to the identity obtained by the authentication. If there was no authenticated job creator then this should be set to NULL.

2.1.6 Destruction time

The Destruction time object represents the instant when the job shall be deleted.

The initial destruction time is set by the service when the job is created. The client may change the life expectancy of a job but the service may forbid this change or may set limits on the allowed Destruction time. The format is ISO8601.

When the Destruction time has been exceeded, the service automaticly deletes the job and generated results. Any previously generated results of the job are no longer available.

2.1.7 Execution duration

The Execution duration object represents the duration for which a job shall run.

The initial execution duration is set by the service when the job is created. The client may change this duration but the service may forbid this change or may set limits on the allowed Execution duration. The duration is defined in real clock seconds by an integer.

When the Execution duration has been exceeded, the service automaticly aborts the job. Any previously generated results of the job are still available.

2.1.8 Error

The Error object represents a human readable error message for the job.

2.1.9 Results

The Results object represents a list of job Result object.

2.1.10 Result

A job Result is an object resulting from the computation that may be fetched from the service when the job has completed.

2.1.11 Parameters

The Parameters object represents a list of the job Parameter.

2.1.12 Parameter

The Parameter object represents a parameter of the job.

The client can set or update parameters only when the Phase of the job is set to PENDING.

2.2 Resources

2.2.1 Identification

In a REST binding, the domain model is represented as distinct web-resources each with its own URI. In addition to objects in the domain model, we have added a start and an abort resource to handle asynchronous call.

In the REST interface of UWS, each objects defined above is available as a web resource with its own URI. The URIs must form a hierarchy as shown in the table below:

URI Object
/{jobs} Job List
/{jobs}/{job-id} Job
/{jobs}/{job-id}/phase Phase of job {job-id}
/{jobs}/{job-id}/quote Quote of job {job-id}
/{jobs}/{job-id}/owner Owner of job {job-id}
/{jobs}/{job-id}/destruction Destruction instant of job {job-id}
/{jobs}/{job-id}/executionduration Execution duration of job {job-id}
/{jobs}/{job-id}/error Error message of job {job-id}
/{jobs}/{job-id}/results Results List of job {job-id}
/{jobs}/{job-id}/results/{result-id} The Result {result-id} of job {job-id}
/{jobs}/{job-id}/parameters Parameters List of job {job-id}
/{jobs}/{job-id}/parameters/{parameter-name} The parameter {parameter-name} of job {job-id}

There are also two URIs used to control the job:

URI Description
/{jobs}/{job-id}/start Used to start the job {job-id}
/{jobs}/{job-id}/abort Used to abort the job {job-id}

The service implementer is free to choose the names given in parentheses above. The other names are part of the UWS standard.

The URI for the Job List, in its absolute form is the root URI for the whole UWS. This URI should be given as the access URL in the UWS registration.

2.2.2 Representation

Each of the UWS objects is mapped to a resource with its own URI as detailed in the table above, and for each URI, a HTTP GET fetches a representation of that resource.

In UWS 3 types of representations are used depending on objects and context: XML, plain text and HTML.

If an object is a container for other objects (Job List, Job, Result List, Parameter List) then an XML representation of the object should be returned, otherwise for simple atomic types (Quote, Execution Duration...) a plain text representation (mimetype: "text/plain") should be returned.

HTTP allows multiple representations of a resource distinguished by their MIME types and selected by the HTTP "Accept" headers of a HTTP GET request. A UWS implementation can exploit this to support both web browsers and rich clients in the same tree of resources. Although the default behaviour is to return XML, a UWS could return HTML or XHTML to clients that accept these types. These clients are assumed to be web browsers and the UWS is generating its own user interface. The HTML interface generated should allow full control of the UWS via the use of HTML forms and appropriate links.

Clients which are assumed to be part of remote applications that drive UWS without showing the details to their users should accept only "application/xml,text/plain" type. A UWS must therefore return XML representations of the resources in preference to the HTML representation. A technique that may be used to always return XML that modern browsers can transform on the client-side to HTML is via the xml-stylesheet processing instruction, which can be used to point to a suitable XSL resource to perform the transformation.

2.2.2.1 Job List

For the Job List object an XML representation must be returned. This representation is specified by the <uws:jobs> element in the UWS schema.

The representation of the Job List is a list of links to the resources representing the jobs. The list may be empty if the UWS is idle.

         <?xml version="1.0" encoding="UTF-8"?>
         <uws:jobs xmlns:uws="http://www.ivoa.net/xml/UWS/v1.0"
               xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
               xsi:schemaLocation="http://www.ivoa.net/xml/UWS/v1.0 http://ivoa.net/xml/UWS/UWS-v1.0.xsd"
               xmlns:xlink="http://www.w3.org/1999/xlink">
            <uws:jobref id="job1" xlink:href="http://example.org/uws/jobs/job1">
               <uws:phase>COMPLETED</uws:phase>
            </uws:jobref>
            <uws:jobref id="job2" xlink:href="http://example.org/uws/jobs/job2">
               <uws:phase>EXECUTING</uws:phase>
            </uws:jobref>
         </uws:jobs>
      
2.2.2.2 Job

For the Job object an XML representation must be returned. This representation is specified by the <uws:job> element in the UWS schema.

The <uws:job> element has placeholders of all of the standard UWS objects, and in addition there is a <uws:jobinfo> element which can be used by implementations to include any extra information within the job description. An example of such a job instance is shown below:

         <?xml version="1.0" encoding="UTF-8"?>
         <uws:job xmlns:uws="http://www.ivoa.net/xml/UWS/v1.0"
                  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
                  xsi:schemaLocation="http://www.ivoa.net/xml/UWS/v1.0 http://ivoa.net/xml/UWS/UWS-v1.0.xsd"
                  xmlns:xlink="http://www.w3.org/1999/xlink">
            <uws:jobId>a108c361-d140-f724-8936-1b1cff656f05</uws:jobId>
            <uws:phase>COMPLETED</uws:phase>
            <uws:startTime>2011-09-07T09:02:19+00:00</uws:startTime>
            <uws:endTime>2011-09-07T09:02:52+00:00</uws:endTime>
            <uws:executionDuration>86400</uws:executionDuration>
            <uws:destruction>2011-09-14T09:02:12+00:00</uws:destruction>
            <uws:ownerId xsi:nil="true"/>
            <uws:quote xsi:nil="true"/>
            <uws:parameters>
               <uws:parameter id="param1">value1</uws:parameter>
               <uws:parameter id="param2" byReference="true">http://example.org/uws/param2</uws:parameter>
            </uws:parameters>
            <uws:results>
               <uws:result id="result1" xlink:href="http://example.org/uws/jobs/job1/results/result1"/>
            </uws:results>
         </uws:job>
      
2.2.2.3 Results List

For the Results List object an XML representation must be returned. This representation is specified by the <uws:results> element in the UWS schema.

The representation of the Results List is a list of links to the resources representing the results. These resources may have any URI and any MIME type. A sensible default for their URIs is to make them children of /{jobs}/{job-id}/results, but this is not required. It may sometimes be easier for a service implementer to point to a resource on some web server separate from that running the UWS. Therefore, a client must always parse the Results List to find the results. Each result in a result list must be given a unique identifier. Where a protocol applying UWS specifies standard results it must do so fixing the identifier for those results and fixing the result URIs, however the UWS must still return a valid Results List at /{jobs}/{job-id}/results, even though in this case the identifiers and URIs could be precomputed by the client.

         <?xml version="1.0" encoding="UTF-8"?>
         <uws:results xmlns:uws="http://www.ivoa.net/xml/UWS/v1.0"
                  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
                  xsi:schemaLocation="http://www.ivoa.net/xml/UWS/v1.0 http://ivoa.net/xml/UWS/UWS-v1.0.xsd"
                  xmlns:xlink="http://www.w3.org/1999/xlink">
            <uws:result id="result1" xlink:href="http://example.org/uws/jobs/job1/results/result1"/>
            <uws:result id="result2" xlink:href="http://example.org/uws/jobs/job1/results/result2"/>
         </uws:results>
      
2.2.2.4 Parameters List and parameter

For the Parameters List object an XML representation must be returned. This representation is specified by the <uws:parameters> element in the UWS schema.

The representation of the Parameters List is a list of <uws:parameter> elements. The form that the parameters take will depend on the JDL of the implementing service.

For services where the JDL consists of a list of name/value pairs (typical of the standard IVOA "simple" access protocols), then these would naturally be expressed in the parameter list. Each of these elements can either represent the value of the parameter directly, where the content of the element is a textual representation of the parameter, or in the case where the parameter value cannot be represented legally within XML (e.g. the parameter is a binary type such as a FITS file) then the content of the element is a URL to the parameter value. To indicate this case the attribute byReference is set to true.

         <?xml version="1.0" encoding="UTF-8"?>
         <uws:parameters xmlns:uws="http://www.ivoa.net/xml/UWS/v1.0"
               xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
               xsi:schemaLocation="http://www.ivoa.net/xml/UWS/v1.0 http://ivoa.net/xml/UWS/UWS-v1.0.xsd">
            <uws:parameter id="name1">value1</uws:parameter>
            <uws:parameter id="name2" byReference="true">http://example.org/uws/name2</uws:parameter>
         </uws:parameters>
      

For services where the JDL consists in a document with its own syntax (for instance an XML document with a specific schema, JSON file...), then there would be a single <uws:parameter> element where the content was the URL to that document. In this case the attribute byReference is set to true.

         <?xml version="1.0" encoding="UTF-8"?>
         <uws:parameters xmlns:uws="http://www.ivoa.net/xml/UWS/v1.0"
               xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
               xsi:schemaLocation="http://www.ivoa.net/xml/UWS/v1.0 http://ivoa.net/xml/UWS/UWS-v1.0.xsd">
            <uws:parameter id="jdl" byReference="true">http://example.org/uws/jdl</uws:parameter>
         </uws:parameters>
      

When the value is expressed by reference, the service can give the possibility to upload your document. In this case the URL is given by UWS itself and takes the following form : http://example.org//{jobs}/{job-id}/parameters/{parameter-name}.

2.2.2.5 Phase, Quote, Owner, Destruction time, Execution duration, Error

For these objects a textual representation must be returned.

2.2.3 Manipulation

2.2.3.1 Job List

To retrieve the job list use HTTP GET method on /{jobs} URI. The expected response can be:

200 "OK" - uws:jobs representation
If succeed.
500 "Internal Server Error" - HTML representation
If failure.
2.2.3.2 Job

To retrieve the job summary use HTTP GET method on /{jobs}/{job-id} URI. The expected response can be:

200 "OK" - uws:jobsummary representation
If succeed.
404 "Not Found" - HTML representation
If {job-id} does not exist.
500 "Internal Server Error" - HTML representation
If failure.
2.2.3.2.1 Create

To create a new job (unless the service rejects the request) use HTTP POST method on /{jobs} URI. The expected response can be:

201 "Created" - Empty representation
Redirect to the newly created /{jobs}/{job-id} resource if succeed.
500 "Internal Server Error" - HTML representation
If failure.
2.2.3.2.2 Delete

To delete a job use HTTP DELETE method on /{jobs}/{job-id} URI. The expected response can be:

204 "No Content" - Empty representation
If succeed.
404 "Not Found" - HTML representation
If {job-id} does not exist.
405 "Not Allowed" - HTML representation
If job can not be deleted. A job can be deleted only if one of these status has been reached: COMPLETED, ABORTED or ERROR.
500 "Internal Server Error" - HTML representation
If failure.

Some server and client do not handle HTTP DELETE method. In this case we propose to use the tunneling method. The tunneling can use query parameters. This is particularly useful for browser-based applications that can't fully control the HTTP requests sent. In our case, we propose to use METHOD argument to pass the method HTTP in the query.

2.2.3.2.3 Start

To start a job use the HTTP PUT method on /{jobs}/{job-id}/start URI. The expected response can be:

204 "No Content" - Empty representation
If succeed.
404 "Not Found" - HTML representation
If {job-id} does not exist.
405 "Not Allowed" - HTML representation
If job can not be started. A job can be started only if its status is PENDING.
500 "Internal Server Error" - HTML representation
If failure.
2.2.3.2.4 Abort

To abort a job use the HTTP PUT method on /{jobs}/{job-id}/abort URI. The expected response can be:

204 "No Content" - Empty representation
If succeed.
404 "Not Found" - HTML representation
If {job-id} does not exist.
405 "Not Allowed" - HTML representation
If job can not be aborted. A job can be aborted only if one of these status has been reached: PENDING, QUEUED or EXECUTING.
500 "Internal Server Error" - HTML representation
If failure.
2.2.3.3 Job parameters

To retrieve all the parameters of a job use HTTP GET method on /{jobs}/{job-id}/parameters URI. The expected response can be:

200 "OK" - uws:parameters representation
If succeed.
404 "Not Found" - HTML representation
If {job-id} does not exist.
500 "Internal Server Error" - HTML representation
If failure.

To retrieve a particular parameter of a job use HTTP GET method on /{jobs}/{job-id}/parameters/{parameter-name} URI. The expected response can be:

200 "OK" - {parameter-name} representation
If succeed.
404 "Not Found" - HTML representation
If {job-id} or {parameter-name} does not exist.
500 "Internal Server Error" - HTML representation
If failure.
2.2.3.3.1 Create

To create and set parameter(s) of a job use the HTTP POST method on /{jobs}/{job-id}/parameters URI. You can create and set several parameters in the same time. The expected response can be:

204 "No Content" - Empty representation
If succeed.
400 "Bad Request" - HTML Representation
If at least one of the POSTed parameter is not a service parameter.
403 "Forbidden" - Empty representation
If the job has already been started (ie. its status is not PENDING).
404 "Not Found" - HTML representation
If {job-id} does not exist.
[FIX TODO - c'est possible de ne pas permettre ca ?] 405 "Method Not Allowed" - HTML representation
If the service does not allow to set parameters using this method.
500 "Internal Server Error" - HTML representation
If failure.

If the value of the parameter can be expressed with a textual representation, the Content-Type header of the request should be set to application/x-www-form-urlencoded.

For parameters defined by a document, you can either set them by reference or by uploading the document on the UWS.

If you set several parameters at the same time including an upload, use multipart/form-data.

2.2.3.3.2 Update

To update a parameter use HTTP PUT method on /{jobs}/{job-id}/parameters/{parameter-name} URI. The expected response can be:

204 "No Content" - Empty representation
If succeed (ie. the parameter has been updated).
403 "Forbidden" - Empty representation
If the job has already been started (ie. its status is not PENDING).
404 "Not Found" - HTML representation
If {job-id} or {parameter-name} does not exist.
[FIX TODO - c'est possible de ne pas permettre ca ?] 405 "Method Not Allowed" - HTML representation
If the service does not allow to set parameters using this method.
500 "Internal Server Error" - HTML representation
If failure.
2.2.3.3.3 Delete

To delete a parameter use HTTP DELETE method on /{jobs}/{job-id}/parameters/{parameter-name} URI. The expected response can be:

204 "No Content" - Empty representation
If succeed (ie. the parameter has been deleted).
500 "Internal Server Error" - HTML representation
If failure.
2.2.3.4 Job properties
2.2.3.4.1 destruction

To retrieve the destruction time of a job use HTTP GET method on /{jobs}/{job-id}/destruction URI. The expected response can be:

200 "OK" - String representation
If succeed.
404 "Not Found" - HTML representation
If {job-id} does not exist.
500 "Internal Server Error" - HTML representation
If failure.

The initial destruction time is set by the service when the job is created. To change the life expectancy of a job use HTTP PUT method on /{jobs}/{job-id}/destruction URI. The service may forbids this change. The expected response can be:

204 "No Content" - Empty representation
If succeed (ie. the destruction time has been changed).
400 "Bad Request" - HTML Representation
If format is not ISO8601.
404 "Not Found" - HTML representation
If {job-id} does not exist.
405 "Method Not Allowed" - HTML representation
If the service forbids this change.
500 "Internal Server Error" - HTML representation
If failure.
2.2.3.4.2 executionduration

To retrieve the execution duration of a job use HTTP GET method on /{jobs}/{job-id}/executionduration URI. The expected response can be:

200 "OK" - String representation
If succeed.
404 "Not Found" - HTML representation
If {job-id} does not exist.
500 "Internal Server Error" - HTML representation
If failure.

The initial execution duration is set by the service when the job is created. To change this duration use HTTP PUT method on /{jobs}/{job-id}/executionduration URI. The service may forbids this change. The expected response can be:

204 "No Content" - Empty representation
If succeed (ie. the execution duration has been changed).
400 "Bad Request" - HTML Representation
If format is not an integer.
404 "Not Found" - HTML representation
If {job-id} does not exist.
405 "Method Not Allowed" - HTML representation
If the service forbids this change.
500 "Internal Server Error" - HTML representation
If failure.
2.2.3.4.3 results

To retrieve the results of a job use HTTP GET method on /{jobs}/{job-id}/results URI. The expected response can be:

200 "OK" - uws:results representation
If succeed.
404 "Not Found" - HTML representation
If {job-id} does not exist.
500 "Internal Server Error" - HTML representation
If failure.
2.2.3.4.4 owner, quote, phase, error

To retrieve one of these properties of a job use HTTP GET method on /{jobs}/{job-id}/{property} URI (with {property} stands for owner, quote, phase or error). The expected response can be:

200 "OK" - String representation
If succeed.
404 "Not Found" - HTML representation
If {job-id} does not exist.
500 "Internal Server Error" - HTML representation
If failure.

2.3 Calling sequences

[TODO diagram] je sais pas quel genre c'est cf. http://en.wikipedia.org/wiki/Use_case_diagram

References