Skip to main content Link Menu Expand (external link) Document Search Copy Copied

Jobs

A job is an arbitrary task with some stored state, pertaining to the governing workspace. Typically, jobs are orchestrated by pipeline operations, but can also represent tasks for the user to complete.

Objects

Job

Path Type Format Description
id String Version 4 UUID The job ID
created String ISO 8601 timestamp The instant at which the job was created
type String Job Type The descriptor for the process undertaken by the job
exitCode Integer Process exit status The job exit code
status String Job Status The job status
startTime String ISO 8601 timestamp The instant at which the job run started
endTime String ISO 8601 timestamp The instant at which the job run ended 
{
  "id" : "18126597-a787-4c94-a145-b3b1727d577c",
  "created" : "2024-04-10T14:16:10.93044",
  "type" : "PIPELINE_RUN",
  "maxAttempts" : 0,
  "attempt" : 0,
  "commitId" : "6263726e7690d522f8e89a7107dda4cffa1d4bfc",
  "exitCode" : 0,
  "status" : "COMPLETE",
  "startTime" : "2024-04-10T14:16:37.404",
  "endTime" : "2024-04-10T14:18:01.804",
  "_embedded" : {
    "pipeline" : {
      "id" : "da46d020-47c8-4e63-987a-3d0d1532af04",
      "status" : "READY",
      "name" : "SIT-Generated Pipeline [2024-04-10T15:13:10.386034]",
      "timeout" : 0,
      "maxRetries" : 0,
      "created" : "2024-04-10T14:13:10.796155",
      "lastModified" : "2024-04-10T14:13:10.796156",
      "properties" : { },
      "dataComponents" : [ "tap-matatika-sit", "Warehouse", "dbt" ],
      "actions" : [ ],
      "repositoryPath" : "pipelines/SIT-Generated Pipeline [2024-04-10T15:13:10.386034].yml"
    },
    "profile" : {
      "id" : "auth0|5eb0327cbfd7490bff55feeb",
      "name" : "[email protected]",
      "handle" : "@sit+prod",
      "email" : "[email protected]"
    }
  },
  "_links" : {
    "self" : {
      "href" : "https://catalog.matatika.com/api/jobs/18126597-a787-4c94-a145-b3b1727d577c"
    },
    "delete job" : {
      "href" : "https://catalog.matatika.com/api/jobs/18126597-a787-4c94-a145-b3b1727d577c",
      "type" : "DELETE"
    },
    "logs" : {
      "href" : "https://catalog.matatika.com/api/jobs/18126597-a787-4c94-a145-b3b1727d577c/logs?sequence=0",
      "type" : "GET"
    }
  }
}

Formats

Job Status

String

Value Description
QUEUED The job is queued
RUNNING The job is running
COMPLETE The job finished with no errors
ERROR The job finished with errors
STOPPED The job timed out or was manually stopped

Job Type

String

Value Description
WORKSPACE_INIT A system task to create a Meltano project in a workspace repository - automatically run when a workspace is created
PIPELINE_CONFIG A system task to configure the Meltano project and publish datasets with reference to a pipeline - automatically run when a pipeline is created, or a pipeline with a status of FAILED is updated
PIPELINE_VERFIY A system task to isplay and test the configuration of a pipeline
PIPELINE_RUN A system task to run a pipeline to load data into the workspace default datastore, or some other destination external to the platform - manually run by the user or automatically run on the defined schedule
PROFILE_COLLABORATE A user task to send an invitation
PROFILE_IMPORT A user task to create a pipeline

Requests

View all running or completed jobs for a workspace

GET

/api/workspaces/{workspace-id}/jobs

Returns all running or completed jobs for the workspace {workspace-id}.

Prerequisites

  • Workspace {workspace-id} must exist

Request

Example Snippets

cURL

curl -H "Authorization: Bearer $ACCESS_TOKEN" 'https://catalog.matatika.com:443/api/workspaces/a4c48a86-2d49-4858-baef-6013d5d7b3b8/jobs' -i -X GET \
    -H 'Accept: application/json, application/javascript, text/javascript, text/json' \
    -H 'Content-Type: application/json'

Python (requests)

import requests

url = "https://catalog.matatika.com:443/api/workspaces/a4c48a86-2d49-4858-baef-6013d5d7b3b8/jobs"

headers = {
  'Authorization': ACCESS_TOKEN
}

response = requests.request("GET", url, headers=headers)

print(response.text.encode('utf8'))

Response

200 OK

Job collection with HAL links.

{
  "_embedded" : {
    "jobs" : [ {
      "id" : "d74cba0d-d3dd-43ee-a6c2-6a634bac7130",
      "created" : "2024-04-10T14:12:37.672034",
      "type" : "PROFILE_IMPORT",
      "maxAttempts" : 0,
      "attempt" : 0,
      "status" : "QUEUED",
      "_embedded" : {
        "profile" : {
          "id" : "auth0|5eb0327cbfd7490bff55feeb",
          "name" : "[email protected]",
          "handle" : "@sit+prod",
          "email" : "[email protected]"
        }
      },
      "_links" : {
        "self" : {
          "href" : "https://catalog.matatika.com/api/jobs/d74cba0d-d3dd-43ee-a6c2-6a634bac7130"
        },
        "delete job" : {
          "href" : "https://catalog.matatika.com/api/jobs/d74cba0d-d3dd-43ee-a6c2-6a634bac7130",
          "type" : "DELETE"
        },
        "logs" : {
          "href" : "https://catalog.matatika.com/api/jobs/d74cba0d-d3dd-43ee-a6c2-6a634bac7130/logs?sequence=0",
          "type" : "GET"
        },
        "withdraw job" : {
          "href" : "https://catalog.matatika.com/api/jobs/d74cba0d-d3dd-43ee-a6c2-6a634bac7130/stopped",
          "type" : "PUT"
        }
      }
    }, {
      "id" : "2c5fa329-3e07-478b-972b-cc5bed10a61d",
      "created" : "2024-04-10T14:12:37.672653",
      "type" : "PROFILE_COLLABORATE",
      "maxAttempts" : 0,
      "attempt" : 0,
      "status" : "QUEUED",
      "_embedded" : {
        "profile" : {
          "id" : "auth0|5eb0327cbfd7490bff55feeb",
          "name" : "[email protected]",
          "handle" : "@sit+prod",
          "email" : "[email protected]"
        }
      },
      "_links" : {
        "self" : {
          "href" : "https://catalog.matatika.com/api/jobs/2c5fa329-3e07-478b-972b-cc5bed10a61d"
        },
        "delete job" : {
          "href" : "https://catalog.matatika.com/api/jobs/2c5fa329-3e07-478b-972b-cc5bed10a61d",
          "type" : "DELETE"
        },
        "logs" : {
          "href" : "https://catalog.matatika.com/api/jobs/2c5fa329-3e07-478b-972b-cc5bed10a61d/logs?sequence=0",
          "type" : "GET"
        },
        "withdraw job" : {
          "href" : "https://catalog.matatika.com/api/jobs/2c5fa329-3e07-478b-972b-cc5bed10a61d/stopped",
          "type" : "PUT"
        }
      }
    }, {
      "id" : "69d24ec6-6180-4008-8cb6-9ce3b54f0ce4",
      "created" : "2024-04-10T14:12:39.066897",
      "type" : "WORKSPACE_INIT",
      "maxAttempts" : 0,
      "attempt" : 0,
      "commitId" : "ef337a4649a2448b0ff6a0e0807aa5b6aabee02d",
      "exitCode" : 0,
      "status" : "COMPLETE",
      "startTime" : "2024-04-10T14:12:57.355",
      "endTime" : "2024-04-10T14:13:04.045",
      "_links" : {
        "self" : {
          "href" : "https://catalog.matatika.com/api/jobs/69d24ec6-6180-4008-8cb6-9ce3b54f0ce4"
        },
        "delete job" : {
          "href" : "https://catalog.matatika.com/api/jobs/69d24ec6-6180-4008-8cb6-9ce3b54f0ce4",
          "type" : "DELETE"
        },
        "logs" : {
          "href" : "https://catalog.matatika.com/api/jobs/69d24ec6-6180-4008-8cb6-9ce3b54f0ce4/logs?sequence=0",
          "type" : "GET"
        }
      }
    } ]
  },
  "_links" : {
    "self" : {
      "href" : "https://catalog.matatika.com/api/workspaces/a4c48a86-2d49-4858-baef-6013d5d7b3b8/jobs?page=0&size=20"
    }
  },
  "page" : {
    "size" : 20,
    "totalElements" : 3,
    "totalPages" : 1,
    "number" : 0
  }
}

View all running or completed jobs for a pipeline

GET

/api/pipelines/{pipeline-id}/jobs

Returns all running or completed jobs for the pipeline {pipeline-id}.

Prerequisites

  • Pipeline {pipeline-id} must exist

Request

Example Snippets

cURL

curl -H "Authorization: Bearer $ACCESS_TOKEN" 'https://catalog.matatika.com:443/api/pipelines/da46d020-47c8-4e63-987a-3d0d1532af04/jobs' -i -X GET \
    -H 'Accept: application/json, application/javascript, text/javascript, text/json' \
    -H 'Content-Type: application/json'

Python (requests)

import requests

url = "https://catalog.matatika.com:443/api/pipelines/da46d020-47c8-4e63-987a-3d0d1532af04/jobs"

headers = {
  'Authorization': ACCESS_TOKEN
}

response = requests.request("GET", url, headers=headers)

print(response.text.encode('utf8'))

Response

200 OK

Job collection with HAL links.

{
  "_embedded" : {
    "jobs" : [ {
      "id" : "18126597-a787-4c94-a145-b3b1727d577c",
      "created" : "2024-04-10T14:16:10.93044",
      "type" : "PIPELINE_RUN",
      "maxAttempts" : 0,
      "attempt" : 0,
      "status" : "QUEUED",
      "_embedded" : {
        "pipeline" : {
          "id" : "da46d020-47c8-4e63-987a-3d0d1532af04",
          "status" : "READY",
          "name" : "SIT-Generated Pipeline [2024-04-10T15:13:10.386034]",
          "timeout" : 0,
          "maxRetries" : 0,
          "created" : "2024-04-10T14:13:10.796155",
          "lastModified" : "2024-04-10T14:13:10.796156",
          "properties" : { },
          "dataComponents" : [ "tap-matatika-sit", "Warehouse", "dbt" ],
          "actions" : [ ],
          "repositoryPath" : "pipelines/SIT-Generated Pipeline [2024-04-10T15:13:10.386034].yml"
        },
        "profile" : {
          "id" : "auth0|5eb0327cbfd7490bff55feeb",
          "name" : "[email protected]",
          "handle" : "@sit+prod",
          "email" : "[email protected]"
        }
      },
      "_links" : {
        "self" : {
          "href" : "https://catalog.matatika.com/api/jobs/18126597-a787-4c94-a145-b3b1727d577c"
        },
        "delete job" : {
          "href" : "https://catalog.matatika.com/api/jobs/18126597-a787-4c94-a145-b3b1727d577c",
          "type" : "DELETE"
        },
        "logs" : {
          "href" : "https://catalog.matatika.com/api/jobs/18126597-a787-4c94-a145-b3b1727d577c/logs?sequence=0",
          "type" : "GET"
        },
        "withdraw job" : {
          "href" : "https://catalog.matatika.com/api/jobs/18126597-a787-4c94-a145-b3b1727d577c/stopped",
          "type" : "PUT"
        }
      }
    }, {
      "id" : "65146245-78a8-49e6-80da-207110cea5fa",
      "created" : "2024-04-10T14:14:58.981039",
      "type" : "PIPELINE_VERIFY",
      "maxAttempts" : 0,
      "attempt" : 0,
      "commitId" : "6263726e7690d522f8e89a7107dda4cffa1d4bfc",
      "exitCode" : 0,
      "status" : "COMPLETE",
      "startTime" : "2024-04-10T14:15:25.001",
      "endTime" : "2024-04-10T14:16:03.33",
      "_embedded" : {
        "pipeline" : {
          "id" : "da46d020-47c8-4e63-987a-3d0d1532af04",
          "status" : "READY",
          "name" : "SIT-Generated Pipeline [2024-04-10T15:13:10.386034]",
          "timeout" : 0,
          "maxRetries" : 0,
          "created" : "2024-04-10T14:13:10.796155",
          "lastModified" : "2024-04-10T14:13:10.796156",
          "properties" : { },
          "dataComponents" : [ "tap-matatika-sit", "Warehouse", "dbt" ],
          "actions" : [ ],
          "repositoryPath" : "pipelines/SIT-Generated Pipeline [2024-04-10T15:13:10.386034].yml"
        },
        "profile" : {
          "id" : "auth0|5eb0327cbfd7490bff55feeb",
          "name" : "[email protected]",
          "handle" : "@sit+prod",
          "email" : "[email protected]"
        }
      },
      "_links" : {
        "self" : {
          "href" : "https://catalog.matatika.com/api/jobs/65146245-78a8-49e6-80da-207110cea5fa"
        },
        "delete job" : {
          "href" : "https://catalog.matatika.com/api/jobs/65146245-78a8-49e6-80da-207110cea5fa",
          "type" : "DELETE"
        },
        "logs" : {
          "href" : "https://catalog.matatika.com/api/jobs/65146245-78a8-49e6-80da-207110cea5fa/logs?sequence=0",
          "type" : "GET"
        }
      }
    }, {
      "id" : "34ec46bf-525b-4c40-b804-a84d1e937603",
      "created" : "2024-04-10T14:13:11.130993",
      "type" : "WORKSPACE_CONFIG",
      "maxAttempts" : 0,
      "attempt" : 0,
      "commitId" : "ab933aa7602ceaca527ee55326c78c55b79de8ae",
      "exitCode" : 0,
      "status" : "COMPLETE",
      "startTime" : "2024-04-10T14:13:24.93",
      "endTime" : "2024-04-10T14:14:56.716",
      "_embedded" : {
        "pipeline" : {
          "id" : "da46d020-47c8-4e63-987a-3d0d1532af04",
          "status" : "READY",
          "name" : "SIT-Generated Pipeline [2024-04-10T15:13:10.386034]",
          "timeout" : 0,
          "maxRetries" : 0,
          "created" : "2024-04-10T14:13:10.796155",
          "lastModified" : "2024-04-10T14:13:10.796156",
          "properties" : { },
          "dataComponents" : [ "tap-matatika-sit", "Warehouse", "dbt" ],
          "actions" : [ ],
          "repositoryPath" : "pipelines/SIT-Generated Pipeline [2024-04-10T15:13:10.386034].yml"
        },
        "profile" : {
          "id" : "auth0|5eb0327cbfd7490bff55feeb",
          "name" : "[email protected]",
          "handle" : "@sit+prod",
          "email" : "[email protected]"
        }
      },
      "_links" : {
        "self" : {
          "href" : "https://catalog.matatika.com/api/jobs/34ec46bf-525b-4c40-b804-a84d1e937603"
        },
        "delete job" : {
          "href" : "https://catalog.matatika.com/api/jobs/34ec46bf-525b-4c40-b804-a84d1e937603",
          "type" : "DELETE"
        },
        "logs" : {
          "href" : "https://catalog.matatika.com/api/jobs/34ec46bf-525b-4c40-b804-a84d1e937603/logs?sequence=0",
          "type" : "GET"
        }
      }
    } ]
  },
  "_links" : {
    "self" : {
      "href" : "https://catalog.matatika.com/api/pipelines/da46d020-47c8-4e63-987a-3d0d1532af04/jobs?page=0&size=20"
    }
  },
  "page" : {
    "size" : 20,
    "totalElements" : 3,
    "totalPages" : 1,
    "number" : 0
  }
}

View a job

GET

/api/jobs/{job-id}

Returns the job {job-id}.

Prerequisites

  • Job {job-id} must exist

Request

Example Snippets

cURL

curl -H "Authorization: Bearer $ACCESS_TOKEN" 'https://catalog.matatika.com:443/api/jobs/18126597-a787-4c94-a145-b3b1727d577c' -i -X GET \
    -H 'Accept: application/json, application/javascript, text/javascript, text/json' \
    -H 'Content-Type: application/json'

Python (requests)

import requests

url = "https://catalog.matatika.com:443/api/jobs/18126597-a787-4c94-a145-b3b1727d577c"

headers = {
  'Authorization': ACCESS_TOKEN
}

response = requests.request("GET", url, headers=headers)

print(response.text.encode('utf8'))

Response

200 OK

Job with HAL links.

{
  "id" : "18126597-a787-4c94-a145-b3b1727d577c",
  "created" : "2024-04-10T14:16:10.93044",
  "type" : "PIPELINE_RUN",
  "maxAttempts" : 0,
  "attempt" : 0,
  "commitId" : "6263726e7690d522f8e89a7107dda4cffa1d4bfc",
  "exitCode" : 0,
  "status" : "COMPLETE",
  "startTime" : "2024-04-10T14:16:37.404",
  "endTime" : "2024-04-10T14:18:01.804",
  "_embedded" : {
    "pipeline" : {
      "id" : "da46d020-47c8-4e63-987a-3d0d1532af04",
      "status" : "READY",
      "name" : "SIT-Generated Pipeline [2024-04-10T15:13:10.386034]",
      "timeout" : 0,
      "maxRetries" : 0,
      "created" : "2024-04-10T14:13:10.796155",
      "lastModified" : "2024-04-10T14:13:10.796156",
      "properties" : { },
      "dataComponents" : [ "tap-matatika-sit", "Warehouse", "dbt" ],
      "actions" : [ ],
      "repositoryPath" : "pipelines/SIT-Generated Pipeline [2024-04-10T15:13:10.386034].yml"
    },
    "profile" : {
      "id" : "auth0|5eb0327cbfd7490bff55feeb",
      "name" : "[email protected]",
      "handle" : "@sit+prod",
      "email" : "[email protected]"
    }
  },
  "_links" : {
    "self" : {
      "href" : "https://catalog.matatika.com/api/jobs/18126597-a787-4c94-a145-b3b1727d577c"
    },
    "delete job" : {
      "href" : "https://catalog.matatika.com/api/jobs/18126597-a787-4c94-a145-b3b1727d577c",
      "type" : "DELETE"
    },
    "logs" : {
      "href" : "https://catalog.matatika.com/api/jobs/18126597-a787-4c94-a145-b3b1727d577c/logs?sequence=0",
      "type" : "GET"
    }
  }
}

View the logs of a job

GET

/api/jobs/{job-id}/logs?sequence={sequence}

Returns the logs of the job {job-id}.

Prerequisites

  • Job {job-id} must exist

Request

Query Parameters

Query Parameter Format Default Value Description
sequence Unsigned integer 0 The line number in the logs to read from

Headers

Accept
Media Type(s) Description
text/plain */* Sets the response content type format to plain text
application/stream+json application/x-ndjson Sets the response content type format to NDJSON

Example Snippets

cURL

curl -H "Authorization: Bearer $ACCESS_TOKEN" 'https://catalog.matatika.com:443/api/jobs/18126597-a787-4c94-a145-b3b1727d577c/logs?sequence=0' -i -X GET \
    -H 'Accept: application/x-ndjson' \
    -H 'Content-Type: application/json'

Python (requests)

import requests

url = "https://catalog.matatika.com:443/api/jobs/18126597-a787-4c94-a145-b3b1727d577c/logs?sequence=0"

headers = {
  'Authorization': ACCESS_TOKEN
}

response = requests.request("GET", url, headers=headers)

print(response.text.encode('utf8'))

Response

  • 200: The job logs in the format specified by associated request Accept header
{
  "jobId" : "18126597-a787-4c94-a145-b3b1727d577c",
  "level" : "INFO",
  "message" : "Begin: 18126597-a787-4c94-a145-b3b1727d577c",
  "sequence" : 1,
  "timestamp" : "2024-04-10T14:16:38.579Z"
}
  • 204: No response body provided.

Create a job from a pipeline

POST

/api/pipelines/{pipeline-id}/jobs

Creates a new job from the pipeline {pipeline-id}.

Prerequisites

  • Pipeline {pipeline-id} must exist and not already be running

Request

Example Snippets

cURL

curl -H "Authorization: Bearer $ACCESS_TOKEN" 'https://catalog.matatika.com:443/api/pipelines/da46d020-47c8-4e63-987a-3d0d1532af04/jobs' -i -X POST \
    -H 'Accept: application/json, application/javascript, text/javascript, text/json' \
    -H 'Content-Type: application/json'

Python (requests)

import requests

url = "https://catalog.matatika.com:443/api/pipelines/da46d020-47c8-4e63-987a-3d0d1532af04/jobs"

headers = {
  'Authorization': ACCESS_TOKEN
}

response = requests.request("POST", url, headers=headers)

print(response.text.encode('utf8'))

Response

201 Created

Job with HAL links.

{
  "id" : "18126597-a787-4c94-a145-b3b1727d577c",
  "created" : "2024-04-10T14:16:10.93044",
  "type" : "PIPELINE_RUN",
  "maxAttempts" : 0,
  "attempt" : 0,
  "status" : "QUEUED",
  "_embedded" : {
    "pipeline" : {
      "id" : "da46d020-47c8-4e63-987a-3d0d1532af04",
      "status" : "READY",
      "name" : "SIT-Generated Pipeline [2024-04-10T15:13:10.386034]",
      "timeout" : 0,
      "maxRetries" : 0,
      "created" : "2024-04-10T14:13:10.796155",
      "lastModified" : "2024-04-10T14:13:10.796156",
      "properties" : { },
      "dataComponents" : [ "tap-matatika-sit", "Warehouse", "dbt" ],
      "actions" : [ ],
      "repositoryPath" : "pipelines/SIT-Generated Pipeline [2024-04-10T15:13:10.386034].yml"
    },
    "profile" : {
      "id" : "auth0|5eb0327cbfd7490bff55feeb",
      "name" : "[email protected]",
      "handle" : "@sit+prod",
      "email" : "[email protected]"
    }
  },
  "_links" : {
    "self" : {
      "href" : "https://catalog.matatika.com/api/jobs/18126597-a787-4c94-a145-b3b1727d577c"
    },
    "delete job" : {
      "href" : "https://catalog.matatika.com/api/jobs/18126597-a787-4c94-a145-b3b1727d577c",
      "type" : "DELETE"
    },
    "logs" : {
      "href" : "https://catalog.matatika.com/api/jobs/18126597-a787-4c94-a145-b3b1727d577c/logs?sequence=0",
      "type" : "GET"
    },
    "withdraw job" : {
      "href" : "https://catalog.matatika.com/api/jobs/18126597-a787-4c94-a145-b3b1727d577c/stopped",
      "type" : "PUT"
    }
  }
}

Stop a job

PUT

/api/jobs/{job-id}/stopped

Stops the execution of the job {job-id}.

Prerequisites

  • Job {job-id} must exist
  • Job {job-id} must have status RUNNING

Request

Example Snippets

cURL

curl -H "Authorization: Bearer $ACCESS_TOKEN" 'https://catalog.matatika.com:443/api/jobs/396604aa-549f-4526-b5c6-d92f0483d098/stopped' -i -X PUT \
    -H 'Accept: application/json, application/javascript, text/javascript, text/json' \
    -H 'Content-Type: application/json'

Python (requests)

import requests

url = "https://catalog.matatika.com:443/api/jobs/396604aa-549f-4526-b5c6-d92f0483d098/stopped"

headers = {
  'Authorization': ACCESS_TOKEN
}

response = requests.request("PUT", url, headers=headers)

print(response.text.encode('utf8'))

Response

202 Accepted

Job stop acceptance message.

{
  "message" : "Job stop requested"
}

Delete a job

DELETE

/api/jobs/{job-id}

Deletes and stops the execution of the job {job-id}.

Prerequisites

  • Job {job-id} must exist

Example Snippets

cURL

curl -H "Authorization: Bearer $ACCESS_TOKEN" 'https://catalog.matatika.com:443/api/jobs/18126597-a787-4c94-a145-b3b1727d577c' -i -X DELETE \
    -H 'Accept: application/json, application/javascript, text/javascript, text/json' \
    -H 'Content-Type: application/json'

Python (requests)

import requests

url = "https://catalog.matatika.com:443/api/jobs/18126597-a787-4c94-a145-b3b1727d577c"

headers = {
  'Authorization': ACCESS_TOKEN
}

response = requests.request("DELETE", url, headers=headers)

print(response.text.encode('utf8'))

Response

204 No Content

No response body provided.