Pipelines

A pipeline is a workspace-specific configuration or definition for extracting, loading and transforming data from a given dataplugin - following the concept of ELT. Pipelines are run as jobs to load data into a workspace, either manually or on a predetermined schedule. Only a single pipeline can be run at any given time.

Objects

Pipeline

Path	Type	Format	Description
`id`	`String`	Version 4 UUID	The pipeline ID
`status`	`String`	Pipeline Status
`name`	`String`		The pipeline name
`schedule`	`String`	Cron	The interval at which to launch a new job e.g. `0 0 9-17 * * MON-FRI` launches a job on the hour nine-to-five weekdays
`timeout`	`Integer`	Unsigned	The number of seconds after which the job will terminate - if set to `0`, an implicit default value of 300 seconds is used
`script`	`String`	Bash	Custom script to be executed as the pipeline job
`created`	`String`	ISO 8601 timestamp	When the pipeline was created
`lastModified`	`String`	ISO 8601 timestamp	When the pipeline was last modified
`properties`	`Properties`		The properties to run the pipeline with, defined by the pipeline dataplugin `settings`
`dataSource`	`String`		The pipeline data source dataplugin `name`
`dataStore`	`String`	Version 4 UUID	The pipeline datastore `id`
`_embedded.dataSource`	`DataPlugin`		The pipeline data source dataplugin
`_embedded.dataStore`	`DataStore`		The pipeline data store
`_embedded.'latest job'`	`Job`		The latest job run from the pipeline

{
  "id" : "1dcbb564-53c3-474c-8d18-0ab8b5961144",
  "status" : "READY",
  "name" : "SIT-generated pipeline (updated)",
  "schedule" : "0 0 0 25 12 ?",
  "timeout" : 0,
  "created" : "2022-06-30T15:34:15.653809",
  "lastModified" : "2022-06-30T15:34:15.65381",
  "properties" : {
    "end_date" : "2022-06-30T16:31:44.141753+01:00",
    "oauth_credentials.access_token" : "access_token",
    "reports" : "reports",
    "oauth_credentials.refresh_token" : "refresh_token",
    "view_id" : "1234567890",
    "oauth_credentials.client_id" : "client_id",
    "oauth_credentials.client_secret" : "client_secret",
    "start_date" : "2022-05-30T16:31:44.141674+01:00"
  },
  "dataSource" : "tap-google-analytics",
  "dataStore" : "1e39e629-3622-4546-990a-024fd22927b0",
  "_embedded" : {
    "dataSource" : {
      "id" : "67caf756-749f-43ab-b086-d91a59bb2ebc",
      "name" : "tap-google-analytics",
      "label" : "Google Analytics",
      "imageUrl" : "/assets/images/datasource/tap-google-analytics.svg",
      "description" : "App and website analytics platform hosted by Google\n\n## Prerequisites\n\nBefore importing data from Google Analytics you or your webmaster will need to have performed some initial setup to grant\nyourself access to the Google Analytics for your website.\n\n[Google Analytics Prerequisites](https://www.matatika.com/docs/data-sources/tap-google-analytics/prerequisites)\n\n## Data source settings\n### View ID\n\nOnce you have connected with your google account, the `View ID` drop down will allow you to choose which view you want to get your data from.\n\n### Start Date\n\nProviding a start date allows you to set how much historical data you wish to import into you workspace. \n\nThis date has to be in the ISO-8601 date format, see the example below.\n\nFormat: `YYYY-MM-DDTHH:MM:SSZ`\n\nExample: `2020-10-01T13:12:11Z`\n\n### End Date\n\nProviding an end date allows you to limit how much data you wish to import into you workspace. \n\nThis date has to be in the ISO-8601 date format, see the example below.\n\nFormat: `YYYY-MM-DDTHH:MM:SSZ`\n\nExample: `2020-10-01T13:12:11Z`\n\n### Reports\n\nProviding the path to a reports json file allows you to supply the tap with a user defined definition of tables and information you want to sync from Google Analytics.\nThis does require you to commit a file to your workspace repository so that Google Analytics can find it during run time, and may also break the dbt transforms that are required to have our included datasets work.\nFor more information on how to define a custom reports file see the [Meltano Hub Documentation](https://hub.meltano.com/extractors/google-analytics#reports).\n\n## Learn more\n\n[Matatika Google Analytics Data Source Documentation](https://www.matatika.com/docs/data-sources/tap-google-analytics)\n\n[Meltano Hub Google Analytics Documentation](https://hub.meltano.com/extractors/google-analytics)"
    },
    "dataStore" : {
      "id" : "1e39e629-3622-4546-990a-024fd22927b0",
      "name" : "Default",
      "imageUrl" : "/assets/images/datastore/target-postgres.png",
      "loader" : "target-postgres--transferwise"
    },
    "latest job" : {
      "id" : "752e1887-3198-48a9-98a4-f9bf1f4b76d4",
      "created" : "2022-06-30T15:34:17.35072",
      "type" : "PIPELINE_CONFIG",
      "exitCode" : 0,
      "status" : "COMPLETE",
      "startTime" : "2022-06-30T15:34:41.313",
      "endTime" : "2022-06-30T15:35:50.585",
      "_links" : {
        "self" : {
          "href" : "https://catalog.matatika.com/api/jobs/752e1887-3198-48a9-98a4-f9bf1f4b76d4"
        },
        "delete job" : {
          "href" : "https://catalog.matatika.com/api/jobs/752e1887-3198-48a9-98a4-f9bf1f4b76d4",
          "type" : "DELETE"
        },
        "logs" : {
          "href" : "https://catalog.matatika.com/api/jobs/752e1887-3198-48a9-98a4-f9bf1f4b76d4/logs?sequence=0",
          "type" : "GET"
        }
      }
    }
  },
  "_links" : {
    "update pipeline" : {
      "href" : "https://catalog.matatika.com/api/workspaces/53ad82e6-fad5-417b-ad02-4e81cfb845f8/pipelines/1dcbb564-53c3-474c-8d18-0ab8b5961144",
      "type" : "PUT"
    },
    "delete pipeline" : {
      "href" : "https://catalog.matatika.com/api/pipelines/1dcbb564-53c3-474c-8d18-0ab8b5961144",
      "type" : "DELETE"
    },
    "draft pipeline" : {
      "href" : "https://catalog.matatika.com/api/workspaces/53ad82e6-fad5-417b-ad02-4e81cfb845f8/pipelines/1dcbb564-53c3-474c-8d18-0ab8b5961144/draft",
      "type" : "PUT"
    },
    "self" : {
      "href" : "https://catalog.matatika.com/api/pipelines/1dcbb564-53c3-474c-8d18-0ab8b5961144"
    },
    "jobs" : {
      "href" : "https://catalog.matatika.com/api/pipelines/1dcbb564-53c3-474c-8d18-0ab8b5961144/jobs",
      "type" : "GET"
    },
    "add subscription" : {
      "href" : "https://catalog.matatika.com/api/pipelines/1dcbb564-53c3-474c-8d18-0ab8b5961144/subscriptions"
    },
    "datastore" : {
      "href" : "https://catalog.matatika.com/api/datastores/1e39e629-3622-4546-990a-024fd22927b0"
    },
    "verify pipeline" : {
      "href" : "https://catalog.matatika.com/api/pipelines/1dcbb564-53c3-474c-8d18-0ab8b5961144/verification",
      "type" : "POST"
    },
    "create job" : {
      "href" : "https://catalog.matatika.com/api/pipelines/1dcbb564-53c3-474c-8d18-0ab8b5961144/jobs",
      "type" : "POST"
    },
    "latest job" : {
      "href" : "https://catalog.matatika.com/api/jobs/752e1887-3198-48a9-98a4-f9bf1f4b76d4"
    }
  }
}

Properties

For each setting s in the dataplugin settings:

Path	Type	Description
`s.name`	`s.kind`	Refer to `s.description`

Formats

Pipeline Status

Value	Description
`READY`	The pipeline completed processing resource changes
`PROVISIONING`	The pipeline is processing resource changes
`FAILED`	The pipeline failed to process resource changes

Requests

View all pipelines in a workspace
View a pipeline
Initialise a pipeline in a workspace
Create or update a pipeline in a workspace
Create or update a pipeline as a draft
Validate a pipeline configuration in a workspace
Verify a pipeline
Delete a pipeline

View all pipelines in a workspace

GET

/api/workspaces/{workspace-id}/pipelines

Returns all configured pipelines in the workspace {workspace-id}.

Prerequisites

Workspace {workspace-id} must exist

Request

Example Snippets

cURL

curl -H "Authorization: Bearer $ACCESS_TOKEN" 'https://catalog.matatika.com/api/workspaces/53ad82e6-fad5-417b-ad02-4e81cfb845f8/pipelines' -i -X GET \
    -H 'Accept: application/json, application/javascript, text/javascript, text/json' \
    -H 'Content-Type: application/json'

Python (requests)

import requests

url = "https://catalog.matatika.com/api/workspaces/53ad82e6-fad5-417b-ad02-4e81cfb845f8/pipelines"

headers = {
  'Authorization': ACCESS_TOKEN
}

response = requests.request("GET", url, headers=headers)

print(response.text.encode('utf8'))

Response

200 OK

Pipeline collection with HAL links.

{
  "_embedded" : {
    "pipelines" : [ {
      "id" : "1dcbb564-53c3-474c-8d18-0ab8b5961144",
      "status" : "READY",
      "name" : "SIT-generated pipeline (updated)",
      "schedule" : "0 0 0 25 12 ?",
      "timeout" : 0,
      "created" : "2022-06-30T15:34:15.653809",
      "lastModified" : "2022-06-30T15:34:15.65381",
      "properties" : { },
      "dataSource" : "tap-google-analytics",
      "dataStore" : "1e39e629-3622-4546-990a-024fd22927b0",
      "_embedded" : {
        "dataSource" : {
          "id" : "67caf756-749f-43ab-b086-d91a59bb2ebc",
          "name" : "tap-google-analytics",
          "label" : "Google Analytics",
          "imageUrl" : "/assets/images/datasource/tap-google-analytics.svg",
          "description" : "App and website analytics platform hosted by Google\n\n## Prerequisites\n\nBefore importing data from Google Analytics you or your webmaster will need to have performed some initial setup to grant\nyourself access to the Google Analytics for your website.\n\n[Google Analytics Prerequisites](https://www.matatika.com/docs/data-sources/tap-google-analytics/prerequisites)\n\n## Data source settings\n### View ID\n\nOnce you have connected with your google account, the `View ID` drop down will allow you to choose which view you want to get your data from.\n\n### Start Date\n\nProviding a start date allows you to set how much historical data you wish to import into you workspace. \n\nThis date has to be in the ISO-8601 date format, see the example below.\n\nFormat: `YYYY-MM-DDTHH:MM:SSZ`\n\nExample: `2020-10-01T13:12:11Z`\n\n### End Date\n\nProviding an end date allows you to limit how much data you wish to import into you workspace. \n\nThis date has to be in the ISO-8601 date format, see the example below.\n\nFormat: `YYYY-MM-DDTHH:MM:SSZ`\n\nExample: `2020-10-01T13:12:11Z`\n\n### Reports\n\nProviding the path to a reports json file allows you to supply the tap with a user defined definition of tables and information you want to sync from Google Analytics.\nThis does require you to commit a file to your workspace repository so that Google Analytics can find it during run time, and may also break the dbt transforms that are required to have our included datasets work.\nFor more information on how to define a custom reports file see the [Meltano Hub Documentation](https://hub.meltano.com/extractors/google-analytics#reports).\n\n## Learn more\n\n[Matatika Google Analytics Data Source Documentation](https://www.matatika.com/docs/data-sources/tap-google-analytics)\n\n[Meltano Hub Google Analytics Documentation](https://hub.meltano.com/extractors/google-analytics)"
        },
        "dataStore" : {
          "id" : "1e39e629-3622-4546-990a-024fd22927b0",
          "name" : "Default",
          "imageUrl" : "/assets/images/datastore/target-postgres.png",
          "loader" : "target-postgres--transferwise"
        },
        "latest job" : {
          "id" : "752e1887-3198-48a9-98a4-f9bf1f4b76d4",
          "created" : "2022-06-30T15:34:17.35072",
          "type" : "PIPELINE_CONFIG",
          "exitCode" : 0,
          "status" : "COMPLETE",
          "startTime" : "2022-06-30T15:34:41.313",
          "endTime" : "2022-06-30T15:35:50.585"
        }
      },
      "_links" : {
        "update pipeline" : {
          "href" : "https://catalog.matatika.com/api/workspaces/53ad82e6-fad5-417b-ad02-4e81cfb845f8/pipelines/1dcbb564-53c3-474c-8d18-0ab8b5961144",
          "type" : "PUT"
        },
        "delete pipeline" : {
          "href" : "https://catalog.matatika.com/api/pipelines/1dcbb564-53c3-474c-8d18-0ab8b5961144",
          "type" : "DELETE"
        },
        "draft pipeline" : {
          "href" : "https://catalog.matatika.com/api/workspaces/53ad82e6-fad5-417b-ad02-4e81cfb845f8/pipelines/1dcbb564-53c3-474c-8d18-0ab8b5961144/draft",
          "type" : "PUT"
        },
        "self" : {
          "href" : "https://catalog.matatika.com/api/pipelines/1dcbb564-53c3-474c-8d18-0ab8b5961144"
        },
        "jobs" : {
          "href" : "https://catalog.matatika.com/api/pipelines/1dcbb564-53c3-474c-8d18-0ab8b5961144/jobs",
          "type" : "GET"
        },
        "add subscription" : {
          "href" : "https://catalog.matatika.com/api/pipelines/1dcbb564-53c3-474c-8d18-0ab8b5961144/subscriptions"
        },
        "datastore" : {
          "href" : "https://catalog.matatika.com/api/datastores/1e39e629-3622-4546-990a-024fd22927b0"
        },
        "verify pipeline" : {
          "href" : "https://catalog.matatika.com/api/pipelines/1dcbb564-53c3-474c-8d18-0ab8b5961144/verification",
          "type" : "POST"
        },
        "create job" : {
          "href" : "https://catalog.matatika.com/api/pipelines/1dcbb564-53c3-474c-8d18-0ab8b5961144/jobs",
          "type" : "POST"
        },
        "latest job" : {
          "href" : "https://catalog.matatika.com/api/jobs/752e1887-3198-48a9-98a4-f9bf1f4b76d4"
        }
      }
    }, {
      "id" : "5ff689e1-708e-4007-a2e2-3619edb7ba86",
      "status" : "DRAFT",
      "name" : "SIT-generated pipeline (updated)",
      "schedule" : "0 0 0 25 12 ?",
      "timeout" : 0,
      "created" : "2022-06-30T15:31:44.470698",
      "lastModified" : "2022-06-30T15:31:44.4707",
      "properties" : { },
      "dataSource" : "tap-google-analytics",
      "dataStore" : "1e39e629-3622-4546-990a-024fd22927b0",
      "_embedded" : {
        "dataSource" : {
          "id" : "67caf756-749f-43ab-b086-d91a59bb2ebc",
          "name" : "tap-google-analytics",
          "label" : "Google Analytics",
          "imageUrl" : "/assets/images/datasource/tap-google-analytics.svg",
          "description" : "App and website analytics platform hosted by Google\n\n## Prerequisites\n\nBefore importing data from Google Analytics you or your webmaster will need to have performed some initial setup to grant\nyourself access to the Google Analytics for your website.\n\n[Google Analytics Prerequisites](https://www.matatika.com/docs/data-sources/tap-google-analytics/prerequisites)\n\n## Data source settings\n### View ID\n\nOnce you have connected with your google account, the `View ID` drop down will allow you to choose which view you want to get your data from.\n\n### Start Date\n\nProviding a start date allows you to set how much historical data you wish to import into you workspace. \n\nThis date has to be in the ISO-8601 date format, see the example below.\n\nFormat: `YYYY-MM-DDTHH:MM:SSZ`\n\nExample: `2020-10-01T13:12:11Z`\n\n### End Date\n\nProviding an end date allows you to limit how much data you wish to import into you workspace. \n\nThis date has to be in the ISO-8601 date format, see the example below.\n\nFormat: `YYYY-MM-DDTHH:MM:SSZ`\n\nExample: `2020-10-01T13:12:11Z`\n\n### Reports\n\nProviding the path to a reports json file allows you to supply the tap with a user defined definition of tables and information you want to sync from Google Analytics.\nThis does require you to commit a file to your workspace repository so that Google Analytics can find it during run time, and may also break the dbt transforms that are required to have our included datasets work.\nFor more information on how to define a custom reports file see the [Meltano Hub Documentation](https://hub.meltano.com/extractors/google-analytics#reports).\n\n## Learn more\n\n[Matatika Google Analytics Data Source Documentation](https://www.matatika.com/docs/data-sources/tap-google-analytics)\n\n[Meltano Hub Google Analytics Documentation](https://hub.meltano.com/extractors/google-analytics)"
        },
        "dataStore" : {
          "id" : "1e39e629-3622-4546-990a-024fd22927b0",
          "name" : "Default",
          "imageUrl" : "/assets/images/datastore/target-postgres.png",
          "loader" : "target-postgres--transferwise"
        },
        "latest job" : {
          "id" : "4c6511d0-2fe2-4a4c-b461-e7a1a578803f",
          "created" : "2022-06-30T15:31:45.394542",
          "type" : "PIPELINE_CONFIG",
          "exitCode" : 0,
          "status" : "COMPLETE",
          "startTime" : "2022-06-30T15:32:08.358",
          "endTime" : "2022-06-30T15:34:10.237"
        }
      },
      "_links" : {
        "update pipeline" : {
          "href" : "https://catalog.matatika.com/api/workspaces/53ad82e6-fad5-417b-ad02-4e81cfb845f8/pipelines/5ff689e1-708e-4007-a2e2-3619edb7ba86",
          "type" : "PUT"
        },
        "delete pipeline" : {
          "href" : "https://catalog.matatika.com/api/pipelines/5ff689e1-708e-4007-a2e2-3619edb7ba86",
          "type" : "DELETE"
        },
        "draft pipeline" : {
          "href" : "https://catalog.matatika.com/api/workspaces/53ad82e6-fad5-417b-ad02-4e81cfb845f8/pipelines/5ff689e1-708e-4007-a2e2-3619edb7ba86/draft",
          "type" : "PUT"
        },
        "self" : {
          "href" : "https://catalog.matatika.com/api/pipelines/5ff689e1-708e-4007-a2e2-3619edb7ba86"
        },
        "jobs" : {
          "href" : "https://catalog.matatika.com/api/pipelines/5ff689e1-708e-4007-a2e2-3619edb7ba86/jobs",
          "type" : "GET"
        },
        "add subscription" : {
          "href" : "https://catalog.matatika.com/api/pipelines/5ff689e1-708e-4007-a2e2-3619edb7ba86/subscriptions"
        },
        "datastore" : {
          "href" : "https://catalog.matatika.com/api/datastores/1e39e629-3622-4546-990a-024fd22927b0"
        },
        "verify pipeline" : {
          "href" : "https://catalog.matatika.com/api/pipelines/5ff689e1-708e-4007-a2e2-3619edb7ba86/verification",
          "type" : "POST"
        },
        "latest job" : {
          "href" : "https://catalog.matatika.com/api/jobs/4c6511d0-2fe2-4a4c-b461-e7a1a578803f"
        }
      }
    } ]
  },
  "_links" : {
    "self" : {
      "href" : "https://catalog.matatika.com/api/workspaces/53ad82e6-fad5-417b-ad02-4e81cfb845f8/pipelines?page=0&size=20&sort=name,asc"
    }
  },
  "page" : {
    "size" : 20,
    "totalElements" : 2,
    "totalPages" : 1,
    "number" : 0
  }
}

View a pipeline

GET

/api/pipelines/{pipeline-id}

Returns the pipeline {pipeline-id}.

Prerequisites

Pipeline {pipeline-id} must exist

Request

Example Snippets

cURL

curl -H "Authorization: Bearer $ACCESS_TOKEN" 'https://catalog.matatika.com/api/pipelines/1dcbb564-53c3-474c-8d18-0ab8b5961144' -i -X GET \
    -H 'Accept: application/json, application/javascript, text/javascript, text/json' \
    -H 'Content-Type: application/json'

Python (requests)

import requests

url = "https://catalog.matatika.com/api/pipelines/1dcbb564-53c3-474c-8d18-0ab8b5961144"

headers = {
  'Authorization': ACCESS_TOKEN
}

response = requests.request("GET", url, headers=headers)

print(response.text.encode('utf8'))

Response

200 OK

Pipeline with HAL links.

{
  "id" : "1dcbb564-53c3-474c-8d18-0ab8b5961144",
  "status" : "READY",
  "name" : "SIT-generated pipeline (updated)",
  "schedule" : "0 0 0 25 12 ?",
  "timeout" : 0,
  "created" : "2022-06-30T15:34:15.653809",
  "lastModified" : "2022-06-30T15:34:15.65381",
  "properties" : {
    "end_date" : "2022-06-30T16:31:44.141753+01:00",
    "oauth_credentials.access_token" : "access_token",
    "reports" : "reports",
    "oauth_credentials.refresh_token" : "refresh_token",
    "view_id" : "1234567890",
    "oauth_credentials.client_id" : "client_id",
    "oauth_credentials.client_secret" : "client_secret",
    "start_date" : "2022-05-30T16:31:44.141674+01:00"
  },
  "dataSource" : "tap-google-analytics",
  "dataStore" : "1e39e629-3622-4546-990a-024fd22927b0",
  "_embedded" : {
    "dataSource" : {
      "id" : "67caf756-749f-43ab-b086-d91a59bb2ebc",
      "name" : "tap-google-analytics",
      "label" : "Google Analytics",
      "imageUrl" : "/assets/images/datasource/tap-google-analytics.svg",
      "description" : "App and website analytics platform hosted by Google\n\n## Prerequisites\n\nBefore importing data from Google Analytics you or your webmaster will need to have performed some initial setup to grant\nyourself access to the Google Analytics for your website.\n\n[Google Analytics Prerequisites](https://www.matatika.com/docs/data-sources/tap-google-analytics/prerequisites)\n\n## Data source settings\n### View ID\n\nOnce you have connected with your google account, the `View ID` drop down will allow you to choose which view you want to get your data from.\n\n### Start Date\n\nProviding a start date allows you to set how much historical data you wish to import into you workspace. \n\nThis date has to be in the ISO-8601 date format, see the example below.\n\nFormat: `YYYY-MM-DDTHH:MM:SSZ`\n\nExample: `2020-10-01T13:12:11Z`\n\n### End Date\n\nProviding an end date allows you to limit how much data you wish to import into you workspace. \n\nThis date has to be in the ISO-8601 date format, see the example below.\n\nFormat: `YYYY-MM-DDTHH:MM:SSZ`\n\nExample: `2020-10-01T13:12:11Z`\n\n### Reports\n\nProviding the path to a reports json file allows you to supply the tap with a user defined definition of tables and information you want to sync from Google Analytics.\nThis does require you to commit a file to your workspace repository so that Google Analytics can find it during run time, and may also break the dbt transforms that are required to have our included datasets work.\nFor more information on how to define a custom reports file see the [Meltano Hub Documentation](https://hub.meltano.com/extractors/google-analytics#reports).\n\n## Learn more\n\n[Matatika Google Analytics Data Source Documentation](https://www.matatika.com/docs/data-sources/tap-google-analytics)\n\n[Meltano Hub Google Analytics Documentation](https://hub.meltano.com/extractors/google-analytics)"
    },
    "dataStore" : {
      "id" : "1e39e629-3622-4546-990a-024fd22927b0",
      "name" : "Default",
      "imageUrl" : "/assets/images/datastore/target-postgres.png",
      "loader" : "target-postgres--transferwise"
    },
    "latest job" : {
      "id" : "752e1887-3198-48a9-98a4-f9bf1f4b76d4",
      "created" : "2022-06-30T15:34:17.35072",
      "type" : "PIPELINE_CONFIG",
      "exitCode" : 0,
      "status" : "COMPLETE",
      "startTime" : "2022-06-30T15:34:41.313",
      "endTime" : "2022-06-30T15:35:50.585",
      "_links" : {
        "self" : {
          "href" : "https://catalog.matatika.com/api/jobs/752e1887-3198-48a9-98a4-f9bf1f4b76d4"
        },
        "delete job" : {
          "href" : "https://catalog.matatika.com/api/jobs/752e1887-3198-48a9-98a4-f9bf1f4b76d4",
          "type" : "DELETE"
        },
        "logs" : {
          "href" : "https://catalog.matatika.com/api/jobs/752e1887-3198-48a9-98a4-f9bf1f4b76d4/logs?sequence=0",
          "type" : "GET"
        }
      }
    }
  },
  "_links" : {
    "update pipeline" : {
      "href" : "https://catalog.matatika.com/api/workspaces/53ad82e6-fad5-417b-ad02-4e81cfb845f8/pipelines/1dcbb564-53c3-474c-8d18-0ab8b5961144",
      "type" : "PUT"
    },
    "delete pipeline" : {
      "href" : "https://catalog.matatika.com/api/pipelines/1dcbb564-53c3-474c-8d18-0ab8b5961144",
      "type" : "DELETE"
    },
    "draft pipeline" : {
      "href" : "https://catalog.matatika.com/api/workspaces/53ad82e6-fad5-417b-ad02-4e81cfb845f8/pipelines/1dcbb564-53c3-474c-8d18-0ab8b5961144/draft",
      "type" : "PUT"
    },
    "self" : {
      "href" : "https://catalog.matatika.com/api/pipelines/1dcbb564-53c3-474c-8d18-0ab8b5961144"
    },
    "jobs" : {
      "href" : "https://catalog.matatika.com/api/pipelines/1dcbb564-53c3-474c-8d18-0ab8b5961144/jobs",
      "type" : "GET"
    },
    "add subscription" : {
      "href" : "https://catalog.matatika.com/api/pipelines/1dcbb564-53c3-474c-8d18-0ab8b5961144/subscriptions"
    },
    "datastore" : {
      "href" : "https://catalog.matatika.com/api/datastores/1e39e629-3622-4546-990a-024fd22927b0"
    },
    "verify pipeline" : {
      "href" : "https://catalog.matatika.com/api/pipelines/1dcbb564-53c3-474c-8d18-0ab8b5961144/verification",
      "type" : "POST"
    },
    "create job" : {
      "href" : "https://catalog.matatika.com/api/pipelines/1dcbb564-53c3-474c-8d18-0ab8b5961144/jobs",
      "type" : "POST"
    },
    "latest job" : {
      "href" : "https://catalog.matatika.com/api/jobs/752e1887-3198-48a9-98a4-f9bf1f4b76d4"
    }
  }
}

Initialise a pipeline in a workspace

POST

/api/workspaces/{workspace-id}/pipelines

Initialises a new pipeline in the workspace {workspace-id}.

Prerequisites

Workspace {workspace-id} must exist

Request

Example Snippets

cURL

curl -H "Authorization: Bearer $ACCESS_TOKEN" 'https://catalog.matatika.com/api/workspaces/53ad82e6-fad5-417b-ad02-4e81cfb845f8/pipelines' -i -X POST \
    -H 'Accept: application/json, application/javascript, text/javascript, text/json' \
    -H 'Content-Type: application/json'

Python (requests)

import requests

url = "https://catalog.matatika.com/api/workspaces/53ad82e6-fad5-417b-ad02-4e81cfb845f8/pipelines"

headers = {
  'Authorization': ACCESS_TOKEN
}

response = requests.request("POST", url, headers=headers)

print(response.text.encode('utf8'))

Response

200 OK

Pipeline with HAL links.

{
  "id" : "5ff689e1-708e-4007-a2e2-3619edb7ba86",
  "status" : "PROVISIONING",
  "timeout" : 0,
  "created" : "2022-06-30T15:31:43.28017",
  "lastModified" : "2022-06-30T15:31:43.280172",
  "properties" : { },
  "_links" : {
    "create pipeline" : {
      "href" : "https://catalog.matatika.com/api/workspaces/53ad82e6-fad5-417b-ad02-4e81cfb845f8/pipelines/5ff689e1-708e-4007-a2e2-3619edb7ba86",
      "type" : "PUT"
    },
    "draft pipeline" : {
      "href" : "https://catalog.matatika.com/api/workspaces/53ad82e6-fad5-417b-ad02-4e81cfb845f8/pipelines/5ff689e1-708e-4007-a2e2-3619edb7ba86/draft",
      "type" : "PUT"
    },
    "validate pipeline" : {
      "href" : "https://catalog.matatika.com/api/workspaces/53ad82e6-fad5-417b-ad02-4e81cfb845f8/pipelines/validation",
      "type" : "POST"
    }
  }
}

Create or update a pipeline in a workspace

PUT

/api/workspaces/{workspace-id}/pipelines/{pipeline-id}

Creates or updates the pipeline {pipeline-id} in the workspace {workspace-id}.

Prerequisites

Workspace {workspace-id} must exist

Request

Body

Pipeline resource.

{
  "name" : "SIT-generated pipeline",
  "dataSource" : "tap-google-analytics",
  "schedule" : "0 0 0 25 12 ?",
  "properties" : {
    "end_date" : "2022-06-30T16:31:44.141753+01:00",
    "oauth_credentials.access_token" : "access_token",
    "reports" : "reports",
    "view_id" : "1234567890",
    "oauth_credentials.refresh_token" : "refresh_token",
    "oauth_credentials.client_id" : "client_id",
    "oauth_credentials.client_secret" : "client_secret",
    "start_date" : "2022-05-30T16:31:44.141674+01:00"
  }
}

Example Snippets

cURL

curl -H "Authorization: Bearer $ACCESS_TOKEN" 'https://catalog.matatika.com/api/workspaces/53ad82e6-fad5-417b-ad02-4e81cfb845f8/pipelines/5ff689e1-708e-4007-a2e2-3619edb7ba86' -i -X PUT \
    -H 'Accept: application/json, application/javascript, text/javascript, text/json' \
    -H 'Content-Type: application/json' \
    -d '{
  "name" : "SIT-generated pipeline",
  "dataSource" : "tap-google-analytics",
  "schedule" : "0 0 0 25 12 ?",
  "properties" : {
    "end_date" : "2022-06-30T16:31:44.141753+01:00",
    "oauth_credentials.access_token" : "access_token",
    "reports" : "reports",
    "view_id" : "1234567890",
    "oauth_credentials.refresh_token" : "refresh_token",
    "oauth_credentials.client_id" : "client_id",
    "oauth_credentials.client_secret" : "client_secret",
    "start_date" : "2022-05-30T16:31:44.141674+01:00"
  }
}'

Python (requests)

import requests

url = "https://catalog.matatika.com/api/workspaces/53ad82e6-fad5-417b-ad02-4e81cfb845f8/pipelines/5ff689e1-708e-4007-a2e2-3619edb7ba86"

data = {
  "name" : "SIT-generated pipeline",
  "dataSource" : "tap-google-analytics",
  "schedule" : "0 0 0 25 12 ?",
  "properties" : {
    "end_date" : "2022-06-30T16:31:44.141753+01:00",
    "oauth_credentials.access_token" : "access_token",
    "reports" : "reports",
    "view_id" : "1234567890",
    "oauth_credentials.refresh_token" : "refresh_token",
    "oauth_credentials.client_id" : "client_id",
    "oauth_credentials.client_secret" : "client_secret",
    "start_date" : "2022-05-30T16:31:44.141674+01:00"
  }
}
headers = {
  'Authorization': ACCESS_TOKEN
}

response = requests.request("PUT", url, headers=headers, data=data)

print(response.text.encode('utf8'))

Response

200 OK / 201 Created

Pipeline with HAL links.

{
  "id" : "5ff689e1-708e-4007-a2e2-3619edb7ba86",
  "status" : "PROVISIONING",
  "name" : "SIT-generated pipeline",
  "schedule" : "0 0 0 25 12 ?",
  "timeout" : 0,
  "created" : "2022-06-30T15:31:44.470698",
  "lastModified" : "2022-06-30T15:31:44.4707",
  "properties" : {
    "end_date" : "2022-06-30T16:31:44.141753+01:00",
    "oauth_credentials.access_token" : "access_token",
    "reports" : "reports",
    "oauth_credentials.refresh_token" : "refresh_token",
    "view_id" : "1234567890",
    "oauth_credentials.client_id" : "client_id",
    "oauth_credentials.client_secret" : "client_secret",
    "start_date" : "2022-05-30T16:31:44.141674+01:00"
  },
  "dataSource" : "tap-google-analytics",
  "dataStore" : "1e39e629-3622-4546-990a-024fd22927b0",
  "_embedded" : {
    "dataSource" : {
      "id" : "67caf756-749f-43ab-b086-d91a59bb2ebc",
      "name" : "tap-google-analytics",
      "label" : "Google Analytics",
      "imageUrl" : "/assets/images/datasource/tap-google-analytics.svg",
      "description" : "App and website analytics platform hosted by Google\n\n## Prerequisites\n\nBefore importing data from Google Analytics you or your webmaster will need to have performed some initial setup to grant\nyourself access to the Google Analytics for your website.\n\n[Google Analytics Prerequisites](https://www.matatika.com/docs/data-sources/tap-google-analytics/prerequisites)\n\n## Data source settings\n### View ID\n\nOnce you have connected with your google account, the `View ID` drop down will allow you to choose which view you want to get your data from.\n\n### Start Date\n\nProviding a start date allows you to set how much historical data you wish to import into you workspace. \n\nThis date has to be in the ISO-8601 date format, see the example below.\n\nFormat: `YYYY-MM-DDTHH:MM:SSZ`\n\nExample: `2020-10-01T13:12:11Z`\n\n### End Date\n\nProviding an end date allows you to limit how much data you wish to import into you workspace. \n\nThis date has to be in the ISO-8601 date format, see the example below.\n\nFormat: `YYYY-MM-DDTHH:MM:SSZ`\n\nExample: `2020-10-01T13:12:11Z`\n\n### Reports\n\nProviding the path to a reports json file allows you to supply the tap with a user defined definition of tables and information you want to sync from Google Analytics.\nThis does require you to commit a file to your workspace repository so that Google Analytics can find it during run time, and may also break the dbt transforms that are required to have our included datasets work.\nFor more information on how to define a custom reports file see the [Meltano Hub Documentation](https://hub.meltano.com/extractors/google-analytics#reports).\n\n## Learn more\n\n[Matatika Google Analytics Data Source Documentation](https://www.matatika.com/docs/data-sources/tap-google-analytics)\n\n[Meltano Hub Google Analytics Documentation](https://hub.meltano.com/extractors/google-analytics)"
    },
    "dataStore" : {
      "id" : "1e39e629-3622-4546-990a-024fd22927b0",
      "name" : "Default",
      "imageUrl" : "/assets/images/datastore/target-postgres.png",
      "loader" : "target-postgres--transferwise"
    },
    "latest job" : {
      "id" : "4c6511d0-2fe2-4a4c-b461-e7a1a578803f",
      "created" : "2022-06-30T15:31:45.394542",
      "type" : "PIPELINE_CONFIG",
      "status" : "QUEUED",
      "_links" : {
        "self" : {
          "href" : "https://catalog.matatika.com/api/jobs/4c6511d0-2fe2-4a4c-b461-e7a1a578803f"
        },
        "delete job" : {
          "href" : "https://catalog.matatika.com/api/jobs/4c6511d0-2fe2-4a4c-b461-e7a1a578803f",
          "type" : "DELETE"
        },
        "logs" : {
          "href" : "https://catalog.matatika.com/api/jobs/4c6511d0-2fe2-4a4c-b461-e7a1a578803f/logs?sequence=0",
          "type" : "GET"
        },
        "withdraw job" : {
          "href" : "https://catalog.matatika.com/api/jobs/4c6511d0-2fe2-4a4c-b461-e7a1a578803f/stopped",
          "type" : "PUT"
        }
      }
    }
  },
  "_links" : {
    "update pipeline" : {
      "href" : "https://catalog.matatika.com/api/workspaces/53ad82e6-fad5-417b-ad02-4e81cfb845f8/pipelines/5ff689e1-708e-4007-a2e2-3619edb7ba86",
      "type" : "PUT"
    },
    "delete pipeline" : {
      "href" : "https://catalog.matatika.com/api/pipelines/5ff689e1-708e-4007-a2e2-3619edb7ba86",
      "type" : "DELETE"
    },
    "draft pipeline" : {
      "href" : "https://catalog.matatika.com/api/workspaces/53ad82e6-fad5-417b-ad02-4e81cfb845f8/pipelines/5ff689e1-708e-4007-a2e2-3619edb7ba86/draft",
      "type" : "PUT"
    },
    "self" : {
      "href" : "https://catalog.matatika.com/api/pipelines/5ff689e1-708e-4007-a2e2-3619edb7ba86"
    },
    "jobs" : {
      "href" : "https://catalog.matatika.com/api/pipelines/5ff689e1-708e-4007-a2e2-3619edb7ba86/jobs",
      "type" : "GET"
    },
    "add subscription" : {
      "href" : "https://catalog.matatika.com/api/pipelines/5ff689e1-708e-4007-a2e2-3619edb7ba86/subscriptions"
    },
    "datastore" : {
      "href" : "https://catalog.matatika.com/api/datastores/1e39e629-3622-4546-990a-024fd22927b0"
    },
    "withdraw job" : {
      "href" : "https://catalog.matatika.com/api/jobs/4c6511d0-2fe2-4a4c-b461-e7a1a578803f/stopped",
      "type" : "PUT"
    },
    "latest job" : {
      "href" : "https://catalog.matatika.com/api/jobs/4c6511d0-2fe2-4a4c-b461-e7a1a578803f"
    }
  }
}

Create or update a pipeline as a draft

PUT

/api/workspaces/{workspace-id}/pipelines/{pipeline-id}/draft

Creates or updates the pipeline {pipeline-id} in the workspace {workspace-id} as a draft.

Prerequisites

Workspace {workspace-id} must exist

Request

Body

Pipeline resource.

{
  "name" : "SIT-generated pipeline",
  "dataSource" : "tap-google-analytics",
  "schedule" : "0 0 0 25 12 ?",
  "properties" : {
    "end_date" : "2022-06-30T16:31:44.141753+01:00",
    "oauth_credentials.access_token" : "access_token",
    "reports" : "reports",
    "view_id" : "1234567890",
    "oauth_credentials.refresh_token" : "refresh_token",
    "oauth_credentials.client_id" : "client_id",
    "oauth_credentials.client_secret" : "client_secret",
    "start_date" : "2022-05-30T16:31:44.141674+01:00"
  }
}

Example Snippets

cURL

curl -H "Authorization: Bearer $ACCESS_TOKEN" 'https://catalog.matatika.com/api/workspaces/53ad82e6-fad5-417b-ad02-4e81cfb845f8/pipelines/1dcbb564-53c3-474c-8d18-0ab8b5961144/draft' -i -X PUT \
    -H 'Accept: application/json, application/javascript, text/javascript, text/json' \
    -H 'Content-Type: application/json' \
    -d '{
  "name" : "SIT-generated pipeline",
  "dataSource" : "tap-google-analytics",
  "schedule" : "0 0 0 25 12 ?",
  "properties" : {
    "end_date" : "2022-06-30T16:31:44.141753+01:00",
    "oauth_credentials.access_token" : "access_token",
    "reports" : "reports",
    "view_id" : "1234567890",
    "oauth_credentials.refresh_token" : "refresh_token",
    "oauth_credentials.client_id" : "client_id",
    "oauth_credentials.client_secret" : "client_secret",
    "start_date" : "2022-05-30T16:31:44.141674+01:00"
  }
}'

Python (requests)

import requests

url = "https://catalog.matatika.com/api/workspaces/53ad82e6-fad5-417b-ad02-4e81cfb845f8/pipelines/1dcbb564-53c3-474c-8d18-0ab8b5961144/draft"

data = {
  "name" : "SIT-generated pipeline",
  "dataSource" : "tap-google-analytics",
  "schedule" : "0 0 0 25 12 ?",
  "properties" : {
    "end_date" : "2022-06-30T16:31:44.141753+01:00",
    "oauth_credentials.access_token" : "access_token",
    "reports" : "reports",
    "view_id" : "1234567890",
    "oauth_credentials.refresh_token" : "refresh_token",
    "oauth_credentials.client_id" : "client_id",
    "oauth_credentials.client_secret" : "client_secret",
    "start_date" : "2022-05-30T16:31:44.141674+01:00"
  }
}
headers = {
  'Authorization': ACCESS_TOKEN
}

response = requests.request("PUT", url, headers=headers, data=data)

print(response.text.encode('utf8'))

Response

200 OK / 201 Created

Pipeline with HAL links.

{
  "id" : "1dcbb564-53c3-474c-8d18-0ab8b5961144",
  "status" : "PROVISIONING",
  "name" : "SIT-generated pipeline",
  "schedule" : "0 0 0 25 12 ?",
  "timeout" : 0,
  "created" : "2022-06-30T15:34:15.653809",
  "lastModified" : "2022-06-30T15:34:15.65381",
  "properties" : {
    "end_date" : "2022-06-30T16:31:44.141753+01:00",
    "oauth_credentials.access_token" : "access_token",
    "reports" : "reports",
    "oauth_credentials.refresh_token" : "refresh_token",
    "view_id" : "1234567890",
    "oauth_credentials.client_id" : "client_id",
    "oauth_credentials.client_secret" : "client_secret",
    "start_date" : "2022-05-30T16:31:44.141674+01:00"
  },
  "dataSource" : "tap-google-analytics",
  "dataStore" : "1e39e629-3622-4546-990a-024fd22927b0",
  "_embedded" : {
    "dataSource" : {
      "id" : "67caf756-749f-43ab-b086-d91a59bb2ebc",
      "name" : "tap-google-analytics",
      "label" : "Google Analytics",
      "imageUrl" : "/assets/images/datasource/tap-google-analytics.svg",
      "description" : "App and website analytics platform hosted by Google\n\n## Prerequisites\n\nBefore importing data from Google Analytics you or your webmaster will need to have performed some initial setup to grant\nyourself access to the Google Analytics for your website.\n\n[Google Analytics Prerequisites](https://www.matatika.com/docs/data-sources/tap-google-analytics/prerequisites)\n\n## Data source settings\n### View ID\n\nOnce you have connected with your google account, the `View ID` drop down will allow you to choose which view you want to get your data from.\n\n### Start Date\n\nProviding a start date allows you to set how much historical data you wish to import into you workspace. \n\nThis date has to be in the ISO-8601 date format, see the example below.\n\nFormat: `YYYY-MM-DDTHH:MM:SSZ`\n\nExample: `2020-10-01T13:12:11Z`\n\n### End Date\n\nProviding an end date allows you to limit how much data you wish to import into you workspace. \n\nThis date has to be in the ISO-8601 date format, see the example below.\n\nFormat: `YYYY-MM-DDTHH:MM:SSZ`\n\nExample: `2020-10-01T13:12:11Z`\n\n### Reports\n\nProviding the path to a reports json file allows you to supply the tap with a user defined definition of tables and information you want to sync from Google Analytics.\nThis does require you to commit a file to your workspace repository so that Google Analytics can find it during run time, and may also break the dbt transforms that are required to have our included datasets work.\nFor more information on how to define a custom reports file see the [Meltano Hub Documentation](https://hub.meltano.com/extractors/google-analytics#reports).\n\n## Learn more\n\n[Matatika Google Analytics Data Source Documentation](https://www.matatika.com/docs/data-sources/tap-google-analytics)\n\n[Meltano Hub Google Analytics Documentation](https://hub.meltano.com/extractors/google-analytics)"
    },
    "dataStore" : {
      "id" : "1e39e629-3622-4546-990a-024fd22927b0",
      "name" : "Default",
      "imageUrl" : "/assets/images/datastore/target-postgres.png",
      "loader" : "target-postgres--transferwise"
    },
    "latest job" : {
      "id" : "752e1887-3198-48a9-98a4-f9bf1f4b76d4",
      "created" : "2022-06-30T15:34:17.35072",
      "type" : "PIPELINE_CONFIG",
      "status" : "QUEUED",
      "_links" : {
        "self" : {
          "href" : "https://catalog.matatika.com/api/jobs/752e1887-3198-48a9-98a4-f9bf1f4b76d4"
        },
        "delete job" : {
          "href" : "https://catalog.matatika.com/api/jobs/752e1887-3198-48a9-98a4-f9bf1f4b76d4",
          "type" : "DELETE"
        },
        "logs" : {
          "href" : "https://catalog.matatika.com/api/jobs/752e1887-3198-48a9-98a4-f9bf1f4b76d4/logs?sequence=0",
          "type" : "GET"
        },
        "withdraw job" : {
          "href" : "https://catalog.matatika.com/api/jobs/752e1887-3198-48a9-98a4-f9bf1f4b76d4/stopped",
          "type" : "PUT"
        }
      }
    }
  },
  "_links" : {
    "update pipeline" : {
      "href" : "https://catalog.matatika.com/api/workspaces/53ad82e6-fad5-417b-ad02-4e81cfb845f8/pipelines/1dcbb564-53c3-474c-8d18-0ab8b5961144",
      "type" : "PUT"
    },
    "delete pipeline" : {
      "href" : "https://catalog.matatika.com/api/pipelines/1dcbb564-53c3-474c-8d18-0ab8b5961144",
      "type" : "DELETE"
    },
    "draft pipeline" : {
      "href" : "https://catalog.matatika.com/api/workspaces/53ad82e6-fad5-417b-ad02-4e81cfb845f8/pipelines/1dcbb564-53c3-474c-8d18-0ab8b5961144/draft",
      "type" : "PUT"
    },
    "self" : {
      "href" : "https://catalog.matatika.com/api/pipelines/1dcbb564-53c3-474c-8d18-0ab8b5961144"
    },
    "jobs" : {
      "href" : "https://catalog.matatika.com/api/pipelines/1dcbb564-53c3-474c-8d18-0ab8b5961144/jobs",
      "type" : "GET"
    },
    "add subscription" : {
      "href" : "https://catalog.matatika.com/api/pipelines/1dcbb564-53c3-474c-8d18-0ab8b5961144/subscriptions"
    },
    "datastore" : {
      "href" : "https://catalog.matatika.com/api/datastores/1e39e629-3622-4546-990a-024fd22927b0"
    },
    "withdraw job" : {
      "href" : "https://catalog.matatika.com/api/jobs/752e1887-3198-48a9-98a4-f9bf1f4b76d4/stopped",
      "type" : "PUT"
    },
    "latest job" : {
      "href" : "https://catalog.matatika.com/api/jobs/752e1887-3198-48a9-98a4-f9bf1f4b76d4"
    }
  }
}

Validate a pipeline configuration in a workspace

POST

/api/workspaces/{workspace-id}/pipelines/validation

Validates a pipeline configuration in the workspace {workspace-id}.

Prerequisites

Workspace {workspace-id} must exist

Request

Body

Pipeline resource.

{
  "name" : "SIT-generated pipeline",
  "dataSource" : "tap-google-analytics"
}

Example Snippets

cURL

curl -H "Authorization: Bearer $ACCESS_TOKEN" 'https://catalog.matatika.com/api/workspaces/53ad82e6-fad5-417b-ad02-4e81cfb845f8/pipelines/validation' -i -X POST \
    -H 'Accept: application/json, application/javascript, text/javascript, text/json' \
    -H 'Content-Type: application/json' \
    -d '{
  "name" : "SIT-generated pipeline",
  "dataSource" : "tap-google-analytics"
}'

Python (requests)

import requests

url = "https://catalog.matatika.com/api/workspaces/53ad82e6-fad5-417b-ad02-4e81cfb845f8/pipelines/validation"

data = {
  "name" : "SIT-generated pipeline",
  "dataSource" : "tap-google-analytics"
}
headers = {
  'Authorization': ACCESS_TOKEN
}

response = requests.request("POST", url, headers=headers, data=data)

print(response.text.encode('utf8'))

Response

200 OK

No response body provided.

400 Bad Request

Pipeline property validation errors.

{
  "timestamp" : "2022-06-30T15:31:44.0997",
  "status" : 400,
  "error" : "Bad Request",
  "message" : "Validation failed for object='resource'. Error count: 4",
  "errors" : [ {
    "codes" : [ "NotBlank.oauth_credentials.access_token", "NotBlank" ],
    "defaultMessage" : "No value given for setting",
    "objectName" : "resource",
    "field" : "properties.oauth_credentials.access_token",
    "bindingFailure" : true,
    "code" : "NotBlank"
  }, {
    "codes" : [ "NotBlank.oauth_credentials.refresh_token", "NotBlank" ],
    "defaultMessage" : "No value given for setting",
    "objectName" : "resource",
    "field" : "properties.oauth_credentials.refresh_token",
    "bindingFailure" : true,
    "code" : "NotBlank"
  }, {
    "codes" : [ "NotBlank.view_id", "NotBlank" ],
    "defaultMessage" : "No value given for setting",
    "objectName" : "resource",
    "field" : "properties.view_id",
    "bindingFailure" : true,
    "code" : "NotBlank"
  }, {
    "codes" : [ "NotBlank.start_date", "NotBlank" ],
    "defaultMessage" : "No value given for setting",
    "objectName" : "resource",
    "field" : "properties.start_date",
    "bindingFailure" : true,
    "code" : "NotBlank"
  } ],
  "path" : "/api/workspaces/53ad82e6-fad5-417b-ad02-4e81cfb845f8/pipelines/validation"
}

Verify a pipeline

POST

/api/pipelines/{pipeline-id}/verification

Verifies the configuration of the pipeline {pipeline-id}.

Prerequisites

Pipeline {pipeline-id} must exist

Request

Example Snippets

cURL

curl -H "Authorization: Bearer $ACCESS_TOKEN" 'https://catalog.matatika.com/api/pipelines/bb9c9375-c67f-4c5b-9b4d-0de3ef770e6d/verification' -i -X POST \
    -H 'Accept: application/json, application/javascript, text/javascript, text/json' \
    -H 'Content-Type: application/json'

Python (requests)

import requests

url = "https://catalog.matatika.com/api/pipelines/bb9c9375-c67f-4c5b-9b4d-0de3ef770e6d/verification"

headers = {
  'Authorization': ACCESS_TOKEN
}

response = requests.request("POST", url, headers=headers)

print(response.text.encode('utf8'))

Response

200 OK

Job with HAL links.

{
  "id" : "923877ed-2951-48a9-9668-1d7316cfbb2a",
  "created" : "2022-06-30T15:27:43.483708",
  "type" : "PIPELINE_VERIFY",
  "status" : "QUEUED",
  "_links" : {
    "self" : {
      "href" : "https://catalog.matatika.com/api/jobs/923877ed-2951-48a9-9668-1d7316cfbb2a"
    },
    "delete job" : {
      "href" : "https://catalog.matatika.com/api/jobs/923877ed-2951-48a9-9668-1d7316cfbb2a",
      "type" : "DELETE"
    },
    "logs" : {
      "href" : "https://catalog.matatika.com/api/jobs/923877ed-2951-48a9-9668-1d7316cfbb2a/logs?sequence=0",
      "type" : "GET"
    },
    "withdraw job" : {
      "href" : "https://catalog.matatika.com/api/jobs/923877ed-2951-48a9-9668-1d7316cfbb2a/stopped",
      "type" : "PUT"
    }
  }
}

Delete a pipeline

DELETE

/api/pipelines/{pipeline-id}

Deletes the pipeline {pipeline-id}.

Prerequisites

Pipeline {pipeline-id} must exist

Request

Example Snippets

cURL

curl -H "Authorization: Bearer $ACCESS_TOKEN" 'https://catalog.matatika.com/api/pipelines/1dcbb564-53c3-474c-8d18-0ab8b5961144' -i -X DELETE \
    -H 'Accept: application/json, application/javascript, text/javascript, text/json' \
    -H 'Content-Type: application/json'

Python (requests)

import requests

url = "https://catalog.matatika.com/api/pipelines/1dcbb564-53c3-474c-8d18-0ab8b5961144"

headers = {
  'Authorization': ACCESS_TOKEN
}

response = requests.request("DELETE", url, headers=headers)

print(response.text.encode('utf8'))

Response

204 No Content

No response body provided.

Pipelines

Objects

Pipeline

Properties

Formats

Pipeline Status

Requests

See Also

View all pipelines in a workspace

Prerequisites

Request

Example Snippets

Response

View a pipeline

Prerequisites

Request

Example Snippets

Response

Initialise a pipeline in a workspace

Prerequisites

Request

Example Snippets

Response

Create or update a pipeline in a workspace

Prerequisites

Request

Body

Example Snippets

Response

Create or update a pipeline as a draft

Prerequisites

Request

Body

Example Snippets

Response

Validate a pipeline configuration in a workspace

Prerequisites

Request

Body

Example Snippets

Response

Verify a pipeline

Prerequisites

Request

Example Snippets

Response

Delete a pipeline

Prerequisites

Request

Example Snippets

Response