Link Search Menu Expand Document

Pipelines

A pipeline defines a set of runnable actions composed from datacomponents to complete a set of tasks - for example, ELT. Pipelines are run as jobs, either manually or on a predetermined schedule. Only a single pipeline can be run at any given time.

Objects

Pipeline

Path Type Format Description
id String Version 4 UUID The pipeline ID
status String Pipeline Status  
name String   The pipeline name
schedule String Cron The interval at which to launch a new job e.g. 0 0 9-17 * * MON-FRI launches a job on the hour nine-to-five weekdays
timeout Integer Unsigned The number of seconds after which the job will terminate - if set to 0, an implicit default value of 300 seconds is used
script String Bash script Custom script to execute during a job
created String ISO 8601 timestamp When the pipeline was created
lastModified String ISO 8601 timestamp When the pipeline was last modified
properties Properties   The pipeline properties, defined by the dataplugin settings of each datacomponent
dataComponents Array of String Array of Datacomponent names The pipeline datacomponent names
actions Array of String Array of Dataplugin names or commands The pipeline actions to run during a job 
{
  "id" : "c3bb4dcd-63e7-4811-862e-99da31e488e4",
  "status" : "PROVISIONING",
  "name" : "SIT-generated pipeline [2022-11-04T15:19:49.097677]",
  "schedule" : "0 0 0 25 12 ?",
  "timeout" : 0,
  "created" : "2022-11-04T15:19:49.734323",
  "lastModified" : "2022-11-04T15:19:49.734323",
  "properties" : {
    "tap-google-analytics.view_id" : "1234567890",
    "tap-google-analytics.oauth_credentials.client_secret" : "client_secret",
    "tap-google-analytics.reports" : "reports",
    "tap-google-analytics.oauth_credentials.refresh_token" : "refresh_token",
    "tap-google-analytics.start_date" : "2022-10-04T15:19:49.462298Z",
    "tap-google-analytics.end_date" : "2022-11-04T15:19:49.462404Z",
    "tap-google-analytics.oauth_credentials.client_id" : "client_id",
    "tap-google-analytics.oauth_credentials.access_token" : "access_token"
  },
  "dataComponents" : [ "tap-google-analytics", "Warehouse", "dbt" ],
  "actions" : [ ],
  "_embedded" : {
    "dataComponents" : [ {
      "id" : "1a355ff6-051f-4a00-b510-2d51321147b5",
      "created" : "2022-11-04T15:19:48.729983",
      "lastModified" : "2022-11-04T15:19:48.729983",
      "name" : "tap-google-analytics",
      "properties" : { },
      "commands" : { },
      "_embedded" : {
        "dataplugin" : {
          "id" : "e6414e68-225e-43a6-8e2e-800f3d89512e",
          "pluginType" : "EXTRACTOR",
          "name" : "tap-google-analytics",
          "namespace" : "tap_google_analytics",
          "variant" : "meltano",
          "label" : "Google Analytics",
          "description" : "App and website analytics platform hosted by Google\n\n## Prerequisites\n\nBefore importing data from Google Analytics you or your webmaster will need to have performed some initial setup to grant\nyourself access to the Google Analytics for your website.\n\n[Google Analytics Prerequisites](https://www.matatika.com/docs/data-sources/tap-google-analytics/prerequisites)\n\n## Data source settings\n### View ID\n\nOnce you have connected with your google account, the `View ID` drop down will allow you to choose which view you want to get your data from.\n\n### Start Date\n\nProviding a start date allows you to set how much historical data you wish to import into you workspace.\n\nThis date has to be in the ISO-8601 date format, see the example below.\n\nFormat: `YYYY-MM-DDTHH:MM:SSZ`\n\nExample: `2020-10-01T13:12:11Z`\n\n### End Date\n\nProviding an end date allows you to limit how much data you wish to import into you workspace.\n\nThis date has to be in the ISO-8601 date format, see the example below.\n\nFormat: `YYYY-MM-DDTHH:MM:SSZ`\n\nExample: `2020-10-01T13:12:11Z`\n\n### Reports\n\nProviding the path to a reports json file allows you to supply the tap with a user defined definition of tables and information you want to sync from Google Analytics.\nThis does require you to commit a file to your workspace repository so that Google Analytics can find it during run time, and may also break the dbt transforms that are required to have our included datasets work.\nFor more information on how to define a custom reports file see the [Meltano Hub Documentation](https://hub.meltano.com/extractors/google-analytics#reports).\n\n## Learn more\n\n[Matatika Google Analytics Data Source Documentation](https://www.matatika.com/docs/data-sources/tap-google-analytics)\n\n[Meltano Hub Google Analytics Documentation](https://hub.meltano.com/extractors/google-analytics)",
          "logoUrl" : "/assets/images/datasource/tap-google-analytics.svg",
          "hidden" : false,
          "docs" : "https://www.matatika.com/docs/data-sources/tap-google-analytics/",
          "pipUrl" : "git+https://github.com/Matatika/[email protected]",
          "repo" : "https://github.com/Matatika/tap-google-analytics",
          "capabilities" : [ "CATALOG", "DISCOVER", "STATE" ],
          "select" : [ ],
          "update" : { },
          "vars" : { },
          "settings" : [ {
            "name" : "oauth_credentials.authorization_url",
            "aliases" : [ ],
            "label" : "OAuth identity provider authorization endpoint used create and refresh tokens",
            "value" : "https://oauth2.googleapis.com/token",
            "kind" : "HIDDEN",
            "protected" : false
          }, {
            "name" : "oauth_credentials.scope",
            "aliases" : [ ],
            "label" : "OAuth scopes we need to request access to",
            "value" : "profile email https://www.googleapis.com/auth/analytics.readonly",
            "kind" : "HIDDEN",
            "protected" : false
          }, {
            "name" : "oauth_credentials.access_token",
            "aliases" : [ ],
            "label" : "Access Token",
            "kind" : "HIDDEN",
            "protected" : false
          }, {
            "name" : "oauth_credentials.refresh_token",
            "aliases" : [ ],
            "label" : "OAuth Refresh Token",
            "kind" : "HIDDEN",
            "protected" : false
          }, {
            "name" : "oauth_credentials.refresh_proxy_url",
            "aliases" : [ ],
            "label" : "Optional - will be called with 'oauth_credentials.refresh_token' to refresh the access token",
            "kind" : "HIDDEN",
            "required" : "false",
            "protected" : false
          }, {
            "name" : "oauth_credentials.refresh_proxy_url_auth",
            "aliases" : [ ],
            "label" : "Optional - Sets Authorization header on 'oauth_credentials.refresh_url' request",
            "kind" : "HIDDEN",
            "required" : "false",
            "protected" : false
          }, {
            "name" : "oauth_credentials.client_id",
            "aliases" : [ ],
            "label" : "Optional - OAuth Client ID used if refresh_proxy_url not supplied",
            "kind" : "HIDDEN",
            "required" : "false",
            "protected" : false
          }, {
            "name" : "oauth_credentials.client_secret",
            "aliases" : [ ],
            "label" : "Optional - OAuth Client Secret used if refresh_proxy_url not supplied",
            "kind" : "HIDDEN",
            "required" : "false",
            "protected" : false
          }, {
            "name" : "view_id",
            "aliases" : [ ],
            "label" : "View ID",
            "placeholder" : "Ex. 198343027",
            "kind" : "STRING",
            "protected" : false
          }, {
            "name" : "reports",
            "aliases" : [ ],
            "label" : "Reports",
            "placeholder" : "Ex. my_report_definition.json",
            "kind" : "STRING",
            "required" : "false",
            "protected" : false
          }, {
            "name" : "start_date",
            "aliases" : [ ],
            "kind" : "DATE_ISO8601",
            "protected" : false
          }, {
            "name" : "end_date",
            "aliases" : [ ],
            "kind" : "DATE_ISO8601",
            "required" : "false",
            "protected" : false
          } ],
          "variants" : [ ],
          "commands" : { },
          "matatikaHidden" : false,
          "_links" : {
            "self" : {
              "href" : "https://catalog.matatika.com/api/dataplugins/e6414e68-225e-43a6-8e2e-800f3d89512e"
            },
            "update dataplugin" : {
              "href" : "https://catalog.matatika.com/api/workspaces/d542d2b9-7843-4a50-9bdb-ef445734904c/dataplugins/e6414e68-225e-43a6-8e2e-800f3d89512e",
              "type" : "PUT"
            }
          }
        }
      },
      "managed" : false,
      "_links" : {
        "self" : {
          "href" : "https://catalog.matatika.com/api/datacomponents/1a355ff6-051f-4a00-b510-2d51321147b5"
        },
        "update datacomponent" : {
          "href" : "https://catalog.matatika.com/api/datacomponents/1a355ff6-051f-4a00-b510-2d51321147b5"
        },
        "delete datacomponent" : {
          "href" : "https://catalog.matatika.com/api/datacomponents/1a355ff6-051f-4a00-b510-2d51321147b5"
        }
      }
    }, {
      "id" : "9ff03243-d68c-4d74-a0ec-2f10263d2904",
      "created" : "2022-11-04T15:19:21.212897",
      "lastModified" : "2022-11-04T15:19:21.212898",
      "name" : "Warehouse",
      "properties" : {
        "password" : "Ht56HwdV_TcriFS46_gIDE08gw",
        "dbname" : "lgxcdzf",
        "default_target_schema" : "analytics",
        "port" : "5432",
        "host" : "nutty-ugli.db.elephantsql.com",
        "user" : "lgxcdzf"
      },
      "commands" : { },
      "_embedded" : {
        "dataplugin" : {
          "id" : "88dca526-43ac-478b-a680-e3d52388944f",
          "pluginType" : "LOADER",
          "name" : "target-postgres",
          "namespace" : "postgres_transferwise",
          "variant" : "transferwise",
          "label" : "PostgreSQL",
          "description" : "PostgreSQL Data Warehouse\n\nThe World's Most Advanced Open Source Relational Database\n\n## Settings\n\n### User\nPostgres user\n\n### Password\nPostgres password\n\n### Host\nPostgres host\nExample: `test-host.db.elephantsql.com`\n\n### Port\nPostgres port\nExample: `5432`\n\n### Database Name\nPostgres database name\n\n### Default Target Schema\nName of the schema where tables will be created (no database prefix)",
          "logoUrl" : "/assets/images/datastore/target-postgres.png",
          "hidden" : false,
          "docs" : "https://hub.meltano.com/loaders/postgres.html",
          "pipUrl" : "git+https://github.com/Matatika/[email protected]",
          "repo" : "https://github.com/transferwise/pipelinewise-target-postgres",
          "executable" : "target-postgres",
          "capabilities" : [ ],
          "select" : [ ],
          "update" : { },
          "vars" : { },
          "settings" : [ {
            "name" : "user",
            "aliases" : [ "username" ],
            "label" : "User",
            "kind" : "STRING",
            "description" : "PostgreSQL user",
            "protected" : false
          }, {
            "name" : "password",
            "aliases" : [ ],
            "label" : "Password",
            "kind" : "PASSWORD",
            "description" : "PostgreSQL password",
            "protected" : false
          }, {
            "name" : "host",
            "aliases" : [ "address" ],
            "label" : "Host",
            "kind" : "STRING",
            "description" : "PostgreSQL host",
            "protected" : false
          }, {
            "name" : "port",
            "aliases" : [ ],
            "label" : "Port",
            "value" : "5432",
            "kind" : "INTEGER",
            "description" : "PostgreSQL port",
            "protected" : false
          }, {
            "name" : "dbname",
            "aliases" : [ "database" ],
            "label" : "Database Name",
            "kind" : "STRING",
            "description" : "PostgreSQL database name",
            "protected" : false
          }, {
            "name" : "default_target_schema",
            "aliases" : [ ],
            "label" : "Default Target Schema",
            "value" : "analytics",
            "kind" : "STRING",
            "description" : "Name of the schema where the tables will be created.",
            "protected" : false
          }, {
            "name" : "ssl",
            "aliases" : [ ],
            "label" : "SSL",
            "value" : "false",
            "kind" : "HIDDEN",
            "required" : "false",
            "protected" : false,
            "value_post_processor" : "STRINGIFY"
          }, {
            "name" : "batch_size_rows",
            "aliases" : [ ],
            "label" : "Batch Size Rows",
            "value" : "100000",
            "kind" : "INTEGER",
            "description" : "Maximum number of rows in each batch. At the end of each batch, the rows in the batch are loaded into Postgres.",
            "required" : "false",
            "protected" : false
          }, {
            "name" : "underscore_camel_case_fields",
            "aliases" : [ ],
            "label" : "Underscore Camel Case Fields",
            "value" : "true",
            "kind" : "HIDDEN",
            "description" : "Enable underscoring camel case field names, make the resulting postgres column names the same as from the Meltano variant of target-postgres.",
            "required" : "false",
            "protected" : false
          }, {
            "name" : "flush_all_streams",
            "aliases" : [ ],
            "label" : "Flush All Streams",
            "value" : "false",
            "kind" : "HIDDEN",
            "description" : "Flush and load every stream into Postgres when one batch is full. Warning: This may trigger the COPY command to use files with low number of records.",
            "required" : "false",
            "protected" : false
          }, {
            "name" : "parallelism",
            "aliases" : [ ],
            "label" : "Parallelism",
            "value" : "0",
            "kind" : "HIDDEN",
            "description" : "The number of threads used to flush tables. 0 will create a thread for each stream, up to parallelism_max. -1 will create a thread for each CPU core. Any other positive number will create that number of threads, up to parallelism_max.",
            "required" : "false",
            "protected" : false
          }, {
            "name" : "parallelism_max",
            "aliases" : [ ],
            "label" : "Max Parallelism",
            "value" : "16",
            "kind" : "HIDDEN",
            "description" : "Max number of parallel threads to use when flushing tables.",
            "required" : "false",
            "protected" : false
          }, {
            "name" : "default_target_schema_select_permission",
            "aliases" : [ ],
            "label" : "Default Target Schema Select Permission",
            "kind" : "HIDDEN",
            "description" : "Grant USAGE privilege on newly created schemas and grant SELECT privilege on newly created tables to a specific role or a list of roles. If `schema_mapping` is not defined then every stream sent by the tap is granted accordingly.",
            "required" : "false",
            "protected" : false
          }, {
            "name" : "schema_mapping",
            "aliases" : [ ],
            "label" : "Schema Mapping",
            "kind" : "HIDDEN",
            "description" : "Useful if you want to load multiple streams from one tap to multiple Postgres schemas.\n\nIf the tap sends the `stream_id` in `<schema_name>-<table_name>` format then this option overwrites the `default_target_schema` value.\nNote, that using `schema_mapping` you can overwrite the `default_target_schema_select_permission` value to grant SELECT permissions to different groups per schemas or optionally you can create indices automatically for the replicated tables.\n",
            "required" : "false",
            "protected" : false
          }, {
            "name" : "add_metadata_columns",
            "aliases" : [ ],
            "label" : "Add Metadata Columns",
            "value" : "true",
            "kind" : "HIDDEN",
            "description" : "Metadata columns add extra row level information about data ingestions, (i.e. when was the row read in source, when was inserted or deleted in postgres etc.) Metadata columns are creating automatically by adding extra columns to the tables with a column prefix `_SDC_`. The column names are following the stitch naming conventions documented at https://www.stitchdata.com/docs/data-structure/integration-schemas#sdc-columns. Enabling metadata columns will flag the deleted rows by setting the `_SDC_DELETED_AT` metadata column. Without the `add_metadata_columns` option the deleted rows from singer taps will not be recongisable in Postgres.",
            "required" : "false",
            "protected" : false
          }, {
            "name" : "hard_delete",
            "aliases" : [ ],
            "label" : "Hard Delete",
            "value" : "false",
            "kind" : "HIDDEN",
            "description" : "When `hard_delete` option is true then DELETE SQL commands will be performed in Postgres to delete rows in tables. It's achieved by continuously checking the `_SDC_DELETED_AT` metadata column sent by the singer tap. Due to deleting rows requires metadata columns, `hard_delete` option automatically enables the `add_metadata_columns` option as well.",
            "required" : "false",
            "protected" : false
          }, {
            "name" : "data_flattening_max_level",
            "aliases" : [ ],
            "label" : "Data Flattening Max Level",
            "value" : "10",
            "kind" : "HIDDEN",
            "description" : "Object type RECORD items from taps can be transformed to flattened columns by creating columns automatically. When value is 0 (default) then flattening functionality is turned off.",
            "required" : "false",
            "protected" : false
          }, {
            "name" : "primary_key_required",
            "aliases" : [ ],
            "label" : "Primary Key Required",
            "value" : "false",
            "kind" : "BOOLEAN",
            "description" : "Log based and Incremental replications on tables with no Primary Key cause duplicates when merging UPDATE events. When set to true, stop loading data if no Primary Key is defined.",
            "required" : "false",
            "protected" : false
          }, {
            "name" : "validate_records",
            "aliases" : [ ],
            "label" : "Validate Records",
            "value" : "false",
            "kind" : "BOOLEAN",
            "description" : "Validate every single record message to the corresponding JSON schema. This option is disabled by default and invalid RECORD messages will fail only at load time by Postgres. Enabling this option will detect invalid records earlier but could cause performance degradation.",
            "required" : "false",
            "protected" : false
          }, {
            "name" : "temp_dir",
            "aliases" : [ ],
            "label" : "Temporary Directory",
            "kind" : "HIDDEN",
            "description" : "(Default: platform-dependent) Directory of temporary CSV files with RECORD messages.",
            "required" : "false",
            "protected" : false
          } ],
          "variants" : [ ],
          "commands" : { },
          "matatikaHidden" : false,
          "_links" : {
            "self" : {
              "href" : "https://catalog.matatika.com/api/dataplugins/88dca526-43ac-478b-a680-e3d52388944f"
            },
            "update dataplugin" : {
              "href" : "https://catalog.matatika.com/api/workspaces/d542d2b9-7843-4a50-9bdb-ef445734904c/dataplugins/88dca526-43ac-478b-a680-e3d52388944f",
              "type" : "PUT"
            }
          }
        }
      },
      "managed" : true,
      "_links" : {
        "self" : {
          "href" : "https://catalog.matatika.com/api/datacomponents/9ff03243-d68c-4d74-a0ec-2f10263d2904"
        },
        "update datacomponent" : {
          "href" : "https://catalog.matatika.com/api/datacomponents/9ff03243-d68c-4d74-a0ec-2f10263d2904"
        },
        "delete datacomponent" : {
          "href" : "https://catalog.matatika.com/api/datacomponents/9ff03243-d68c-4d74-a0ec-2f10263d2904"
        }
      }
    }, {
      "id" : "ae26fbdd-6fd2-43ad-ba8f-d659e6a3e00c",
      "created" : "2022-11-04T15:19:21.258297",
      "lastModified" : "2022-11-04T15:19:21.258298",
      "name" : "dbt",
      "properties" : { },
      "commands" : {
        "compile" : {
          "args" : "compile",
          "description" : "Generates executable SQL from source model, test, and analysis files. Compiled SQL files are written to the target/ directory."
        },
        "seed" : {
          "args" : "seed",
          "description" : "Load data from csv files into your data warehouse."
        },
        "test" : {
          "args" : "test",
          "description" : "Runs tests on data in deployed models."
        },
        "deps" : {
          "args" : "deps",
          "description" : "Pull the most recent version of the dependencies listed in packages.yml"
        },
        "run" : {
          "args" : "run",
          "description" : "Compile SQL and execute against the current target database."
        },
        "clean" : {
          "args" : "clean",
          "description" : "Delete all folders in the clean-targets list (usually the dbt_modules and target directories.)"
        },
        "snapshot" : {
          "args" : "snapshot",
          "description" : "Execute snapshots defined in your project."
        }
      },
      "_embedded" : {
        "dataplugin" : {
          "id" : "5b397a9a-ed83-4c60-9bb7-6ea049099e29",
          "pluginType" : "TRANSFORMER",
          "name" : "dbt",
          "namespace" : "dbt",
          "variant" : "dbt",
          "logoUrl" : "/assets/images/transformer/dbt.png",
          "hidden" : false,
          "docs" : "https://meltano.com/docs/transforms.html",
          "pipUrl" : "dbt-core~=1.0.0 dbt-postgres~=1.0.0 dbt-redshift~=1.0.0 dbt-snowflake~=1.0.0 dbt-bigquery~=1.0.0\n",
          "repo" : "https://github.com/dbt-labs/dbt-core",
          "capabilities" : [ ],
          "select" : [ ],
          "update" : { },
          "vars" : { },
          "settings" : [ {
            "name" : "project_dir",
            "aliases" : [ ],
            "value" : "$MELTANO_PROJECT_ROOT/transform",
            "kind" : "STRING",
            "protected" : false
          }, {
            "name" : "profiles_dir",
            "aliases" : [ ],
            "value" : "$MELTANO_PROJECT_ROOT/transform/profile",
            "kind" : "STRING",
            "env" : "DBT_PROFILES_DIR",
            "protected" : false
          }, {
            "name" : "target",
            "aliases" : [ ],
            "value" : "$MELTANO_LOAD__DIALECT",
            "kind" : "STRING",
            "protected" : false
          }, {
            "name" : "source_schema",
            "aliases" : [ ],
            "value" : "$MELTANO_LOAD__TARGET_SCHEMA",
            "kind" : "STRING",
            "protected" : false
          }, {
            "name" : "target_schema",
            "aliases" : [ ],
            "value" : "analytics",
            "kind" : "STRING",
            "protected" : false
          }, {
            "name" : "models",
            "aliases" : [ ],
            "value" : "$MELTANO_TRANSFORM__PACKAGE_NAME $MELTANO_EXTRACTOR_NAMESPACE my_meltano_project",
            "kind" : "STRING",
            "protected" : false
          } ],
          "variants" : [ ],
          "commands" : {
            "compile" : {
              "args" : "compile",
              "description" : "Generates executable SQL from source model, test, and analysis files. Compiled SQL files are written to the target/ directory."
            },
            "seed" : {
              "args" : "seed",
              "description" : "Load data from csv files into your data warehouse."
            },
            "test" : {
              "args" : "test",
              "description" : "Runs tests on data in deployed models."
            },
            "deps" : {
              "args" : "deps",
              "description" : "Pull the most recent version of the dependencies listed in packages.yml"
            },
            "run" : {
              "args" : "run",
              "description" : "Compile SQL and execute against the current target database."
            },
            "clean" : {
              "args" : "clean",
              "description" : "Delete all folders in the clean-targets list (usually the dbt_modules and target directories.)"
            },
            "snapshot" : {
              "args" : "snapshot",
              "description" : "Execute snapshots defined in your project."
            }
          },
          "matatikaHidden" : false,
          "_links" : {
            "self" : {
              "href" : "https://catalog.matatika.com/api/dataplugins/5b397a9a-ed83-4c60-9bb7-6ea049099e29"
            },
            "update dataplugin" : {
              "href" : "https://catalog.matatika.com/api/workspaces/d542d2b9-7843-4a50-9bdb-ef445734904c/dataplugins/5b397a9a-ed83-4c60-9bb7-6ea049099e29",
              "type" : "PUT"
            }
          }
        }
      },
      "managed" : true,
      "_links" : {
        "self" : {
          "href" : "https://catalog.matatika.com/api/datacomponents/ae26fbdd-6fd2-43ad-ba8f-d659e6a3e00c"
        },
        "update datacomponent" : {
          "href" : "https://catalog.matatika.com/api/datacomponents/ae26fbdd-6fd2-43ad-ba8f-d659e6a3e00c"
        },
        "delete datacomponent" : {
          "href" : "https://catalog.matatika.com/api/datacomponents/ae26fbdd-6fd2-43ad-ba8f-d659e6a3e00c"
        }
      }
    } ],
    "latest job" : {
      "id" : "05654151-e938-49c5-b317-07ef153e69bb",
      "created" : "2022-11-04T15:19:51.342897",
      "type" : "WORKSPACE_CONFIG",
      "status" : "QUEUED",
      "_links" : {
        "self" : {
          "href" : "https://catalog.matatika.com/api/jobs/05654151-e938-49c5-b317-07ef153e69bb"
        },
        "delete job" : {
          "href" : "https://catalog.matatika.com/api/jobs/05654151-e938-49c5-b317-07ef153e69bb",
          "type" : "DELETE"
        },
        "logs" : {
          "href" : "https://catalog.matatika.com/api/jobs/05654151-e938-49c5-b317-07ef153e69bb/logs?sequence=0",
          "type" : "GET"
        },
        "withdraw job" : {
          "href" : "https://catalog.matatika.com/api/jobs/05654151-e938-49c5-b317-07ef153e69bb/stopped",
          "type" : "PUT"
        }
      }
    }
  },
  "_links" : {
    "update pipeline" : {
      "href" : "https://catalog.matatika.com/api/workspaces/d542d2b9-7843-4a50-9bdb-ef445734904c/pipelines/c3bb4dcd-63e7-4811-862e-99da31e488e4",
      "type" : "PUT"
    },
    "delete pipeline" : {
      "href" : "https://catalog.matatika.com/api/pipelines/c3bb4dcd-63e7-4811-862e-99da31e488e4",
      "type" : "DELETE"
    },
    "draft pipeline" : {
      "href" : "https://catalog.matatika.com/api/workspaces/d542d2b9-7843-4a50-9bdb-ef445734904c/pipelines/c3bb4dcd-63e7-4811-862e-99da31e488e4/draft",
      "type" : "PUT"
    },
    "self" : {
      "href" : "https://catalog.matatika.com/api/pipelines/c3bb4dcd-63e7-4811-862e-99da31e488e4"
    },
    "environment" : {
      "href" : "https://catalog.matatika.com/api/pipelines/c3bb4dcd-63e7-4811-862e-99da31e488e4/environment"
    },
    "jobs" : {
      "href" : "https://catalog.matatika.com/api/pipelines/c3bb4dcd-63e7-4811-862e-99da31e488e4/jobs",
      "type" : "GET"
    },
    "add subscription" : {
      "href" : "https://catalog.matatika.com/api/pipelines/c3bb4dcd-63e7-4811-862e-99da31e488e4/subscriptions"
    },
    "withdraw job" : {
      "href" : "https://catalog.matatika.com/api/jobs/05654151-e938-49c5-b317-07ef153e69bb/stopped",
      "type" : "PUT"
    },
    "latest job" : {
      "href" : "https://catalog.matatika.com/api/jobs/05654151-e938-49c5-b317-07ef153e69bb"
    }
  }
}

Properties

For each setting s in the datacomponents’ dataplugin settings for each

Path Type Description
s.name s.kind Refer to s.description
  • Any required settings not satisfied by a datacomponent property must be provided as a pipeline property
  • Any settings that are already satisfied by a datacomponent property can be overridden by a pipeline property

Formats

Pipeline Status

Value Description
READY The pipeline completed processing resource changes
PROVISIONING The pipeline is processing resource changes
FAILED The pipeline failed to process resource changes

Requests

See Also

View all pipelines in a workspace

GET

/api/workspaces/{workspace-id}/pipelines

Returns all configured pipelines in the workspace {workspace-id}.

Prerequisites

  • Workspace {workspace-id} must exist

Request

Example Snippets

  • cURL
curl -H "Authorization: Bearer $ACCESS_TOKEN" 'https://catalog.matatika.com/api/workspaces/d542d2b9-7843-4a50-9bdb-ef445734904c/pipelines' -i -X GET \
    -H 'Accept: application/json, application/javascript, text/javascript, text/json' \
    -H 'Content-Type: application/json'
  • Python (requests)
import requests

url = "https://catalog.matatika.com/api/workspaces/d542d2b9-7843-4a50-9bdb-ef445734904c/pipelines"

headers = {
  'Authorization': ACCESS_TOKEN
}

response = requests.request("GET", url, headers=headers)

print(response.text.encode('utf8'))

Response

200 OK

Pipeline collection with HAL links.

{
  "_embedded" : {
    "pipelines" : [ {
      "id" : "c3bb4dcd-63e7-4811-862e-99da31e488e4",
      "status" : "PROVISIONING",
      "name" : "SIT-generated pipeline [2022-11-04T15:19:49.097677]",
      "schedule" : "0 0 0 25 12 ?",
      "timeout" : 0,
      "created" : "2022-11-04T15:19:49.734323",
      "lastModified" : "2022-11-04T15:19:49.734323",
      "properties" : { },
      "dataComponents" : [ "tap-google-analytics", "Warehouse", "dbt" ],
      "actions" : [ ],
      "_embedded" : {
        "dataComponents" : [ {
          "id" : "1a355ff6-051f-4a00-b510-2d51321147b5",
          "created" : "2022-11-04T15:19:48.729983",
          "lastModified" : "2022-11-04T15:19:48.729983",
          "name" : "tap-google-analytics",
          "properties" : { },
          "commands" : { },
          "_embedded" : {
            "dataplugin" : {
              "id" : "e6414e68-225e-43a6-8e2e-800f3d89512e",
              "pluginType" : "EXTRACTOR",
              "name" : "tap-google-analytics",
              "namespace" : "tap_google_analytics",
              "variant" : "meltano",
              "label" : "Google Analytics",
              "description" : "App and website analytics platform hosted by Google\n\n## Prerequisites\n\nBefore importing data from Google Analytics you or your webmaster will need to have performed some initial setup to grant\nyourself access to the Google Analytics for your website.\n\n[Google Analytics Prerequisites](https://www.matatika.com/docs/data-sources/tap-google-analytics/prerequisites)\n\n## Data source settings\n### View ID\n\nOnce you have connected with your google account, the `View ID` drop down will allow you to choose which view you want to get your data from.\n\n### Start Date\n\nProviding a start date allows you to set how much historical data you wish to import into you workspace.\n\nThis date has to be in the ISO-8601 date format, see the example below.\n\nFormat: `YYYY-MM-DDTHH:MM:SSZ`\n\nExample: `2020-10-01T13:12:11Z`\n\n### End Date\n\nProviding an end date allows you to limit how much data you wish to import into you workspace.\n\nThis date has to be in the ISO-8601 date format, see the example below.\n\nFormat: `YYYY-MM-DDTHH:MM:SSZ`\n\nExample: `2020-10-01T13:12:11Z`\n\n### Reports\n\nProviding the path to a reports json file allows you to supply the tap with a user defined definition of tables and information you want to sync from Google Analytics.\nThis does require you to commit a file to your workspace repository so that Google Analytics can find it during run time, and may also break the dbt transforms that are required to have our included datasets work.\nFor more information on how to define a custom reports file see the [Meltano Hub Documentation](https://hub.meltano.com/extractors/google-analytics#reports).\n\n## Learn more\n\n[Matatika Google Analytics Data Source Documentation](https://www.matatika.com/docs/data-sources/tap-google-analytics)\n\n[Meltano Hub Google Analytics Documentation](https://hub.meltano.com/extractors/google-analytics)",
              "logoUrl" : "/assets/images/datasource/tap-google-analytics.svg",
              "hidden" : false,
              "docs" : "https://www.matatika.com/docs/data-sources/tap-google-analytics/",
              "pipUrl" : "git+https://github.com/Matatika/[email protected]",
              "repo" : "https://github.com/Matatika/tap-google-analytics",
              "capabilities" : [ "CATALOG", "DISCOVER", "STATE" ],
              "select" : [ ],
              "update" : { },
              "vars" : { },
              "settings" : [ {
                "name" : "oauth_credentials.authorization_url",
                "aliases" : [ ],
                "label" : "OAuth identity provider authorization endpoint used create and refresh tokens",
                "value" : "https://oauth2.googleapis.com/token",
                "kind" : "HIDDEN",
                "protected" : false
              }, {
                "name" : "oauth_credentials.scope",
                "aliases" : [ ],
                "label" : "OAuth scopes we need to request access to",
                "value" : "profile email https://www.googleapis.com/auth/analytics.readonly",
                "kind" : "HIDDEN",
                "protected" : false
              }, {
                "name" : "oauth_credentials.access_token",
                "aliases" : [ ],
                "label" : "Access Token",
                "kind" : "HIDDEN",
                "protected" : false
              }, {
                "name" : "oauth_credentials.refresh_token",
                "aliases" : [ ],
                "label" : "OAuth Refresh Token",
                "kind" : "HIDDEN",
                "protected" : false
              }, {
                "name" : "oauth_credentials.refresh_proxy_url",
                "aliases" : [ ],
                "label" : "Optional - will be called with 'oauth_credentials.refresh_token' to refresh the access token",
                "kind" : "HIDDEN",
                "required" : "false",
                "protected" : false
              }, {
                "name" : "oauth_credentials.refresh_proxy_url_auth",
                "aliases" : [ ],
                "label" : "Optional - Sets Authorization header on 'oauth_credentials.refresh_url' request",
                "kind" : "HIDDEN",
                "required" : "false",
                "protected" : false
              }, {
                "name" : "oauth_credentials.client_id",
                "aliases" : [ ],
                "label" : "Optional - OAuth Client ID used if refresh_proxy_url not supplied",
                "kind" : "HIDDEN",
                "required" : "false",
                "protected" : false
              }, {
                "name" : "oauth_credentials.client_secret",
                "aliases" : [ ],
                "label" : "Optional - OAuth Client Secret used if refresh_proxy_url not supplied",
                "kind" : "HIDDEN",
                "required" : "false",
                "protected" : false
              }, {
                "name" : "view_id",
                "aliases" : [ ],
                "label" : "View ID",
                "placeholder" : "Ex. 198343027",
                "kind" : "STRING",
                "protected" : false
              }, {
                "name" : "reports",
                "aliases" : [ ],
                "label" : "Reports",
                "placeholder" : "Ex. my_report_definition.json",
                "kind" : "STRING",
                "required" : "false",
                "protected" : false
              }, {
                "name" : "start_date",
                "aliases" : [ ],
                "kind" : "DATE_ISO8601",
                "protected" : false
              }, {
                "name" : "end_date",
                "aliases" : [ ],
                "kind" : "DATE_ISO8601",
                "required" : "false",
                "protected" : false
              } ],
              "variants" : [ ],
              "commands" : { },
              "matatikaHidden" : false,
              "_links" : {
                "self" : {
                  "href" : "https://catalog.matatika.com/api/dataplugins/e6414e68-225e-43a6-8e2e-800f3d89512e"
                },
                "update dataplugin" : {
                  "href" : "https://catalog.matatika.com/api/workspaces/d542d2b9-7843-4a50-9bdb-ef445734904c/dataplugins/e6414e68-225e-43a6-8e2e-800f3d89512e",
                  "type" : "PUT"
                }
              }
            }
          },
          "managed" : false,
          "_links" : {
            "self" : {
              "href" : "https://catalog.matatika.com/api/datacomponents/1a355ff6-051f-4a00-b510-2d51321147b5"
            },
            "update datacomponent" : {
              "href" : "https://catalog.matatika.com/api/datacomponents/1a355ff6-051f-4a00-b510-2d51321147b5"
            },
            "delete datacomponent" : {
              "href" : "https://catalog.matatika.com/api/datacomponents/1a355ff6-051f-4a00-b510-2d51321147b5"
            }
          }
        }, {
          "id" : "9ff03243-d68c-4d74-a0ec-2f10263d2904",
          "created" : "2022-11-04T15:19:21.212897",
          "lastModified" : "2022-11-04T15:19:21.212898",
          "name" : "Warehouse",
          "properties" : {
            "password" : "Ht56HwdV_TcriFS46_gIDE08gw",
            "dbname" : "lgxcdzf",
            "default_target_schema" : "analytics",
            "port" : "5432",
            "host" : "nutty-ugli.db.elephantsql.com",
            "user" : "lgxcdzf"
          },
          "commands" : { },
          "_embedded" : {
            "dataplugin" : {
              "id" : "88dca526-43ac-478b-a680-e3d52388944f",
              "pluginType" : "LOADER",
              "name" : "target-postgres",
              "namespace" : "postgres_transferwise",
              "variant" : "transferwise",
              "label" : "PostgreSQL",
              "description" : "PostgreSQL Data Warehouse\n\nThe World's Most Advanced Open Source Relational Database\n\n## Settings\n\n### User\nPostgres user\n\n### Password\nPostgres password\n\n### Host\nPostgres host\nExample: `test-host.db.elephantsql.com`\n\n### Port\nPostgres port\nExample: `5432`\n\n### Database Name\nPostgres database name\n\n### Default Target Schema\nName of the schema where tables will be created (no database prefix)",
              "logoUrl" : "/assets/images/datastore/target-postgres.png",
              "hidden" : false,
              "docs" : "https://hub.meltano.com/loaders/postgres.html",
              "pipUrl" : "git+https://github.com/Matatika/[email protected]",
              "repo" : "https://github.com/transferwise/pipelinewise-target-postgres",
              "executable" : "target-postgres",
              "capabilities" : [ ],
              "select" : [ ],
              "update" : { },
              "vars" : { },
              "settings" : [ {
                "name" : "user",
                "aliases" : [ "username" ],
                "label" : "User",
                "kind" : "STRING",
                "description" : "PostgreSQL user",
                "protected" : false
              }, {
                "name" : "password",
                "aliases" : [ ],
                "label" : "Password",
                "kind" : "PASSWORD",
                "description" : "PostgreSQL password",
                "protected" : false
              }, {
                "name" : "host",
                "aliases" : [ "address" ],
                "label" : "Host",
                "kind" : "STRING",
                "description" : "PostgreSQL host",
                "protected" : false
              }, {
                "name" : "port",
                "aliases" : [ ],
                "label" : "Port",
                "value" : "5432",
                "kind" : "INTEGER",
                "description" : "PostgreSQL port",
                "protected" : false
              }, {
                "name" : "dbname",
                "aliases" : [ "database" ],
                "label" : "Database Name",
                "kind" : "STRING",
                "description" : "PostgreSQL database name",
                "protected" : false
              }, {
                "name" : "default_target_schema",
                "aliases" : [ ],
                "label" : "Default Target Schema",
                "value" : "analytics",
                "kind" : "STRING",
                "description" : "Name of the schema where the tables will be created.",
                "protected" : false
              }, {
                "name" : "ssl",
                "aliases" : [ ],
                "label" : "SSL",
                "value" : "false",
                "kind" : "HIDDEN",
                "required" : "false",
                "protected" : false,
                "value_post_processor" : "STRINGIFY"
              }, {
                "name" : "batch_size_rows",
                "aliases" : [ ],
                "label" : "Batch Size Rows",
                "value" : "100000",
                "kind" : "INTEGER",
                "description" : "Maximum number of rows in each batch. At the end of each batch, the rows in the batch are loaded into Postgres.",
                "required" : "false",
                "protected" : false
              }, {
                "name" : "underscore_camel_case_fields",
                "aliases" : [ ],
                "label" : "Underscore Camel Case Fields",
                "value" : "true",
                "kind" : "HIDDEN",
                "description" : "Enable underscoring camel case field names, make the resulting postgres column names the same as from the Meltano variant of target-postgres.",
                "required" : "false",
                "protected" : false
              }, {
                "name" : "flush_all_streams",
                "aliases" : [ ],
                "label" : "Flush All Streams",
                "value" : "false",
                "kind" : "HIDDEN",
                "description" : "Flush and load every stream into Postgres when one batch is full. Warning: This may trigger the COPY command to use files with low number of records.",
                "required" : "false",
                "protected" : false
              }, {
                "name" : "parallelism",
                "aliases" : [ ],
                "label" : "Parallelism",
                "value" : "0",
                "kind" : "HIDDEN",
                "description" : "The number of threads used to flush tables. 0 will create a thread for each stream, up to parallelism_max. -1 will create a thread for each CPU core. Any other positive number will create that number of threads, up to parallelism_max.",
                "required" : "false",
                "protected" : false
              }, {
                "name" : "parallelism_max",
                "aliases" : [ ],
                "label" : "Max Parallelism",
                "value" : "16",
                "kind" : "HIDDEN",
                "description" : "Max number of parallel threads to use when flushing tables.",
                "required" : "false",
                "protected" : false
              }, {
                "name" : "default_target_schema_select_permission",
                "aliases" : [ ],
                "label" : "Default Target Schema Select Permission",
                "kind" : "HIDDEN",
                "description" : "Grant USAGE privilege on newly created schemas and grant SELECT privilege on newly created tables to a specific role or a list of roles. If `schema_mapping` is not defined then every stream sent by the tap is granted accordingly.",
                "required" : "false",
                "protected" : false
              }, {
                "name" : "schema_mapping",
                "aliases" : [ ],
                "label" : "Schema Mapping",
                "kind" : "HIDDEN",
                "description" : "Useful if you want to load multiple streams from one tap to multiple Postgres schemas.\n\nIf the tap sends the `stream_id` in `<schema_name>-<table_name>` format then this option overwrites the `default_target_schema` value.\nNote, that using `schema_mapping` you can overwrite the `default_target_schema_select_permission` value to grant SELECT permissions to different groups per schemas or optionally you can create indices automatically for the replicated tables.\n",
                "required" : "false",
                "protected" : false
              }, {
                "name" : "add_metadata_columns",
                "aliases" : [ ],
                "label" : "Add Metadata Columns",
                "value" : "true",
                "kind" : "HIDDEN",
                "description" : "Metadata columns add extra row level information about data ingestions, (i.e. when was the row read in source, when was inserted or deleted in postgres etc.) Metadata columns are creating automatically by adding extra columns to the tables with a column prefix `_SDC_`. The column names are following the stitch naming conventions documented at https://www.stitchdata.com/docs/data-structure/integration-schemas#sdc-columns. Enabling metadata columns will flag the deleted rows by setting the `_SDC_DELETED_AT` metadata column. Without the `add_metadata_columns` option the deleted rows from singer taps will not be recongisable in Postgres.",
                "required" : "false",
                "protected" : false
              }, {
                "name" : "hard_delete",
                "aliases" : [ ],
                "label" : "Hard Delete",
                "value" : "false",
                "kind" : "HIDDEN",
                "description" : "When `hard_delete` option is true then DELETE SQL commands will be performed in Postgres to delete rows in tables. It's achieved by continuously checking the `_SDC_DELETED_AT` metadata column sent by the singer tap. Due to deleting rows requires metadata columns, `hard_delete` option automatically enables the `add_metadata_columns` option as well.",
                "required" : "false",
                "protected" : false
              }, {
                "name" : "data_flattening_max_level",
                "aliases" : [ ],
                "label" : "Data Flattening Max Level",
                "value" : "10",
                "kind" : "HIDDEN",
                "description" : "Object type RECORD items from taps can be transformed to flattened columns by creating columns automatically. When value is 0 (default) then flattening functionality is turned off.",
                "required" : "false",
                "protected" : false
              }, {
                "name" : "primary_key_required",
                "aliases" : [ ],
                "label" : "Primary Key Required",
                "value" : "false",
                "kind" : "BOOLEAN",
                "description" : "Log based and Incremental replications on tables with no Primary Key cause duplicates when merging UPDATE events. When set to true, stop loading data if no Primary Key is defined.",
                "required" : "false",
                "protected" : false
              }, {
                "name" : "validate_records",
                "aliases" : [ ],
                "label" : "Validate Records",
                "value" : "false",
                "kind" : "BOOLEAN",
                "description" : "Validate every single record message to the corresponding JSON schema. This option is disabled by default and invalid RECORD messages will fail only at load time by Postgres. Enabling this option will detect invalid records earlier but could cause performance degradation.",
                "required" : "false",
                "protected" : false
              }, {
                "name" : "temp_dir",
                "aliases" : [ ],
                "label" : "Temporary Directory",
                "kind" : "HIDDEN",
                "description" : "(Default: platform-dependent) Directory of temporary CSV files with RECORD messages.",
                "required" : "false",
                "protected" : false
              } ],
              "variants" : [ ],
              "commands" : { },
              "matatikaHidden" : false,
              "_links" : {
                "self" : {
                  "href" : "https://catalog.matatika.com/api/dataplugins/88dca526-43ac-478b-a680-e3d52388944f"
                },
                "update dataplugin" : {
                  "href" : "https://catalog.matatika.com/api/workspaces/d542d2b9-7843-4a50-9bdb-ef445734904c/dataplugins/88dca526-43ac-478b-a680-e3d52388944f",
                  "type" : "PUT"
                }
              }
            }
          },
          "managed" : true,
          "_links" : {
            "self" : {
              "href" : "https://catalog.matatika.com/api/datacomponents/9ff03243-d68c-4d74-a0ec-2f10263d2904"
            },
            "update datacomponent" : {
              "href" : "https://catalog.matatika.com/api/datacomponents/9ff03243-d68c-4d74-a0ec-2f10263d2904"
            },
            "delete datacomponent" : {
              "href" : "https://catalog.matatika.com/api/datacomponents/9ff03243-d68c-4d74-a0ec-2f10263d2904"
            }
          }
        }, {
          "id" : "ae26fbdd-6fd2-43ad-ba8f-d659e6a3e00c",
          "created" : "2022-11-04T15:19:21.258297",
          "lastModified" : "2022-11-04T15:19:21.258298",
          "name" : "dbt",
          "properties" : { },
          "commands" : {
            "compile" : {
              "args" : "compile",
              "description" : "Generates executable SQL from source model, test, and analysis files. Compiled SQL files are written to the target/ directory."
            },
            "seed" : {
              "args" : "seed",
              "description" : "Load data from csv files into your data warehouse."
            },
            "test" : {
              "args" : "test",
              "description" : "Runs tests on data in deployed models."
            },
            "deps" : {
              "args" : "deps",
              "description" : "Pull the most recent version of the dependencies listed in packages.yml"
            },
            "run" : {
              "args" : "run",
              "description" : "Compile SQL and execute against the current target database."
            },
            "clean" : {
              "args" : "clean",
              "description" : "Delete all folders in the clean-targets list (usually the dbt_modules and target directories.)"
            },
            "snapshot" : {
              "args" : "snapshot",
              "description" : "Execute snapshots defined in your project."
            }
          },
          "_embedded" : {
            "dataplugin" : {
              "id" : "5b397a9a-ed83-4c60-9bb7-6ea049099e29",
              "pluginType" : "TRANSFORMER",
              "name" : "dbt",
              "namespace" : "dbt",
              "variant" : "dbt",
              "logoUrl" : "/assets/images/transformer/dbt.png",
              "hidden" : false,
              "docs" : "https://meltano.com/docs/transforms.html",
              "pipUrl" : "dbt-core~=1.0.0 dbt-postgres~=1.0.0 dbt-redshift~=1.0.0 dbt-snowflake~=1.0.0 dbt-bigquery~=1.0.0\n",
              "repo" : "https://github.com/dbt-labs/dbt-core",
              "capabilities" : [ ],
              "select" : [ ],
              "update" : { },
              "vars" : { },
              "settings" : [ {
                "name" : "project_dir",
                "aliases" : [ ],
                "value" : "$MELTANO_PROJECT_ROOT/transform",
                "kind" : "STRING",
                "protected" : false
              }, {
                "name" : "profiles_dir",
                "aliases" : [ ],
                "value" : "$MELTANO_PROJECT_ROOT/transform/profile",
                "kind" : "STRING",
                "env" : "DBT_PROFILES_DIR",
                "protected" : false
              }, {
                "name" : "target",
                "aliases" : [ ],
                "value" : "$MELTANO_LOAD__DIALECT",
                "kind" : "STRING",
                "protected" : false
              }, {
                "name" : "source_schema",
                "aliases" : [ ],
                "value" : "$MELTANO_LOAD__TARGET_SCHEMA",
                "kind" : "STRING",
                "protected" : false
              }, {
                "name" : "target_schema",
                "aliases" : [ ],
                "value" : "analytics",
                "kind" : "STRING",
                "protected" : false
              }, {
                "name" : "models",
                "aliases" : [ ],
                "value" : "$MELTANO_TRANSFORM__PACKAGE_NAME $MELTANO_EXTRACTOR_NAMESPACE my_meltano_project",
                "kind" : "STRING",
                "protected" : false
              } ],
              "variants" : [ ],
              "commands" : {
                "compile" : {
                  "args" : "compile",
                  "description" : "Generates executable SQL from source model, test, and analysis files. Compiled SQL files are written to the target/ directory."
                },
                "seed" : {
                  "args" : "seed",
                  "description" : "Load data from csv files into your data warehouse."
                },
                "test" : {
                  "args" : "test",
                  "description" : "Runs tests on data in deployed models."
                },
                "deps" : {
                  "args" : "deps",
                  "description" : "Pull the most recent version of the dependencies listed in packages.yml"
                },
                "run" : {
                  "args" : "run",
                  "description" : "Compile SQL and execute against the current target database."
                },
                "clean" : {
                  "args" : "clean",
                  "description" : "Delete all folders in the clean-targets list (usually the dbt_modules and target directories.)"
                },
                "snapshot" : {
                  "args" : "snapshot",
                  "description" : "Execute snapshots defined in your project."
                }
              },
              "matatikaHidden" : false,
              "_links" : {
                "self" : {
                  "href" : "https://catalog.matatika.com/api/dataplugins/5b397a9a-ed83-4c60-9bb7-6ea049099e29"
                },
                "update dataplugin" : {
                  "href" : "https://catalog.matatika.com/api/workspaces/d542d2b9-7843-4a50-9bdb-ef445734904c/dataplugins/5b397a9a-ed83-4c60-9bb7-6ea049099e29",
                  "type" : "PUT"
                }
              }
            }
          },
          "managed" : true,
          "_links" : {
            "self" : {
              "href" : "https://catalog.matatika.com/api/datacomponents/ae26fbdd-6fd2-43ad-ba8f-d659e6a3e00c"
            },
            "update datacomponent" : {
              "href" : "https://catalog.matatika.com/api/datacomponents/ae26fbdd-6fd2-43ad-ba8f-d659e6a3e00c"
            },
            "delete datacomponent" : {
              "href" : "https://catalog.matatika.com/api/datacomponents/ae26fbdd-6fd2-43ad-ba8f-d659e6a3e00c"
            }
          }
        } ],
        "latest job" : {
          "id" : "05654151-e938-49c5-b317-07ef153e69bb",
          "created" : "2022-11-04T15:19:51.342897",
          "type" : "WORKSPACE_CONFIG",
          "status" : "QUEUED"
        }
      },
      "_links" : {
        "update pipeline" : {
          "href" : "https://catalog.matatika.com/api/workspaces/d542d2b9-7843-4a50-9bdb-ef445734904c/pipelines/c3bb4dcd-63e7-4811-862e-99da31e488e4",
          "type" : "PUT"
        },
        "delete pipeline" : {
          "href" : "https://catalog.matatika.com/api/pipelines/c3bb4dcd-63e7-4811-862e-99da31e488e4",
          "type" : "DELETE"
        },
        "draft pipeline" : {
          "href" : "https://catalog.matatika.com/api/workspaces/d542d2b9-7843-4a50-9bdb-ef445734904c/pipelines/c3bb4dcd-63e7-4811-862e-99da31e488e4/draft",
          "type" : "PUT"
        },
        "self" : {
          "href" : "https://catalog.matatika.com/api/pipelines/c3bb4dcd-63e7-4811-862e-99da31e488e4"
        },
        "environment" : {
          "href" : "https://catalog.matatika.com/api/pipelines/c3bb4dcd-63e7-4811-862e-99da31e488e4/environment"
        },
        "jobs" : {
          "href" : "https://catalog.matatika.com/api/pipelines/c3bb4dcd-63e7-4811-862e-99da31e488e4/jobs",
          "type" : "GET"
        },
        "add subscription" : {
          "href" : "https://catalog.matatika.com/api/pipelines/c3bb4dcd-63e7-4811-862e-99da31e488e4/subscriptions"
        },
        "withdraw job" : {
          "href" : "https://catalog.matatika.com/api/jobs/05654151-e938-49c5-b317-07ef153e69bb/stopped",
          "type" : "PUT"
        },
        "latest job" : {
          "href" : "https://catalog.matatika.com/api/jobs/05654151-e938-49c5-b317-07ef153e69bb"
        }
      }
    } ]
  },
  "_links" : {
    "self" : {
      "href" : "https://catalog.matatika.com/api/workspaces/d542d2b9-7843-4a50-9bdb-ef445734904c/pipelines?page=0&size=20&sort=name,asc"
    }
  },
  "page" : {
    "size" : 20,
    "totalElements" : 1,
    "totalPages" : 1,
    "number" : 0
  }
}

View a pipeline

GET

/api/pipelines/{pipeline-id}

Returns the pipeline {pipeline-id}.

Prerequisites

  • Pipeline {pipeline-id} must exist

Request

Example Snippets

  • cURL
curl -H "Authorization: Bearer $ACCESS_TOKEN" 'https://catalog.matatika.com/api/pipelines/c3bb4dcd-63e7-4811-862e-99da31e488e4' -i -X GET \
    -H 'Accept: application/json, application/javascript, text/javascript, text/json' \
    -H 'Content-Type: application/json'
  • Python (requests)
import requests

url = "https://catalog.matatika.com/api/pipelines/c3bb4dcd-63e7-4811-862e-99da31e488e4"

headers = {
  'Authorization': ACCESS_TOKEN
}

response = requests.request("GET", url, headers=headers)

print(response.text.encode('utf8'))

Response

200 OK

Pipeline with HAL links.

{
  "id" : "c3bb4dcd-63e7-4811-862e-99da31e488e4",
  "status" : "PROVISIONING",
  "name" : "SIT-generated pipeline [2022-11-04T15:19:49.097677]",
  "schedule" : "0 0 0 25 12 ?",
  "timeout" : 0,
  "created" : "2022-11-04T15:19:49.734323",
  "lastModified" : "2022-11-04T15:19:49.734323",
  "properties" : {
    "tap-google-analytics.view_id" : "1234567890",
    "tap-google-analytics.oauth_credentials.client_secret" : "client_secret",
    "tap-google-analytics.reports" : "reports",
    "tap-google-analytics.oauth_credentials.refresh_token" : "refresh_token",
    "tap-google-analytics.start_date" : "2022-10-04T15:19:49.462298Z",
    "tap-google-analytics.end_date" : "2022-11-04T15:19:49.462404Z",
    "tap-google-analytics.oauth_credentials.client_id" : "client_id",
    "tap-google-analytics.oauth_credentials.access_token" : "access_token"
  },
  "dataComponents" : [ "tap-google-analytics", "Warehouse", "dbt" ],
  "actions" : [ ],
  "_embedded" : {
    "dataComponents" : [ {
      "id" : "1a355ff6-051f-4a00-b510-2d51321147b5",
      "created" : "2022-11-04T15:19:48.729983",
      "lastModified" : "2022-11-04T15:19:48.729983",
      "name" : "tap-google-analytics",
      "properties" : { },
      "commands" : { },
      "_embedded" : {
        "dataplugin" : {
          "id" : "e6414e68-225e-43a6-8e2e-800f3d89512e",
          "pluginType" : "EXTRACTOR",
          "name" : "tap-google-analytics",
          "namespace" : "tap_google_analytics",
          "variant" : "meltano",
          "label" : "Google Analytics",
          "description" : "App and website analytics platform hosted by Google\n\n## Prerequisites\n\nBefore importing data from Google Analytics you or your webmaster will need to have performed some initial setup to grant\nyourself access to the Google Analytics for your website.\n\n[Google Analytics Prerequisites](https://www.matatika.com/docs/data-sources/tap-google-analytics/prerequisites)\n\n## Data source settings\n### View ID\n\nOnce you have connected with your google account, the `View ID` drop down will allow you to choose which view you want to get your data from.\n\n### Start Date\n\nProviding a start date allows you to set how much historical data you wish to import into you workspace.\n\nThis date has to be in the ISO-8601 date format, see the example below.\n\nFormat: `YYYY-MM-DDTHH:MM:SSZ`\n\nExample: `2020-10-01T13:12:11Z`\n\n### End Date\n\nProviding an end date allows you to limit how much data you wish to import into you workspace.\n\nThis date has to be in the ISO-8601 date format, see the example below.\n\nFormat: `YYYY-MM-DDTHH:MM:SSZ`\n\nExample: `2020-10-01T13:12:11Z`\n\n### Reports\n\nProviding the path to a reports json file allows you to supply the tap with a user defined definition of tables and information you want to sync from Google Analytics.\nThis does require you to commit a file to your workspace repository so that Google Analytics can find it during run time, and may also break the dbt transforms that are required to have our included datasets work.\nFor more information on how to define a custom reports file see the [Meltano Hub Documentation](https://hub.meltano.com/extractors/google-analytics#reports).\n\n## Learn more\n\n[Matatika Google Analytics Data Source Documentation](https://www.matatika.com/docs/data-sources/tap-google-analytics)\n\n[Meltano Hub Google Analytics Documentation](https://hub.meltano.com/extractors/google-analytics)",
          "logoUrl" : "/assets/images/datasource/tap-google-analytics.svg",
          "hidden" : false,
          "docs" : "https://www.matatika.com/docs/data-sources/tap-google-analytics/",
          "pipUrl" : "git+https://github.com/Matatika/[email protected]",
          "repo" : "https://github.com/Matatika/tap-google-analytics",
          "capabilities" : [ "CATALOG", "DISCOVER", "STATE" ],
          "select" : [ ],
          "update" : { },
          "vars" : { },
          "settings" : [ {
            "name" : "oauth_credentials.authorization_url",
            "aliases" : [ ],
            "label" : "OAuth identity provider authorization endpoint used create and refresh tokens",
            "value" : "https://oauth2.googleapis.com/token",
            "kind" : "HIDDEN",
            "protected" : false
          }, {
            "name" : "oauth_credentials.scope",
            "aliases" : [ ],
            "label" : "OAuth scopes we need to request access to",
            "value" : "profile email https://www.googleapis.com/auth/analytics.readonly",
            "kind" : "HIDDEN",
            "protected" : false
          }, {
            "name" : "oauth_credentials.access_token",
            "aliases" : [ ],
            "label" : "Access Token",
            "kind" : "HIDDEN",
            "protected" : false
          }, {
            "name" : "oauth_credentials.refresh_token",
            "aliases" : [ ],
            "label" : "OAuth Refresh Token",
            "kind" : "HIDDEN",
            "protected" : false
          }, {
            "name" : "oauth_credentials.refresh_proxy_url",
            "aliases" : [ ],
            "label" : "Optional - will be called with 'oauth_credentials.refresh_token' to refresh the access token",
            "kind" : "HIDDEN",
            "required" : "false",
            "protected" : false
          }, {
            "name" : "oauth_credentials.refresh_proxy_url_auth",
            "aliases" : [ ],
            "label" : "Optional - Sets Authorization header on 'oauth_credentials.refresh_url' request",
            "kind" : "HIDDEN",
            "required" : "false",
            "protected" : false
          }, {
            "name" : "oauth_credentials.client_id",
            "aliases" : [ ],
            "label" : "Optional - OAuth Client ID used if refresh_proxy_url not supplied",
            "kind" : "HIDDEN",
            "required" : "false",
            "protected" : false
          }, {
            "name" : "oauth_credentials.client_secret",
            "aliases" : [ ],
            "label" : "Optional - OAuth Client Secret used if refresh_proxy_url not supplied",
            "kind" : "HIDDEN",
            "required" : "false",
            "protected" : false
          }, {
            "name" : "view_id",
            "aliases" : [ ],
            "label" : "View ID",
            "placeholder" : "Ex. 198343027",
            "kind" : "STRING",
            "protected" : false
          }, {
            "name" : "reports",
            "aliases" : [ ],
            "label" : "Reports",
            "placeholder" : "Ex. my_report_definition.json",
            "kind" : "STRING",
            "required" : "false",
            "protected" : false
          }, {
            "name" : "start_date",
            "aliases" : [ ],
            "kind" : "DATE_ISO8601",
            "protected" : false
          }, {
            "name" : "end_date",
            "aliases" : [ ],
            "kind" : "DATE_ISO8601",
            "required" : "false",
            "protected" : false
          } ],
          "variants" : [ ],
          "commands" : { },
          "matatikaHidden" : false,
          "_links" : {
            "self" : {
              "href" : "https://catalog.matatika.com/api/dataplugins/e6414e68-225e-43a6-8e2e-800f3d89512e"
            },
            "update dataplugin" : {
              "href" : "https://catalog.matatika.com/api/workspaces/d542d2b9-7843-4a50-9bdb-ef445734904c/dataplugins/e6414e68-225e-43a6-8e2e-800f3d89512e",
              "type" : "PUT"
            }
          }
        }
      },
      "managed" : false,
      "_links" : {
        "self" : {
          "href" : "https://catalog.matatika.com/api/datacomponents/1a355ff6-051f-4a00-b510-2d51321147b5"
        },
        "update datacomponent" : {
          "href" : "https://catalog.matatika.com/api/datacomponents/1a355ff6-051f-4a00-b510-2d51321147b5"
        },
        "delete datacomponent" : {
          "href" : "https://catalog.matatika.com/api/datacomponents/1a355ff6-051f-4a00-b510-2d51321147b5"
        }
      }
    }, {
      "id" : "9ff03243-d68c-4d74-a0ec-2f10263d2904",
      "created" : "2022-11-04T15:19:21.212897",
      "lastModified" : "2022-11-04T15:19:21.212898",
      "name" : "Warehouse",
      "properties" : {
        "password" : "Ht56HwdV_TcriFS46_gIDE08gw",
        "dbname" : "lgxcdzf",
        "default_target_schema" : "analytics",
        "port" : "5432",
        "host" : "nutty-ugli.db.elephantsql.com",
        "user" : "lgxcdzf"
      },
      "commands" : { },
      "_embedded" : {
        "dataplugin" : {
          "id" : "88dca526-43ac-478b-a680-e3d52388944f",
          "pluginType" : "LOADER",
          "name" : "target-postgres",
          "namespace" : "postgres_transferwise",
          "variant" : "transferwise",
          "label" : "PostgreSQL",
          "description" : "PostgreSQL Data Warehouse\n\nThe World's Most Advanced Open Source Relational Database\n\n## Settings\n\n### User\nPostgres user\n\n### Password\nPostgres password\n\n### Host\nPostgres host\nExample: `test-host.db.elephantsql.com`\n\n### Port\nPostgres port\nExample: `5432`\n\n### Database Name\nPostgres database name\n\n### Default Target Schema\nName of the schema where tables will be created (no database prefix)",
          "logoUrl" : "/assets/images/datastore/target-postgres.png",
          "hidden" : false,
          "docs" : "https://hub.meltano.com/loaders/postgres.html",
          "pipUrl" : "git+https://github.com/Matatika/[email protected]",
          "repo" : "https://github.com/transferwise/pipelinewise-target-postgres",
          "executable" : "target-postgres",
          "capabilities" : [ ],
          "select" : [ ],
          "update" : { },
          "vars" : { },
          "settings" : [ {
            "name" : "user",
            "aliases" : [ "username" ],
            "label" : "User",
            "kind" : "STRING",
            "description" : "PostgreSQL user",
            "protected" : false
          }, {
            "name" : "password",
            "aliases" : [ ],
            "label" : "Password",
            "kind" : "PASSWORD",
            "description" : "PostgreSQL password",
            "protected" : false
          }, {
            "name" : "host",
            "aliases" : [ "address" ],
            "label" : "Host",
            "kind" : "STRING",
            "description" : "PostgreSQL host",
            "protected" : false
          }, {
            "name" : "port",
            "aliases" : [ ],
            "label" : "Port",
            "value" : "5432",
            "kind" : "INTEGER",
            "description" : "PostgreSQL port",
            "protected" : false
          }, {
            "name" : "dbname",
            "aliases" : [ "database" ],
            "label" : "Database Name",
            "kind" : "STRING",
            "description" : "PostgreSQL database name",
            "protected" : false
          }, {
            "name" : "default_target_schema",
            "aliases" : [ ],
            "label" : "Default Target Schema",
            "value" : "analytics",
            "kind" : "STRING",
            "description" : "Name of the schema where the tables will be created.",
            "protected" : false
          }, {
            "name" : "ssl",
            "aliases" : [ ],
            "label" : "SSL",
            "value" : "false",
            "kind" : "HIDDEN",
            "required" : "false",
            "protected" : false,
            "value_post_processor" : "STRINGIFY"
          }, {
            "name" : "batch_size_rows",
            "aliases" : [ ],
            "label" : "Batch Size Rows",
            "value" : "100000",
            "kind" : "INTEGER",
            "description" : "Maximum number of rows in each batch. At the end of each batch, the rows in the batch are loaded into Postgres.",
            "required" : "false",
            "protected" : false
          }, {
            "name" : "underscore_camel_case_fields",
            "aliases" : [ ],
            "label" : "Underscore Camel Case Fields",
            "value" : "true",
            "kind" : "HIDDEN",
            "description" : "Enable underscoring camel case field names, make the resulting postgres column names the same as from the Meltano variant of target-postgres.",
            "required" : "false",
            "protected" : false
          }, {
            "name" : "flush_all_streams",
            "aliases" : [ ],
            "label" : "Flush All Streams",
            "value" : "false",
            "kind" : "HIDDEN",
            "description" : "Flush and load every stream into Postgres when one batch is full. Warning: This may trigger the COPY command to use files with low number of records.",
            "required" : "false",
            "protected" : false
          }, {
            "name" : "parallelism",
            "aliases" : [ ],
            "label" : "Parallelism",
            "value" : "0",
            "kind" : "HIDDEN",
            "description" : "The number of threads used to flush tables. 0 will create a thread for each stream, up to parallelism_max. -1 will create a thread for each CPU core. Any other positive number will create that number of threads, up to parallelism_max.",
            "required" : "false",
            "protected" : false
          }, {
            "name" : "parallelism_max",
            "aliases" : [ ],
            "label" : "Max Parallelism",
            "value" : "16",
            "kind" : "HIDDEN",
            "description" : "Max number of parallel threads to use when flushing tables.",
            "required" : "false",
            "protected" : false
          }, {
            "name" : "default_target_schema_select_permission",
            "aliases" : [ ],
            "label" : "Default Target Schema Select Permission",
            "kind" : "HIDDEN",
            "description" : "Grant USAGE privilege on newly created schemas and grant SELECT privilege on newly created tables to a specific role or a list of roles. If `schema_mapping` is not defined then every stream sent by the tap is granted accordingly.",
            "required" : "false",
            "protected" : false
          }, {
            "name" : "schema_mapping",
            "aliases" : [ ],
            "label" : "Schema Mapping",
            "kind" : "HIDDEN",
            "description" : "Useful if you want to load multiple streams from one tap to multiple Postgres schemas.\n\nIf the tap sends the `stream_id` in `<schema_name>-<table_name>` format then this option overwrites the `default_target_schema` value.\nNote, that using `schema_mapping` you can overwrite the `default_target_schema_select_permission` value to grant SELECT permissions to different groups per schemas or optionally you can create indices automatically for the replicated tables.\n",
            "required" : "false",
            "protected" : false
          }, {
            "name" : "add_metadata_columns",
            "aliases" : [ ],
            "label" : "Add Metadata Columns",
            "value" : "true",
            "kind" : "HIDDEN",
            "description" : "Metadata columns add extra row level information about data ingestions, (i.e. when was the row read in source, when was inserted or deleted in postgres etc.) Metadata columns are creating automatically by adding extra columns to the tables with a column prefix `_SDC_`. The column names are following the stitch naming conventions documented at https://www.stitchdata.com/docs/data-structure/integration-schemas#sdc-columns. Enabling metadata columns will flag the deleted rows by setting the `_SDC_DELETED_AT` metadata column. Without the `add_metadata_columns` option the deleted rows from singer taps will not be recongisable in Postgres.",
            "required" : "false",
            "protected" : false
          }, {
            "name" : "hard_delete",
            "aliases" : [ ],
            "label" : "Hard Delete",
            "value" : "false",
            "kind" : "HIDDEN",
            "description" : "When `hard_delete` option is true then DELETE SQL commands will be performed in Postgres to delete rows in tables. It's achieved by continuously checking the `_SDC_DELETED_AT` metadata column sent by the singer tap. Due to deleting rows requires metadata columns, `hard_delete` option automatically enables the `add_metadata_columns` option as well.",
            "required" : "false",
            "protected" : false
          }, {
            "name" : "data_flattening_max_level",
            "aliases" : [ ],
            "label" : "Data Flattening Max Level",
            "value" : "10",
            "kind" : "HIDDEN",
            "description" : "Object type RECORD items from taps can be transformed to flattened columns by creating columns automatically. When value is 0 (default) then flattening functionality is turned off.",
            "required" : "false",
            "protected" : false
          }, {
            "name" : "primary_key_required",
            "aliases" : [ ],
            "label" : "Primary Key Required",
            "value" : "false",
            "kind" : "BOOLEAN",
            "description" : "Log based and Incremental replications on tables with no Primary Key cause duplicates when merging UPDATE events. When set to true, stop loading data if no Primary Key is defined.",
            "required" : "false",
            "protected" : false
          }, {
            "name" : "validate_records",
            "aliases" : [ ],
            "label" : "Validate Records",
            "value" : "false",
            "kind" : "BOOLEAN",
            "description" : "Validate every single record message to the corresponding JSON schema. This option is disabled by default and invalid RECORD messages will fail only at load time by Postgres. Enabling this option will detect invalid records earlier but could cause performance degradation.",
            "required" : "false",
            "protected" : false
          }, {
            "name" : "temp_dir",
            "aliases" : [ ],
            "label" : "Temporary Directory",
            "kind" : "HIDDEN",
            "description" : "(Default: platform-dependent) Directory of temporary CSV files with RECORD messages.",
            "required" : "false",
            "protected" : false
          } ],
          "variants" : [ ],
          "commands" : { },
          "matatikaHidden" : false,
          "_links" : {
            "self" : {
              "href" : "https://catalog.matatika.com/api/dataplugins/88dca526-43ac-478b-a680-e3d52388944f"
            },
            "update dataplugin" : {
              "href" : "https://catalog.matatika.com/api/workspaces/d542d2b9-7843-4a50-9bdb-ef445734904c/dataplugins/88dca526-43ac-478b-a680-e3d52388944f",
              "type" : "PUT"
            }
          }
        }
      },
      "managed" : true,
      "_links" : {
        "self" : {
          "href" : "https://catalog.matatika.com/api/datacomponents/9ff03243-d68c-4d74-a0ec-2f10263d2904"
        },
        "update datacomponent" : {
          "href" : "https://catalog.matatika.com/api/datacomponents/9ff03243-d68c-4d74-a0ec-2f10263d2904"
        },
        "delete datacomponent" : {
          "href" : "https://catalog.matatika.com/api/datacomponents/9ff03243-d68c-4d74-a0ec-2f10263d2904"
        }
      }
    }, {
      "id" : "ae26fbdd-6fd2-43ad-ba8f-d659e6a3e00c",
      "created" : "2022-11-04T15:19:21.258297",
      "lastModified" : "2022-11-04T15:19:21.258298",
      "name" : "dbt",
      "properties" : { },
      "commands" : {
        "compile" : {
          "args" : "compile",
          "description" : "Generates executable SQL from source model, test, and analysis files. Compiled SQL files are written to the target/ directory."
        },
        "seed" : {
          "args" : "seed",
          "description" : "Load data from csv files into your data warehouse."
        },
        "test" : {
          "args" : "test",
          "description" : "Runs tests on data in deployed models."
        },
        "deps" : {
          "args" : "deps",
          "description" : "Pull the most recent version of the dependencies listed in packages.yml"
        },
        "run" : {
          "args" : "run",
          "description" : "Compile SQL and execute against the current target database."
        },
        "clean" : {
          "args" : "clean",
          "description" : "Delete all folders in the clean-targets list (usually the dbt_modules and target directories.)"
        },
        "snapshot" : {
          "args" : "snapshot",
          "description" : "Execute snapshots defined in your project."
        }
      },
      "_embedded" : {
        "dataplugin" : {
          "id" : "5b397a9a-ed83-4c60-9bb7-6ea049099e29",
          "pluginType" : "TRANSFORMER",
          "name" : "dbt",
          "namespace" : "dbt",
          "variant" : "dbt",
          "logoUrl" : "/assets/images/transformer/dbt.png",
          "hidden" : false,
          "docs" : "https://meltano.com/docs/transforms.html",
          "pipUrl" : "dbt-core~=1.0.0 dbt-postgres~=1.0.0 dbt-redshift~=1.0.0 dbt-snowflake~=1.0.0 dbt-bigquery~=1.0.0\n",
          "repo" : "https://github.com/dbt-labs/dbt-core",
          "capabilities" : [ ],
          "select" : [ ],
          "update" : { },
          "vars" : { },
          "settings" : [ {
            "name" : "project_dir",
            "aliases" : [ ],
            "value" : "$MELTANO_PROJECT_ROOT/transform",
            "kind" : "STRING",
            "protected" : false
          }, {
            "name" : "profiles_dir",
            "aliases" : [ ],
            "value" : "$MELTANO_PROJECT_ROOT/transform/profile",
            "kind" : "STRING",
            "env" : "DBT_PROFILES_DIR",
            "protected" : false
          }, {
            "name" : "target",
            "aliases" : [ ],
            "value" : "$MELTANO_LOAD__DIALECT",
            "kind" : "STRING",
            "protected" : false
          }, {
            "name" : "source_schema",
            "aliases" : [ ],
            "value" : "$MELTANO_LOAD__TARGET_SCHEMA",
            "kind" : "STRING",
            "protected" : false
          }, {
            "name" : "target_schema",
            "aliases" : [ ],
            "value" : "analytics",
            "kind" : "STRING",
            "protected" : false
          }, {
            "name" : "models",
            "aliases" : [ ],
            "value" : "$MELTANO_TRANSFORM__PACKAGE_NAME $MELTANO_EXTRACTOR_NAMESPACE my_meltano_project",
            "kind" : "STRING",
            "protected" : false
          } ],
          "variants" : [ ],
          "commands" : {
            "compile" : {
              "args" : "compile",
              "description" : "Generates executable SQL from source model, test, and analysis files. Compiled SQL files are written to the target/ directory."
            },
            "seed" : {
              "args" : "seed",
              "description" : "Load data from csv files into your data warehouse."
            },
            "test" : {
              "args" : "test",
              "description" : "Runs tests on data in deployed models."
            },
            "deps" : {
              "args" : "deps",
              "description" : "Pull the most recent version of the dependencies listed in packages.yml"
            },
            "run" : {
              "args" : "run",
              "description" : "Compile SQL and execute against the current target database."
            },
            "clean" : {
              "args" : "clean",
              "description" : "Delete all folders in the clean-targets list (usually the dbt_modules and target directories.)"
            },
            "snapshot" : {
              "args" : "snapshot",
              "description" : "Execute snapshots defined in your project."
            }
          },
          "matatikaHidden" : false,
          "_links" : {
            "self" : {
              "href" : "https://catalog.matatika.com/api/dataplugins/5b397a9a-ed83-4c60-9bb7-6ea049099e29"
            },
            "update dataplugin" : {
              "href" : "https://catalog.matatika.com/api/workspaces/d542d2b9-7843-4a50-9bdb-ef445734904c/dataplugins/5b397a9a-ed83-4c60-9bb7-6ea049099e29",
              "type" : "PUT"
            }
          }
        }
      },
      "managed" : true,
      "_links" : {
        "self" : {
          "href" : "https://catalog.matatika.com/api/datacomponents/ae26fbdd-6fd2-43ad-ba8f-d659e6a3e00c"
        },
        "update datacomponent" : {
          "href" : "https://catalog.matatika.com/api/datacomponents/ae26fbdd-6fd2-43ad-ba8f-d659e6a3e00c"
        },
        "delete datacomponent" : {
          "href" : "https://catalog.matatika.com/api/datacomponents/ae26fbdd-6fd2-43ad-ba8f-d659e6a3e00c"
        }
      }
    } ],
    "latest job" : {
      "id" : "05654151-e938-49c5-b317-07ef153e69bb",
      "created" : "2022-11-04T15:19:51.342897",
      "type" : "WORKSPACE_CONFIG",
      "status" : "QUEUED",
      "_links" : {
        "self" : {
          "href" : "https://catalog.matatika.com/api/jobs/05654151-e938-49c5-b317-07ef153e69bb"
        },
        "delete job" : {
          "href" : "https://catalog.matatika.com/api/jobs/05654151-e938-49c5-b317-07ef153e69bb",
          "type" : "DELETE"
        },
        "logs" : {
          "href" : "https://catalog.matatika.com/api/jobs/05654151-e938-49c5-b317-07ef153e69bb/logs?sequence=0",
          "type" : "GET"
        },
        "withdraw job" : {
          "href" : "https://catalog.matatika.com/api/jobs/05654151-e938-49c5-b317-07ef153e69bb/stopped",
          "type" : "PUT"
        }
      }
    }
  },
  "_links" : {
    "update pipeline" : {
      "href" : "https://catalog.matatika.com/api/workspaces/d542d2b9-7843-4a50-9bdb-ef445734904c/pipelines/c3bb4dcd-63e7-4811-862e-99da31e488e4",
      "type" : "PUT"
    },
    "delete pipeline" : {
      "href" : "https://catalog.matatika.com/api/pipelines/c3bb4dcd-63e7-4811-862e-99da31e488e4",
      "type" : "DELETE"
    },
    "draft pipeline" : {
      "href" : "https://catalog.matatika.com/api/workspaces/d542d2b9-7843-4a50-9bdb-ef445734904c/pipelines/c3bb4dcd-63e7-4811-862e-99da31e488e4/draft",
      "type" : "PUT"
    },
    "self" : {
      "href" : "https://catalog.matatika.com/api/pipelines/c3bb4dcd-63e7-4811-862e-99da31e488e4"
    },
    "environment" : {
      "href" : "https://catalog.matatika.com/api/pipelines/c3bb4dcd-63e7-4811-862e-99da31e488e4/environment"
    },
    "jobs" : {
      "href" : "https://catalog.matatika.com/api/pipelines/c3bb4dcd-63e7-4811-862e-99da31e488e4/jobs",
      "type" : "GET"
    },
    "add subscription" : {
      "href" : "https://catalog.matatika.com/api/pipelines/c3bb4dcd-63e7-4811-862e-99da31e488e4/subscriptions"
    },
    "withdraw job" : {
      "href" : "https://catalog.matatika.com/api/jobs/05654151-e938-49c5-b317-07ef153e69bb/stopped",
      "type" : "PUT"
    },
    "latest job" : {
      "href" : "https://catalog.matatika.com/api/jobs/05654151-e938-49c5-b317-07ef153e69bb"
    }
  }
}

Initialise a pipeline in a workspace

POST

/api/workspaces/{workspace-id}/pipelines

Initialises a new pipeline in the workspace {workspace-id}.

Prerequisites

  • Workspace {workspace-id} must exist

Request

Example Snippets

  • cURL
curl -H "Authorization: Bearer $ACCESS_TOKEN" 'https://catalog.matatika.com/api/workspaces/d542d2b9-7843-4a50-9bdb-ef445734904c/pipelines' -i -X POST \
    -H 'Accept: application/json, application/javascript, text/javascript, text/json' \
    -H 'Content-Type: application/json'
  • Python (requests)
import requests

url = "https://catalog.matatika.com/api/workspaces/d542d2b9-7843-4a50-9bdb-ef445734904c/pipelines"

headers = {
  'Authorization': ACCESS_TOKEN
}

response = requests.request("POST", url, headers=headers)

print(response.text.encode('utf8'))

Response

200 OK

Pipeline with HAL links.

{
  "id" : "c3bb4dcd-63e7-4811-862e-99da31e488e4",
  "status" : "PROVISIONING",
  "timeout" : 0,
  "created" : "2022-11-04T15:19:48.197731024",
  "lastModified" : "2022-11-04T15:19:48.197731224",
  "properties" : { },
  "dataComponents" : [ ],
  "actions" : [ ],
  "_links" : {
    "create pipeline" : {
      "href" : "https://catalog.matatika.com/api/workspaces/d542d2b9-7843-4a50-9bdb-ef445734904c/pipelines/c3bb4dcd-63e7-4811-862e-99da31e488e4",
      "type" : "PUT"
    },
    "draft pipeline" : {
      "href" : "https://catalog.matatika.com/api/workspaces/d542d2b9-7843-4a50-9bdb-ef445734904c/pipelines/c3bb4dcd-63e7-4811-862e-99da31e488e4/draft",
      "type" : "PUT"
    },
    "validate pipeline" : {
      "href" : "https://catalog.matatika.com/api/workspaces/d542d2b9-7843-4a50-9bdb-ef445734904c/pipelines/validation",
      "type" : "POST"
    }
  }
}

Create or update a pipeline in a workspace

PUT

/api/workspaces/{workspace-id}/pipelines/{pipeline-id}

Creates or updates the pipeline {pipeline-id} in the workspace {workspace-id}.

Prerequisites

  • Workspace {workspace-id} must exist

Request

Body

Pipeline resource.

{
  "name" : "SIT-generated pipeline [2022-11-04T15:19:49.097677]",
  "dataComponents" : [ "tap-google-analytics", "Warehouse", "dbt" ],
  "schedule" : "0 0 0 25 12 ?",
  "properties" : {
    "tap-google-analytics.view_id" : "1234567890",
    "tap-google-analytics.oauth_credentials.client_secret" : "client_secret",
    "tap-google-analytics.reports" : "reports",
    "tap-google-analytics.oauth_credentials.refresh_token" : "refresh_token",
    "tap-google-analytics.start_date" : "2022-10-04T15:19:49.462298Z",
    "tap-google-analytics.end_date" : "2022-11-04T15:19:49.462404Z",
    "tap-google-analytics.oauth_credentials.client_id" : "client_id",
    "tap-google-analytics.oauth_credentials.access_token" : "access_token"
  }
}

Example Snippets

  • cURL
curl -H "Authorization: Bearer $ACCESS_TOKEN" 'https://catalog.matatika.com/api/workspaces/d542d2b9-7843-4a50-9bdb-ef445734904c/pipelines/c3bb4dcd-63e7-4811-862e-99da31e488e4' -i -X PUT \
    -H 'Accept: application/json, application/javascript, text/javascript, text/json' \
    -H 'Content-Type: application/json' \
    -d '{
  "name" : "SIT-generated pipeline [2022-11-04T15:19:49.097677]",
  "dataComponents" : [ "tap-google-analytics", "Warehouse", "dbt" ],
  "schedule" : "0 0 0 25 12 ?",
  "properties" : {
    "tap-google-analytics.view_id" : "1234567890",
    "tap-google-analytics.oauth_credentials.client_secret" : "client_secret",
    "tap-google-analytics.reports" : "reports",
    "tap-google-analytics.oauth_credentials.refresh_token" : "refresh_token",
    "tap-google-analytics.start_date" : "2022-10-04T15:19:49.462298Z",
    "tap-google-analytics.end_date" : "2022-11-04T15:19:49.462404Z",
    "tap-google-analytics.oauth_credentials.client_id" : "client_id",
    "tap-google-analytics.oauth_credentials.access_token" : "access_token"
  }
}'
  • Python (requests)
import requests

url = "https://catalog.matatika.com/api/workspaces/d542d2b9-7843-4a50-9bdb-ef445734904c/pipelines/c3bb4dcd-63e7-4811-862e-99da31e488e4"

data = {
  "name" : "SIT-generated pipeline [2022-11-04T15:19:49.097677]",
  "dataComponents" : [ "tap-google-analytics", "Warehouse", "dbt" ],
  "schedule" : "0 0 0 25 12 ?",
  "properties" : {
    "tap-google-analytics.view_id" : "1234567890",
    "tap-google-analytics.oauth_credentials.client_secret" : "client_secret",
    "tap-google-analytics.reports" : "reports",
    "tap-google-analytics.oauth_credentials.refresh_token" : "refresh_token",
    "tap-google-analytics.start_date" : "2022-10-04T15:19:49.462298Z",
    "tap-google-analytics.end_date" : "2022-11-04T15:19:49.462404Z",
    "tap-google-analytics.oauth_credentials.client_id" : "client_id",
    "tap-google-analytics.oauth_credentials.access_token" : "access_token"
  }
}
headers = {
  'Authorization': ACCESS_TOKEN
}

response = requests.request("PUT", url, headers=headers, data=data)

print(response.text.encode('utf8'))

Response

200 OK / 201 Created

Pipeline with HAL links.

{
  "id" : "c3bb4dcd-63e7-4811-862e-99da31e488e4",
  "status" : "PROVISIONING",
  "name" : "SIT-generated pipeline [2022-11-04T15:19:49.097677]",
  "schedule" : "0 0 0 25 12 ?",
  "timeout" : 0,
  "created" : "2022-11-04T15:19:49.734323",
  "lastModified" : "2022-11-04T15:19:49.734323",
  "properties" : {
    "tap-google-analytics.view_id" : "1234567890",
    "tap-google-analytics.oauth_credentials.client_secret" : "client_secret",
    "tap-google-analytics.reports" : "reports",
    "tap-google-analytics.oauth_credentials.refresh_token" : "refresh_token",
    "tap-google-analytics.start_date" : "2022-10-04T15:19:49.462298Z",
    "tap-google-analytics.end_date" : "2022-11-04T15:19:49.462404Z",
    "tap-google-analytics.oauth_credentials.client_id" : "client_id",
    "tap-google-analytics.oauth_credentials.access_token" : "access_token"
  },
  "dataComponents" : [ "tap-google-analytics", "Warehouse", "dbt" ],
  "actions" : [ ],
  "_embedded" : {
    "dataComponents" : [ {
      "id" : "1a355ff6-051f-4a00-b510-2d51321147b5",
      "created" : "2022-11-04T15:19:48.729983",
      "lastModified" : "2022-11-04T15:19:48.729983",
      "name" : "tap-google-analytics",
      "properties" : { },
      "commands" : { },
      "_embedded" : {
        "dataplugin" : {
          "id" : "e6414e68-225e-43a6-8e2e-800f3d89512e",
          "pluginType" : "EXTRACTOR",
          "name" : "tap-google-analytics",
          "namespace" : "tap_google_analytics",
          "variant" : "meltano",
          "label" : "Google Analytics",
          "description" : "App and website analytics platform hosted by Google\n\n## Prerequisites\n\nBefore importing data from Google Analytics you or your webmaster will need to have performed some initial setup to grant\nyourself access to the Google Analytics for your website.\n\n[Google Analytics Prerequisites](https://www.matatika.com/docs/data-sources/tap-google-analytics/prerequisites)\n\n## Data source settings\n### View ID\n\nOnce you have connected with your google account, the `View ID` drop down will allow you to choose which view you want to get your data from.\n\n### Start Date\n\nProviding a start date allows you to set how much historical data you wish to import into you workspace.\n\nThis date has to be in the ISO-8601 date format, see the example below.\n\nFormat: `YYYY-MM-DDTHH:MM:SSZ`\n\nExample: `2020-10-01T13:12:11Z`\n\n### End Date\n\nProviding an end date allows you to limit how much data you wish to import into you workspace.\n\nThis date has to be in the ISO-8601 date format, see the example below.\n\nFormat: `YYYY-MM-DDTHH:MM:SSZ`\n\nExample: `2020-10-01T13:12:11Z`\n\n### Reports\n\nProviding the path to a reports json file allows you to supply the tap with a user defined definition of tables and information you want to sync from Google Analytics.\nThis does require you to commit a file to your workspace repository so that Google Analytics can find it during run time, and may also break the dbt transforms that are required to have our included datasets work.\nFor more information on how to define a custom reports file see the [Meltano Hub Documentation](https://hub.meltano.com/extractors/google-analytics#reports).\n\n## Learn more\n\n[Matatika Google Analytics Data Source Documentation](https://www.matatika.com/docs/data-sources/tap-google-analytics)\n\n[Meltano Hub Google Analytics Documentation](https://hub.meltano.com/extractors/google-analytics)",
          "logoUrl" : "/assets/images/datasource/tap-google-analytics.svg",
          "hidden" : false,
          "docs" : "https://www.matatika.com/docs/data-sources/tap-google-analytics/",
          "pipUrl" : "git+https://github.com/Matatika/[email protected]",
          "repo" : "https://github.com/Matatika/tap-google-analytics",
          "capabilities" : [ "CATALOG", "DISCOVER", "STATE" ],
          "select" : [ ],
          "update" : { },
          "vars" : { },
          "settings" : [ {
            "name" : "oauth_credentials.authorization_url",
            "aliases" : [ ],
            "label" : "OAuth identity provider authorization endpoint used create and refresh tokens",
            "value" : "https://oauth2.googleapis.com/token",
            "kind" : "HIDDEN",
            "protected" : false
          }, {
            "name" : "oauth_credentials.scope",
            "aliases" : [ ],
            "label" : "OAuth scopes we need to request access to",
            "value" : "profile email https://www.googleapis.com/auth/analytics.readonly",
            "kind" : "HIDDEN",
            "protected" : false
          }, {
            "name" : "oauth_credentials.access_token",
            "aliases" : [ ],
            "label" : "Access Token",
            "kind" : "HIDDEN",
            "protected" : false
          }, {
            "name" : "oauth_credentials.refresh_token",
            "aliases" : [ ],
            "label" : "OAuth Refresh Token",
            "kind" : "HIDDEN",
            "protected" : false
          }, {
            "name" : "oauth_credentials.refresh_proxy_url",
            "aliases" : [ ],
            "label" : "Optional - will be called with 'oauth_credentials.refresh_token' to refresh the access token",
            "kind" : "HIDDEN",
            "required" : "false",
            "protected" : false
          }, {
            "name" : "oauth_credentials.refresh_proxy_url_auth",
            "aliases" : [ ],
            "label" : "Optional - Sets Authorization header on 'oauth_credentials.refresh_url' request",
            "kind" : "HIDDEN",
            "required" : "false",
            "protected" : false
          }, {
            "name" : "oauth_credentials.client_id",
            "aliases" : [ ],
            "label" : "Optional - OAuth Client ID used if refresh_proxy_url not supplied",
            "kind" : "HIDDEN",
            "required" : "false",
            "protected" : false
          }, {
            "name" : "oauth_credentials.client_secret",
            "aliases" : [ ],
            "label" : "Optional - OAuth Client Secret used if refresh_proxy_url not supplied",
            "kind" : "HIDDEN",
            "required" : "false",
            "protected" : false
          }, {
            "name" : "view_id",
            "aliases" : [ ],
            "label" : "View ID",
            "placeholder" : "Ex. 198343027",
            "kind" : "STRING",
            "protected" : false
          }, {
            "name" : "reports",
            "aliases" : [ ],
            "label" : "Reports",
            "placeholder" : "Ex. my_report_definition.json",
            "kind" : "STRING",
            "required" : "false",
            "protected" : false
          }, {
            "name" : "start_date",
            "aliases" : [ ],
            "kind" : "DATE_ISO8601",
            "protected" : false
          }, {
            "name" : "end_date",
            "aliases" : [ ],
            "kind" : "DATE_ISO8601",
            "required" : "false",
            "protected" : false
          } ],
          "variants" : [ ],
          "commands" : { },
          "matatikaHidden" : false,
          "_links" : {
            "self" : {
              "href" : "https://catalog.matatika.com/api/dataplugins/e6414e68-225e-43a6-8e2e-800f3d89512e"
            },
            "update dataplugin" : {
              "href" : "https://catalog.matatika.com/api/workspaces/d542d2b9-7843-4a50-9bdb-ef445734904c/dataplugins/e6414e68-225e-43a6-8e2e-800f3d89512e",
              "type" : "PUT"
            }
          }
        }
      },
      "managed" : false,
      "_links" : {
        "self" : {
          "href" : "https://catalog.matatika.com/api/datacomponents/1a355ff6-051f-4a00-b510-2d51321147b5"
        },
        "update datacomponent" : {
          "href" : "https://catalog.matatika.com/api/datacomponents/1a355ff6-051f-4a00-b510-2d51321147b5"
        },
        "delete datacomponent" : {
          "href" : "https://catalog.matatika.com/api/datacomponents/1a355ff6-051f-4a00-b510-2d51321147b5"
        }
      }
    }, {
      "id" : "9ff03243-d68c-4d74-a0ec-2f10263d2904",
      "created" : "2022-11-04T15:19:21.212897",
      "lastModified" : "2022-11-04T15:19:21.212898",
      "name" : "Warehouse",
      "properties" : {
        "password" : "Ht56HwdV_TcriFS46_gIDE08gw",
        "dbname" : "lgxcdzf",
        "default_target_schema" : "analytics",
        "port" : "5432",
        "host" : "nutty-ugli.db.elephantsql.com",
        "user" : "lgxcdzf"
      },
      "commands" : { },
      "_embedded" : {
        "dataplugin" : {
          "id" : "88dca526-43ac-478b-a680-e3d52388944f",
          "pluginType" : "LOADER",
          "name" : "target-postgres",
          "namespace" : "postgres_transferwise",
          "variant" : "transferwise",
          "label" : "PostgreSQL",
          "description" : "PostgreSQL Data Warehouse\n\nThe World's Most Advanced Open Source Relational Database\n\n## Settings\n\n### User\nPostgres user\n\n### Password\nPostgres password\n\n### Host\nPostgres host\nExample: `test-host.db.elephantsql.com`\n\n### Port\nPostgres port\nExample: `5432`\n\n### Database Name\nPostgres database name\n\n### Default Target Schema\nName of the schema where tables will be created (no database prefix)",
          "logoUrl" : "/assets/images/datastore/target-postgres.png",
          "hidden" : false,
          "docs" : "https://hub.meltano.com/loaders/postgres.html",
          "pipUrl" : "git+https://github.com/Matatika/[email protected]",
          "repo" : "https://github.com/transferwise/pipelinewise-target-postgres",
          "executable" : "target-postgres",
          "capabilities" : [ ],
          "select" : [ ],
          "update" : { },
          "vars" : { },
          "settings" : [ {
            "name" : "user",
            "aliases" : [ "username" ],
            "label" : "User",
            "kind" : "STRING",
            "description" : "PostgreSQL user",
            "protected" : false
          }, {
            "name" : "password",
            "aliases" : [ ],
            "label" : "Password",
            "kind" : "PASSWORD",
            "description" : "PostgreSQL password",
            "protected" : false
          }, {
            "name" : "host",
            "aliases" : [ "address" ],
            "label" : "Host",
            "kind" : "STRING",
            "description" : "PostgreSQL host",
            "protected" : false
          }, {
            "name" : "port",
            "aliases" : [ ],
            "label" : "Port",
            "value" : "5432",
            "kind" : "INTEGER",
            "description" : "PostgreSQL port",
            "protected" : false
          }, {
            "name" : "dbname",
            "aliases" : [ "database" ],
            "label" : "Database Name",
            "kind" : "STRING",
            "description" : "PostgreSQL database name",
            "protected" : false
          }, {
            "name" : "default_target_schema",
            "aliases" : [ ],
            "label" : "Default Target Schema",
            "value" : "analytics",
            "kind" : "STRING",
            "description" : "Name of the schema where the tables will be created.",
            "protected" : false
          }, {
            "name" : "ssl",
            "aliases" : [ ],
            "label" : "SSL",
            "value" : "false",
            "kind" : "HIDDEN",
            "required" : "false",
            "protected" : false,
            "value_post_processor" : "STRINGIFY"
          }, {
            "name" : "batch_size_rows",
            "aliases" : [ ],
            "label" : "Batch Size Rows",
            "value" : "100000",
            "kind" : "INTEGER",
            "description" : "Maximum number of rows in each batch. At the end of each batch, the rows in the batch are loaded into Postgres.",
            "required" : "false",
            "protected" : false
          }, {
            "name" : "underscore_camel_case_fields",
            "aliases" : [ ],
            "label" : "Underscore Camel Case Fields",
            "value" : "true",
            "kind" : "HIDDEN",
            "description" : "Enable underscoring camel case field names, make the resulting postgres column names the same as from the Meltano variant of target-postgres.",
            "required" : "false",
            "protected" : false
          }, {
            "name" : "flush_all_streams",
            "aliases" : [ ],
            "label" : "Flush All Streams",
            "value" : "false",
            "kind" : "HIDDEN",
            "description" : "Flush and load every stream into Postgres when one batch is full. Warning: This may trigger the COPY command to use files with low number of records.",
            "required" : "false",
            "protected" : false
          }, {
            "name" : "parallelism",
            "aliases" : [ ],
            "label" : "Parallelism",
            "value" : "0",
            "kind" : "HIDDEN",
            "description" : "The number of threads used to flush tables. 0 will create a thread for each stream, up to parallelism_max. -1 will create a thread for each CPU core. Any other positive number will create that number of threads, up to parallelism_max.",
            "required" : "false",
            "protected" : false
          }, {
            "name" : "parallelism_max",
            "aliases" : [ ],
            "label" : "Max Parallelism",
            "value" : "16",
            "kind" : "HIDDEN",
            "description" : "Max number of parallel threads to use when flushing tables.",
            "required" : "false",
            "protected" : false
          }, {
            "name" : "default_target_schema_select_permission",
            "aliases" : [ ],
            "label" : "Default Target Schema Select Permission",
            "kind" : "HIDDEN",
            "description" : "Grant USAGE privilege on newly created schemas and grant SELECT privilege on newly created tables to a specific role or a list of roles. If `schema_mapping` is not defined then every stream sent by the tap is granted accordingly.",
            "required" : "false",
            "protected" : false
          }, {
            "name" : "schema_mapping",
            "aliases" : [ ],
            "label" : "Schema Mapping",
            "kind" : "HIDDEN",
            "description" : "Useful if you want to load multiple streams from one tap to multiple Postgres schemas.\n\nIf the tap sends the `stream_id` in `<schema_name>-<table_name>` format then this option overwrites the `default_target_schema` value.\nNote, that using `schema_mapping` you can overwrite the `default_target_schema_select_permission` value to grant SELECT permissions to different groups per schemas or optionally you can create indices automatically for the replicated tables.\n",
            "required" : "false",
            "protected" : false
          }, {
            "name" : "add_metadata_columns",
            "aliases" : [ ],
            "label" : "Add Metadata Columns",
            "value" : "true",
            "kind" : "HIDDEN",
            "description" : "Metadata columns add extra row level information about data ingestions, (i.e. when was the row read in source, when was inserted or deleted in postgres etc.) Metadata columns are creating automatically by adding extra columns to the tables with a column prefix `_SDC_`. The column names are following the stitch naming conventions documented at https://www.stitchdata.com/docs/data-structure/integration-schemas#sdc-columns. Enabling metadata columns will flag the deleted rows by setting the `_SDC_DELETED_AT` metadata column. Without the `add_metadata_columns` option the deleted rows from singer taps will not be recongisable in Postgres.",
            "required" : "false",
            "protected" : false
          }, {
            "name" : "hard_delete",
            "aliases" : [ ],
            "label" : "Hard Delete",
            "value" : "false",
            "kind" : "HIDDEN",
            "description" : "When `hard_delete` option is true then DELETE SQL commands will be performed in Postgres to delete rows in tables. It's achieved by continuously checking the `_SDC_DELETED_AT` metadata column sent by the singer tap. Due to deleting rows requires metadata columns, `hard_delete` option automatically enables the `add_metadata_columns` option as well.",
            "required" : "false",
            "protected" : false
          }, {
            "name" : "data_flattening_max_level",
            "aliases" : [ ],
            "label" : "Data Flattening Max Level",
            "value" : "10",
            "kind" : "HIDDEN",
            "description" : "Object type RECORD items from taps can be transformed to flattened columns by creating columns automatically. When value is 0 (default) then flattening functionality is turned off.",
            "required" : "false",
            "protected" : false
          }, {
            "name" : "primary_key_required",
            "aliases" : [ ],
            "label" : "Primary Key Required",
            "value" : "false",
            "kind" : "BOOLEAN",
            "description" : "Log based and Incremental replications on tables with no Primary Key cause duplicates when merging UPDATE events. When set to true, stop loading data if no Primary Key is defined.",
            "required" : "false",
            "protected" : false
          }, {
            "name" : "validate_records",
            "aliases" : [ ],
            "label" : "Validate Records",
            "value" : "false",
            "kind" : "BOOLEAN",
            "description" : "Validate every single record message to the corresponding JSON schema. This option is disabled by default and invalid RECORD messages will fail only at load time by Postgres. Enabling this option will detect invalid records earlier but could cause performance degradation.",
            "required" : "false",
            "protected" : false
          }, {
            "name" : "temp_dir",
            "aliases" : [ ],
            "label" : "Temporary Directory",
            "kind" : "HIDDEN",
            "description" : "(Default: platform-dependent) Directory of temporary CSV files with RECORD messages.",
            "required" : "false",
            "protected" : false
          } ],
          "variants" : [ ],
          "commands" : { },
          "matatikaHidden" : false,
          "_links" : {
            "self" : {
              "href" : "https://catalog.matatika.com/api/dataplugins/88dca526-43ac-478b-a680-e3d52388944f"
            },
            "update dataplugin" : {
              "href" : "https://catalog.matatika.com/api/workspaces/d542d2b9-7843-4a50-9bdb-ef445734904c/dataplugins/88dca526-43ac-478b-a680-e3d52388944f",
              "type" : "PUT"
            }
          }
        }
      },
      "managed" : true,
      "_links" : {
        "self" : {
          "href" : "https://catalog.matatika.com/api/datacomponents/9ff03243-d68c-4d74-a0ec-2f10263d2904"
        },
        "update datacomponent" : {
          "href" : "https://catalog.matatika.com/api/datacomponents/9ff03243-d68c-4d74-a0ec-2f10263d2904"
        },
        "delete datacomponent" : {
          "href" : "https://catalog.matatika.com/api/datacomponents/9ff03243-d68c-4d74-a0ec-2f10263d2904"
        }
      }
    }, {
      "id" : "ae26fbdd-6fd2-43ad-ba8f-d659e6a3e00c",
      "created" : "2022-11-04T15:19:21.258297",
      "lastModified" : "2022-11-04T15:19:21.258298",
      "name" : "dbt",
      "properties" : { },
      "commands" : {
        "compile" : {
          "args" : "compile",
          "description" : "Generates executable SQL from source model, test, and analysis files. Compiled SQL files are written to the target/ directory."
        },
        "seed" : {
          "args" : "seed",
          "description" : "Load data from csv files into your data warehouse."
        },
        "test" : {
          "args" : "test",
          "description" : "Runs tests on data in deployed models."
        },
        "deps" : {
          "args" : "deps",
          "description" : "Pull the most recent version of the dependencies listed in packages.yml"
        },
        "run" : {
          "args" : "run",
          "description" : "Compile SQL and execute against the current target database."
        },
        "clean" : {
          "args" : "clean",
          "description" : "Delete all folders in the clean-targets list (usually the dbt_modules and target directories.)"
        },
        "snapshot" : {
          "args" : "snapshot",
          "description" : "Execute snapshots defined in your project."
        }
      },
      "_embedded" : {
        "dataplugin" : {
          "id" : "5b397a9a-ed83-4c60-9bb7-6ea049099e29",
          "pluginType" : "TRANSFORMER",
          "name" : "dbt",
          "namespace" : "dbt",
          "variant" : "dbt",
          "logoUrl" : "/assets/images/transformer/dbt.png",
          "hidden" : false,
          "docs" : "https://meltano.com/docs/transforms.html",
          "pipUrl" : "dbt-core~=1.0.0 dbt-postgres~=1.0.0 dbt-redshift~=1.0.0 dbt-snowflake~=1.0.0 dbt-bigquery~=1.0.0\n",
          "repo" : "https://github.com/dbt-labs/dbt-core",
          "capabilities" : [ ],
          "select" : [ ],
          "update" : { },
          "vars" : { },
          "settings" : [ {
            "name" : "project_dir",
            "aliases" : [ ],
            "value" : "$MELTANO_PROJECT_ROOT/transform",
            "kind" : "STRING",
            "protected" : false
          }, {
            "name" : "profiles_dir",
            "aliases" : [ ],
            "value" : "$MELTANO_PROJECT_ROOT/transform/profile",
            "kind" : "STRING",
            "env" : "DBT_PROFILES_DIR",
            "protected" : false
          }, {
            "name" : "target",
            "aliases" : [ ],
            "value" : "$MELTANO_LOAD__DIALECT",
            "kind" : "STRING",
            "protected" : false
          }, {
            "name" : "source_schema",
            "aliases" : [ ],
            "value" : "$MELTANO_LOAD__TARGET_SCHEMA",
            "kind" : "STRING",
            "protected" : false
          }, {
            "name" : "target_schema",
            "aliases" : [ ],
            "value" : "analytics",
            "kind" : "STRING",
            "protected" : false
          }, {
            "name" : "models",
            "aliases" : [ ],
            "value" : "$MELTANO_TRANSFORM__PACKAGE_NAME $MELTANO_EXTRACTOR_NAMESPACE my_meltano_project",
            "kind" : "STRING",
            "protected" : false
          } ],
          "variants" : [ ],
          "commands" : {
            "compile" : {
              "args" : "compile",
              "description" : "Generates executable SQL from source model, test, and analysis files. Compiled SQL files are written to the target/ directory."
            },
            "seed" : {
              "args" : "seed",
              "description" : "Load data from csv files into your data warehouse."
            },
            "test" : {
              "args" : "test",
              "description" : "Runs tests on data in deployed models."
            },
            "deps" : {
              "args" : "deps",
              "description" : "Pull the most recent version of the dependencies listed in packages.yml"
            },
            "run" : {
              "args" : "run",
              "description" : "Compile SQL and execute against the current target database."
            },
            "clean" : {
              "args" : "clean",
              "description" : "Delete all folders in the clean-targets list (usually the dbt_modules and target directories.)"
            },
            "snapshot" : {
              "args" : "snapshot",
              "description" : "Execute snapshots defined in your project."
            }
          },
          "matatikaHidden" : false,
          "_links" : {
            "self" : {
              "href" : "https://catalog.matatika.com/api/dataplugins/5b397a9a-ed83-4c60-9bb7-6ea049099e29"
            },
            "update dataplugin" : {
              "href" : "https://catalog.matatika.com/api/workspaces/d542d2b9-7843-4a50-9bdb-ef445734904c/dataplugins/5b397a9a-ed83-4c60-9bb7-6ea049099e29",
              "type" : "PUT"
            }
          }
        }
      },
      "managed" : true,
      "_links" : {
        "self" : {
          "href" : "https://catalog.matatika.com/api/datacomponents/ae26fbdd-6fd2-43ad-ba8f-d659e6a3e00c"
        },
        "update datacomponent" : {
          "href" : "https://catalog.matatika.com/api/datacomponents/ae26fbdd-6fd2-43ad-ba8f-d659e6a3e00c"
        },
        "delete datacomponent" : {
          "href" : "https://catalog.matatika.com/api/datacomponents/ae26fbdd-6fd2-43ad-ba8f-d659e6a3e00c"
        }
      }
    } ],
    "latest job" : {
      "id" : "05654151-e938-49c5-b317-07ef153e69bb",
      "created" : "2022-11-04T15:19:51.342897",
      "type" : "WORKSPACE_CONFIG",
      "status" : "QUEUED",
      "_links" : {
        "self" : {
          "href" : "https://catalog.matatika.com/api/jobs/05654151-e938-49c5-b317-07ef153e69bb"
        },
        "delete job" : {
          "href" : "https://catalog.matatika.com/api/jobs/05654151-e938-49c5-b317-07ef153e69bb",
          "type" : "DELETE"
        },
        "logs" : {
          "href" : "https://catalog.matatika.com/api/jobs/05654151-e938-49c5-b317-07ef153e69bb/logs?sequence=0",
          "type" : "GET"
        },
        "withdraw job" : {
          "href" : "https://catalog.matatika.com/api/jobs/05654151-e938-49c5-b317-07ef153e69bb/stopped",
          "type" : "PUT"
        }
      }
    }
  },
  "_links" : {
    "update pipeline" : {
      "href" : "https://catalog.matatika.com/api/workspaces/d542d2b9-7843-4a50-9bdb-ef445734904c/pipelines/c3bb4dcd-63e7-4811-862e-99da31e488e4",
      "type" : "PUT"
    },
    "delete pipeline" : {
      "href" : "https://catalog.matatika.com/api/pipelines/c3bb4dcd-63e7-4811-862e-99da31e488e4",
      "type" : "DELETE"
    },
    "draft pipeline" : {
      "href" : "https://catalog.matatika.com/api/workspaces/d542d2b9-7843-4a50-9bdb-ef445734904c/pipelines/c3bb4dcd-63e7-4811-862e-99da31e488e4/draft",
      "type" : "PUT"
    },
    "self" : {
      "href" : "https://catalog.matatika.com/api/pipelines/c3bb4dcd-63e7-4811-862e-99da31e488e4"
    },
    "environment" : {
      "href" : "https://catalog.matatika.com/api/pipelines/c3bb4dcd-63e7-4811-862e-99da31e488e4/environment"
    },
    "jobs" : {
      "href" : "https://catalog.matatika.com/api/pipelines/c3bb4dcd-63e7-4811-862e-99da31e488e4/jobs",
      "type" : "GET"
    },
    "add subscription" : {
      "href" : "https://catalog.matatika.com/api/pipelines/c3bb4dcd-63e7-4811-862e-99da31e488e4/subscriptions"
    },
    "withdraw job" : {
      "href" : "https://catalog.matatika.com/api/jobs/05654151-e938-49c5-b317-07ef153e69bb/stopped",
      "type" : "PUT"
    },
    "latest job" : {
      "href" : "https://catalog.matatika.com/api/jobs/05654151-e938-49c5-b317-07ef153e69bb"
    }
  }
}

Create or update a pipeline as a draft

PUT

/api/workspaces/{workspace-id}/pipelines/{pipeline-id}/draft

Creates or updates the pipeline {pipeline-id} in the workspace {workspace-id} as a draft.

Prerequisites

  • Workspace {workspace-id} must exist

Request

Body

Pipeline resource.

{
  "name" : "SIT-generated pipeline [2022-11-04T15:20:52.262260]",
  "dataComponents" : [ "tap-google-analytics", "Warehouse", "dbt" ]
}

Example Snippets

  • cURL
curl -H "Authorization: Bearer $ACCESS_TOKEN" 'https://catalog.matatika.com/api/workspaces/d542d2b9-7843-4a50-9bdb-ef445734904c/pipelines/81aa90ef-f48e-4846-856b-7c26b4c0d8ff/draft' -i -X PUT \
    -H 'Accept: application/json, application/javascript, text/javascript, text/json' \
    -H 'Content-Type: application/json' \
    -d '{
  "name" : "SIT-generated pipeline [2022-11-04T15:20:52.262260]",
  "dataComponents" : [ "tap-google-analytics", "Warehouse", "dbt" ]
}'
  • Python (requests)
import requests

url = "https://catalog.matatika.com/api/workspaces/d542d2b9-7843-4a50-9bdb-ef445734904c/pipelines/81aa90ef-f48e-4846-856b-7c26b4c0d8ff/draft"

data = {
  "name" : "SIT-generated pipeline [2022-11-04T15:20:52.262260]",
  "dataComponents" : [ "tap-google-analytics", "Warehouse", "dbt" ]
}
headers = {
  'Authorization': ACCESS_TOKEN
}

response = requests.request("PUT", url, headers=headers, data=data)

print(response.text.encode('utf8'))

Response

200 OK / 201 Created

Pipeline with HAL links.

{
  "id" : "81aa90ef-f48e-4846-856b-7c26b4c0d8ff",
  "status" : "PROVISIONING",
  "name" : "SIT-generated pipeline [2022-11-04T15:20:52.262260]",
  "timeout" : 0,
  "created" : "2022-11-04T15:20:52.431304",
  "lastModified" : "2022-11-04T15:20:52.431304",
  "properties" : { },
  "dataComponents" : [ "tap-google-analytics", "Warehouse", "dbt" ],
  "actions" : [ ],
  "_embedded" : {
    "dataComponents" : [ {
      "id" : "1a355ff6-051f-4a00-b510-2d51321147b5",
      "created" : "2022-11-04T15:19:48.729983",
      "lastModified" : "2022-11-04T15:19:48.729983",
      "name" : "tap-google-analytics",
      "properties" : { },
      "commands" : { },
      "_embedded" : {
        "dataplugin" : {
          "id" : "e6414e68-225e-43a6-8e2e-800f3d89512e",
          "pluginType" : "EXTRACTOR",
          "name" : "tap-google-analytics",
          "namespace" : "tap_google_analytics",
          "variant" : "meltano",
          "label" : "Google Analytics",
          "logoUrl" : "/assets/images/datasource/tap-google-analytics.svg",
          "hidden" : false,
          "docs" : "https://www.matatika.com/docs/data-sources/tap-google-analytics/",
          "pipUrl" : "git+https://github.com/Matatika/[email protected]",
          "repo" : "https://github.com/Matatika/tap-google-analytics",
          "capabilities" : [ "CATALOG", "DISCOVER", "STATE" ],
          "select" : [ ],
          "update" : { },
          "vars" : { },
          "settings" : [ {
            "name" : "oauth_credentials.authorization_url",
            "aliases" : [ ],
            "label" : "OAuth identity provider authorization endpoint used create and refresh tokens",
            "value" : "https://oauth2.googleapis.com/token",
            "kind" : "HIDDEN",
            "protected" : false
          }, {
            "name" : "oauth_credentials.scope",
            "aliases" : [ ],
            "label" : "OAuth scopes we need to request access to",
            "value" : "profile email https://www.googleapis.com/auth/analytics.readonly",
            "kind" : "HIDDEN",
            "protected" : false
          }, {
            "name" : "oauth_credentials.access_token",
            "aliases" : [ ],
            "label" : "Access Token",
            "kind" : "HIDDEN",
            "protected" : false
          }, {
            "name" : "oauth_credentials.refresh_token",
            "aliases" : [ ],
            "label" : "OAuth Refresh Token",
            "kind" : "HIDDEN",
            "protected" : false
          }, {
            "name" : "oauth_credentials.refresh_proxy_url",
            "aliases" : [ ],
            "label" : "Optional - will be called with 'oauth_credentials.refresh_token' to refresh the access token",
            "kind" : "HIDDEN",
            "required" : "false",
            "protected" : false
          }, {
            "name" : "oauth_credentials.refresh_proxy_url_auth",
            "aliases" : [ ],
            "label" : "Optional - Sets Authorization header on 'oauth_credentials.refresh_url' request",
            "kind" : "HIDDEN",
            "required" : "false",
            "protected" : false
          }, {
            "name" : "oauth_credentials.client_id",
            "aliases" : [ ],
            "label" : "Optional - OAuth Client ID used if refresh_proxy_url not supplied",
            "kind" : "HIDDEN",
            "required" : "false",
            "protected" : false
          }, {
            "name" : "oauth_credentials.client_secret",
            "aliases" : [ ],
            "label" : "Optional - OAuth Client Secret used if refresh_proxy_url not supplied",
            "kind" : "HIDDEN",
            "required" : "false",
            "protected" : false
          }, {
            "name" : "view_id",
            "aliases" : [ ],
            "label" : "View ID",
            "placeholder" : "Ex. 198343027",
            "kind" : "STRING",
            "protected" : false
          }, {
            "name" : "reports",
            "aliases" : [ ],
            "label" : "Reports",
            "placeholder" : "Ex. my_report_definition.json",
            "kind" : "STRING",
            "required" : "false",
            "protected" : false
          }, {
            "name" : "start_date",
            "aliases" : [ ],
            "kind" : "DATE_ISO8601",
            "protected" : false
          }, {
            "name" : "end_date",
            "aliases" : [ ],
            "kind" : "DATE_ISO8601",
            "required" : "false",
            "protected" : false
          } ],
          "variants" : [ ],
          "commands" : { },
          "matatikaHidden" : false,
          "_links" : {
            "self" : {
              "href" : "https://catalog.matatika.com/api/dataplugins/e6414e68-225e-43a6-8e2e-800f3d89512e"
            },
            "update dataplugin" : {
              "href" : "https://catalog.matatika.com/api/workspaces/d542d2b9-7843-4a50-9bdb-ef445734904c/dataplugins/e6414e68-225e-43a6-8e2e-800f3d89512e",
              "type" : "PUT"
            }
          }
        }
      },
      "managed" : false,
      "_links" : {
        "self" : {
          "href" : "https://catalog.matatika.com/api/datacomponents/1a355ff6-051f-4a00-b510-2d51321147b5"
        },
        "update datacomponent" : {
          "href" : "https://catalog.matatika.com/api/datacomponents/1a355ff6-051f-4a00-b510-2d51321147b5"
        },
        "delete datacomponent" : {
          "href" : "https://catalog.matatika.com/api/datacomponents/1a355ff6-051f-4a00-b510-2d51321147b5"
        }
      }
    }, {
      "id" : "9ff03243-d68c-4d74-a0ec-2f10263d2904",
      "created" : "2022-11-04T15:19:21.212897",
      "lastModified" : "2022-11-04T15:19:21.212898",
      "name" : "Warehouse",
      "properties" : {
        "password" : "Ht56HwdV_TcriFS46_gIDE08gw",
        "dbname" : "lgxcdzf",
        "default_target_schema" : "analytics",
        "port" : "5432",
        "host" : "nutty-ugli.db.elephantsql.com",
        "user" : "lgxcdzf"
      },
      "commands" : { },
      "_embedded" : {
        "dataplugin" : {
          "id" : "88dca526-43ac-478b-a680-e3d52388944f",
          "pluginType" : "LOADER",
          "name" : "target-postgres",
          "namespace" : "postgres_transferwise",
          "variant" : "transferwise",
          "label" : "PostgreSQL",
          "logoUrl" : "/assets/images/datastore/target-postgres.png",
          "hidden" : false,
          "docs" : "https://hub.meltano.com/loaders/postgres.html",
          "pipUrl" : "git+https://github.com/Matatika/[email protected]",
          "repo" : "https://github.com/transferwise/pipelinewise-target-postgres",
          "executable" : "target-postgres",
          "capabilities" : [ ],
          "select" : [ ],
          "update" : { },
          "vars" : { },
          "settings" : [ {
            "name" : "user",
            "aliases" : [ "username" ],
            "label" : "User",
            "kind" : "STRING",
            "description" : "PostgreSQL user",
            "protected" : false
          }, {
            "name" : "password",
            "aliases" : [ ],
            "label" : "Password",
            "kind" : "PASSWORD",
            "description" : "PostgreSQL password",
            "protected" : false
          }, {
            "name" : "host",
            "aliases" : [ "address" ],
            "label" : "Host",
            "kind" : "STRING",
            "description" : "PostgreSQL host",
            "protected" : false
          }, {
            "name" : "port",
            "aliases" : [ ],
            "label" : "Port",
            "value" : "5432",
            "kind" : "INTEGER",
            "description" : "PostgreSQL port",
            "protected" : false
          }, {
            "name" : "dbname",
            "aliases" : [ "database" ],
            "label" : "Database Name",
            "kind" : "STRING",
            "description" : "PostgreSQL database name",
            "protected" : false
          }, {
            "name" : "default_target_schema",
            "aliases" : [ ],
            "label" : "Default Target Schema",
            "value" : "analytics",
            "kind" : "STRING",
            "description" : "Name of the schema where the tables will be created.",
            "protected" : false
          }, {
            "name" : "ssl",
            "aliases" : [ ],
            "label" : "SSL",
            "value" : "false",
            "kind" : "HIDDEN",
            "required" : "false",
            "protected" : false,
            "value_post_processor" : "STRINGIFY"
          }, {
            "name" : "batch_size_rows",
            "aliases" : [ ],
            "label" : "Batch Size Rows",
            "value" : "100000",
            "kind" : "INTEGER",
            "description" : "Maximum number of rows in each batch. At the end of each batch, the rows in the batch are loaded into Postgres.",
            "required" : "false",
            "protected" : false
          }, {
            "name" : "underscore_camel_case_fields",
            "aliases" : [ ],
            "label" : "Underscore Camel Case Fields",
            "value" : "true",
            "kind" : "HIDDEN",
            "description" : "Enable underscoring camel case field names, make the resulting postgres column names the same as from the Meltano variant of target-postgres.",
            "required" : "false",
            "protected" : false
          }, {
            "name" : "flush_all_streams",
            "aliases" : [ ],
            "label" : "Flush All Streams",
            "value" : "false",
            "kind" : "HIDDEN",
            "description" : "Flush and load every stream into Postgres when one batch is full. Warning: This may trigger the COPY command to use files with low number of records.",
            "required" : "false",
            "protected" : false
          }, {
            "name" : "parallelism",
            "aliases" : [ ],
            "label" : "Parallelism",
            "value" : "0",
            "kind" : "HIDDEN",
            "description" : "The number of threads used to flush tables. 0 will create a thread for each stream, up to parallelism_max. -1 will create a thread for each CPU core. Any other positive number will create that number of threads, up to parallelism_max.",
            "required" : "false",
            "protected" : false
          }, {
            "name" : "parallelism_max",
            "aliases" : [ ],
            "label" : "Max Parallelism",
            "value" : "16",
            "kind" : "HIDDEN",
            "description" : "Max number of parallel threads to use when flushing tables.",
            "required" : "false",
            "protected" : false
          }, {
            "name" : "default_target_schema_select_permission",
            "aliases" : [ ],
            "label" : "Default Target Schema Select Permission",
            "kind" : "HIDDEN",
            "description" : "Grant USAGE privilege on newly created schemas and grant SELECT privilege on newly created tables to a specific role or a list of roles. If `schema_mapping` is not defined then every stream sent by the tap is granted accordingly.",
            "required" : "false",
            "protected" : false
          }, {
            "name" : "schema_mapping",
            "aliases" : [ ],
            "label" : "Schema Mapping",
            "kind" : "HIDDEN",
            "description" : "Useful if you want to load multiple streams from one tap to multiple Postgres schemas.\n\nIf the tap sends the `stream_id` in `<schema_name>-<table_name>` format then this option overwrites the `default_target_schema` value.\nNote, that using `schema_mapping` you can overwrite the `default_target_schema_select_permission` value to grant SELECT permissions to different groups per schemas or optionally you can create indices automatically for the replicated tables.\n",
            "required" : "false",
            "protected" : false
          }, {
            "name" : "add_metadata_columns",
            "aliases" : [ ],
            "label" : "Add Metadata Columns",
            "value" : "true",
            "kind" : "HIDDEN",
            "description" : "Metadata columns add extra row level information about data ingestions, (i.e. when was the row read in source, when was inserted or deleted in postgres etc.) Metadata columns are creating automatically by adding extra columns to the tables with a column prefix `_SDC_`. The column names are following the stitch naming conventions documented at https://www.stitchdata.com/docs/data-structure/integration-schemas#sdc-columns. Enabling metadata columns will flag the deleted rows by setting the `_SDC_DELETED_AT` metadata column. Without the `add_metadata_columns` option the deleted rows from singer taps will not be recongisable in Postgres.",
            "required" : "false",
            "protected" : false
          }, {
            "name" : "hard_delete",
            "aliases" : [ ],
            "label" : "Hard Delete",
            "value" : "false",
            "kind" : "HIDDEN",
            "description" : "When `hard_delete` option is true then DELETE SQL commands will be performed in Postgres to delete rows in tables. It's achieved by continuously checking the `_SDC_DELETED_AT` metadata column sent by the singer tap. Due to deleting rows requires metadata columns, `hard_delete` option automatically enables the `add_metadata_columns` option as well.",
            "required" : "false",
            "protected" : false
          }, {
            "name" : "data_flattening_max_level",
            "aliases" : [ ],
            "label" : "Data Flattening Max Level",
            "value" : "10",
            "kind" : "HIDDEN",
            "description" : "Object type RECORD items from taps can be transformed to flattened columns by creating columns automatically. When value is 0 (default) then flattening functionality is turned off.",
            "required" : "false",
            "protected" : false
          }, {
            "name" : "primary_key_required",
            "aliases" : [ ],
            "label" : "Primary Key Required",
            "value" : "false",
            "kind" : "BOOLEAN",
            "description" : "Log based and Incremental replications on tables with no Primary Key cause duplicates when merging UPDATE events. When set to true, stop loading data if no Primary Key is defined.",
            "required" : "false",
            "protected" : false
          }, {
            "name" : "validate_records",
            "aliases" : [ ],
            "label" : "Validate Records",
            "value" : "false",
            "kind" : "BOOLEAN",
            "description" : "Validate every single record message to the corresponding JSON schema. This option is disabled by default and invalid RECORD messages will fail only at load time by Postgres. Enabling this option will detect invalid records earlier but could cause performance degradation.",
            "required" : "false",
            "protected" : false
          }, {
            "name" : "temp_dir",
            "aliases" : [ ],
            "label" : "Temporary Directory",
            "kind" : "HIDDEN",
            "description" : "(Default: platform-dependent) Directory of temporary CSV files with RECORD messages.",
            "required" : "false",
            "protected" : false
          } ],
          "variants" : [ ],
          "commands" : { },
          "matatikaHidden" : false,
          "_links" : {
            "self" : {
              "href" : "https://catalog.matatika.com/api/dataplugins/88dca526-43ac-478b-a680-e3d52388944f"
            },
            "update dataplugin" : {
              "href" : "https://catalog.matatika.com/api/workspaces/d542d2b9-7843-4a50-9bdb-ef445734904c/dataplugins/88dca526-43ac-478b-a680-e3d52388944f",
              "type" : "PUT"
            }
          }
        }
      },
      "managed" : true,
      "_links" : {
        "self" : {
          "href" : "https://catalog.matatika.com/api/datacomponents/9ff03243-d68c-4d74-a0ec-2f10263d2904"
        },
        "update datacomponent" : {
          "href" : "https://catalog.matatika.com/api/datacomponents/9ff03243-d68c-4d74-a0ec-2f10263d2904"
        },
        "delete datacomponent" : {
          "href" : "https://catalog.matatika.com/api/datacomponents/9ff03243-d68c-4d74-a0ec-2f10263d2904"
        }
      }
    }, {
      "id" : "613fcaf4-735b-4eff-8f97-808517eccb1d",
      "created" : "2022-11-04T15:20:49.923904",
      "lastModified" : "2022-11-04T15:20:49.923916",
      "name" : "dbt",
      "properties" : { },
      "commands" : {
        "compile" : {
          "args" : "compile",
          "description" : "Generates executable SQL from source model, test, and analysis files. Compiled SQL files are written to the target/ directory."
        },
        "seed" : {
          "args" : "seed",
          "description" : "Load data from csv files into your data warehouse."
        },
        "test" : {
          "args" : "test",
          "description" : "Runs tests on data in deployed models."
        },
        "deps" : {
          "args" : "deps",
          "description" : "Pull the most recent version of the dependencies listed in packages.yml"
        },
        "run" : {
          "args" : "run",
          "description" : "Compile SQL and execute against the current target database."
        },
        "clean" : {
          "args" : "clean",
          "description" : "Delete all folders in the clean-targets list (usually the dbt_modules and target directories.)"
        },
        "snapshot" : {
          "args" : "snapshot",
          "description" : "Execute snapshots defined in your project."
        }
      },
      "_embedded" : {
        "dataplugin" : {
          "id" : "5b397a9a-ed83-4c60-9bb7-6ea049099e29",
          "pluginType" : "TRANSFORMER",
          "name" : "dbt",
          "namespace" : "dbt",
          "variant" : "dbt",
          "label" : "dbt",
          "logoUrl" : "/assets/images/transformer/dbt.png",
          "hidden" : false,
          "docs" : "https://meltano.com/docs/transforms.html",
          "pipUrl" : "dbt-core~=1.0.0 dbt-postgres~=1.0.0 dbt-redshift~=1.0.0 dbt-snowflake~=1.0.0 dbt-bigquery~=1.0.0\n",
          "repo" : "https://github.com/dbt-labs/dbt-core",
          "capabilities" : [ ],
          "select" : [ ],
          "update" : { },
          "vars" : { },
          "settings" : [ {
            "name" : "project_dir",
            "aliases" : [ ],
            "value" : "$MELTANO_PROJECT_ROOT/transform",
            "kind" : "STRING",
            "protected" : false
          }, {
            "name" : "profiles_dir",
            "aliases" : [ ],
            "value" : "$MELTANO_PROJECT_ROOT/transform/profile",
            "kind" : "STRING",
            "env" : "DBT_PROFILES_DIR",
            "protected" : false
          }, {
            "name" : "target",
            "aliases" : [ ],
            "value" : "$MELTANO_LOAD__DIALECT",
            "kind" : "STRING",
            "protected" : false
          }, {
            "name" : "source_schema",
            "aliases" : [ ],
            "value" : "$MELTANO_LOAD__TARGET_SCHEMA",
            "kind" : "STRING",
            "protected" : false
          }, {
            "name" : "target_schema",
            "aliases" : [ ],
            "value" : "analytics",
            "kind" : "STRING",
            "protected" : false
          }, {
            "name" : "models",
            "aliases" : [ ],
            "value" : "$MELTANO_TRANSFORM__PACKAGE_NAME $MELTANO_EXTRACTOR_NAMESPACE my_meltano_project",
            "kind" : "STRING",
            "protected" : false
          } ],
          "variants" : [ ],
          "commands" : {
            "compile" : {
              "args" : "compile",
              "description" : "Generates executable SQL from source model, test, and analysis files. Compiled SQL files are written to the target/ directory."
            },
            "seed" : {
              "args" : "seed",
              "description" : "Load data from csv files into your data warehouse."
            },
            "test" : {
              "args" : "test",
              "description" : "Runs tests on data in deployed models."
            },
            "deps" : {
              "args" : "deps",
              "description" : "Pull the most recent version of the dependencies listed in packages.yml"
            },
            "run" : {
              "args" : "run",
              "description" : "Compile SQL and execute against the current target database."
            },
            "clean" : {
              "args" : "clean",
              "description" : "Delete all folders in the clean-targets list (usually the dbt_modules and target directories.)"
            },
            "snapshot" : {
              "args" : "snapshot",
              "description" : "Execute snapshots defined in your project."
            }
          },
          "matatikaHidden" : false,
          "_links" : {
            "self" : {
              "href" : "https://catalog.matatika.com/api/dataplugins/5b397a9a-ed83-4c60-9bb7-6ea049099e29"
            },
            "update dataplugin" : {
              "href" : "https://catalog.matatika.com/api/workspaces/d542d2b9-7843-4a50-9bdb-ef445734904c/dataplugins/5b397a9a-ed83-4c60-9bb7-6ea049099e29",
              "type" : "PUT"
            }
          }
        }
      },
      "managed" : false,
      "_links" : {
        "self" : {
          "href" : "https://catalog.matatika.com/api/datacomponents/613fcaf4-735b-4eff-8f97-808517eccb1d"
        },
        "update datacomponent" : {
          "href" : "https://catalog.matatika.com/api/datacomponents/613fcaf4-735b-4eff-8f97-808517eccb1d"
        },
        "delete datacomponent" : {
          "href" : "https://catalog.matatika.com/api/datacomponents/613fcaf4-735b-4eff-8f97-808517eccb1d"
        }
      }
    } ],
    "latest job" : {
      "id" : "877aaeda-8c1c-4196-83a1-9f7b5eb14ed2",
      "created" : "2022-11-04T15:20:53.326587",
      "type" : "WORKSPACE_CONFIG",
      "status" : "QUEUED",
      "_links" : {
        "self" : {
          "href" : "https://catalog.matatika.com/api/jobs/877aaeda-8c1c-4196-83a1-9f7b5eb14ed2"
        },
        "delete job" : {
          "href" : "https://catalog.matatika.com/api/jobs/877aaeda-8c1c-4196-83a1-9f7b5eb14ed2",
          "type" : "DELETE"
        },
        "logs" : {
          "href" : "https://catalog.matatika.com/api/jobs/877aaeda-8c1c-4196-83a1-9f7b5eb14ed2/logs?sequence=0",
          "type" : "GET"
        },
        "withdraw job" : {
          "href" : "https://catalog.matatika.com/api/jobs/877aaeda-8c1c-4196-83a1-9f7b5eb14ed2/stopped",
          "type" : "PUT"
        }
      }
    }
  },
  "_links" : {
    "update pipeline" : {
      "href" : "https://catalog.matatika.com/api/workspaces/d542d2b9-7843-4a50-9bdb-ef445734904c/pipelines/81aa90ef-f48e-4846-856b-7c26b4c0d8ff",
      "type" : "PUT"
    },
    "delete pipeline" : {
      "href" : "https://catalog.matatika.com/api/pipelines/81aa90ef-f48e-4846-856b-7c26b4c0d8ff",
      "type" : "DELETE"
    },
    "draft pipeline" : {
      "href" : "https://catalog.matatika.com/api/workspaces/d542d2b9-7843-4a50-9bdb-ef445734904c/pipelines/81aa90ef-f48e-4846-856b-7c26b4c0d8ff/draft",
      "type" : "PUT"
    },
    "self" : {
      "href" : "https://catalog.matatika.com/api/pipelines/81aa90ef-f48e-4846-856b-7c26b4c0d8ff"
    },
    "environment" : {
      "href" : "https://catalog.matatika.com/api/pipelines/81aa90ef-f48e-4846-856b-7c26b4c0d8ff/environment"
    },
    "jobs" : {
      "href" : "https://catalog.matatika.com/api/pipelines/81aa90ef-f48e-4846-856b-7c26b4c0d8ff/jobs",
      "type" : "GET"
    },
    "add subscription" : {
      "href" : "https://catalog.matatika.com/api/pipelines/81aa90ef-f48e-4846-856b-7c26b4c0d8ff/subscriptions"
    },
    "withdraw job" : {
      "href" : "https://catalog.matatika.com/api/jobs/877aaeda-8c1c-4196-83a1-9f7b5eb14ed2/stopped",
      "type" : "PUT"
    },
    "latest job" : {
      "href" : "https://catalog.matatika.com/api/jobs/877aaeda-8c1c-4196-83a1-9f7b5eb14ed2"
    }
  }
}

Validate a pipeline configuration in a workspace

POST

/api/workspaces/{workspace-id}/pipelines/validation

Validates a pipeline configuration in the workspace {workspace-id}.

Prerequisites

  • Workspace {workspace-id} must exist

Request

Body

Pipeline resource.

{
  "name" : "SIT-generated pipeline [2022-11-04T15:19:49.097677]",
  "dataComponents" : [ "tap-google-analytics", "Warehouse", "dbt" ]
}

Example Snippets

  • cURL
curl -H "Authorization: Bearer $ACCESS_TOKEN" 'https://catalog.matatika.com/api/workspaces/d542d2b9-7843-4a50-9bdb-ef445734904c/pipelines/validation' -i -X POST \
    -H 'Accept: application/json, application/javascript, text/javascript, text/json' \
    -H 'Content-Type: application/json' \
    -d '{
  "name" : "SIT-generated pipeline [2022-11-04T15:19:49.097677]",
  "dataComponents" : [ "tap-google-analytics", "Warehouse", "dbt" ]
}'
  • Python (requests)
import requests

url = "https://catalog.matatika.com/api/workspaces/d542d2b9-7843-4a50-9bdb-ef445734904c/pipelines/validation"

data = {
  "name" : "SIT-generated pipeline [2022-11-04T15:19:49.097677]",
  "dataComponents" : [ "tap-google-analytics", "Warehouse", "dbt" ]
}
headers = {
  'Authorization': ACCESS_TOKEN
}

response = requests.request("POST", url, headers=headers, data=data)

print(response.text.encode('utf8'))

Response

200 OK

No response body provided.

400 Bad Request

Pipeline property validation errors.

{
  "timestamp" : "2022-11-04T15:19:49.388912675",
  "status" : 400,
  "error" : "Bad Request",
  "message" : "4 validation errors from 'resource'",
  "errors" : [ {
    "codes" : [ "NotBlank.oauth_credentials.access_token", "NotBlank" ],
    "defaultMessage" : "No value given for setting",
    "objectName" : "resource",
    "field" : "properties.tap-google-analytics.oauth_credentials.access_token",
    "bindingFailure" : true,
    "code" : "NotBlank"
  }, {
    "codes" : [ "NotBlank.oauth_credentials.refresh_token", "NotBlank" ],
    "defaultMessage" : "No value given for setting",
    "objectName" : "resource",
    "field" : "properties.tap-google-analytics.oauth_credentials.refresh_token",
    "bindingFailure" : true,
    "code" : "NotBlank"
  }, {
    "codes" : [ "NotBlank.view_id", "NotBlank" ],
    "defaultMessage" : "No value given for setting",
    "objectName" : "resource",
    "field" : "properties.tap-google-analytics.view_id",
    "bindingFailure" : true,
    "code" : "NotBlank"
  }, {
    "codes" : [ "NotBlank.start_date", "NotBlank" ],
    "defaultMessage" : "No value given for setting",
    "objectName" : "resource",
    "field" : "properties.tap-google-analytics.start_date",
    "bindingFailure" : true,
    "code" : "NotBlank"
  } ],
  "path" : "/api/workspaces/d542d2b9-7843-4a50-9bdb-ef445734904c/pipelines/validation"
}

Verify a pipeline

POST

/api/pipelines/{pipeline-id}/verification

Verifies the configuration of the pipeline {pipeline-id}.

Prerequisites

  • Pipeline {pipeline-id} must exist

Request

Example Snippets

  • cURL
curl -H "Authorization: Bearer $ACCESS_TOKEN" 'https://catalog.matatika.com/api/pipelines/5a11f6b2-9dcc-403b-b783-e49879a49c51/verification' -i -X POST \
    -H 'Accept: application/json, application/javascript, text/javascript, text/json' \
    -H 'Content-Type: application/json'
  • Python (requests)
import requests

url = "https://catalog.matatika.com/api/pipelines/5a11f6b2-9dcc-403b-b783-e49879a49c51/verification"

headers = {
  'Authorization': ACCESS_TOKEN
}

response = requests.request("POST", url, headers=headers)

print(response.text.encode('utf8'))

Response

200 OK

Job with HAL links.

{
  "id" : "25c505be-a395-45bd-b6d6-a62872832348",
  "created" : "2022-11-04T15:24:24.224782",
  "type" : "PIPELINE_VERIFY",
  "status" : "QUEUED",
  "_links" : {
    "self" : {
      "href" : "https://catalog.matatika.com/api/jobs/25c505be-a395-45bd-b6d6-a62872832348"
    },
    "delete job" : {
      "href" : "https://catalog.matatika.com/api/jobs/25c505be-a395-45bd-b6d6-a62872832348",
      "type" : "DELETE"
    },
    "logs" : {
      "href" : "https://catalog.matatika.com/api/jobs/25c505be-a395-45bd-b6d6-a62872832348/logs?sequence=0",
      "type" : "GET"
    },
    "withdraw job" : {
      "href" : "https://catalog.matatika.com/api/jobs/25c505be-a395-45bd-b6d6-a62872832348/stopped",
      "type" : "PUT"
    }
  }
}

Delete a pipeline

DELETE

/api/pipelines/{pipeline-id}

Deletes the pipeline {pipeline-id}.

Prerequisites

  • Pipeline {pipeline-id} must exist

Request

Example Snippets

  • cURL
curl -H "Authorization: Bearer $ACCESS_TOKEN" 'https://catalog.matatika.com/api/pipelines/c3bb4dcd-63e7-4811-862e-99da31e488e4' -i -X DELETE \
    -H 'Accept: application/json, application/javascript, text/javascript, text/json' \
    -H 'Content-Type: application/json'
  • Python (requests)
import requests

url = "https://catalog.matatika.com/api/pipelines/c3bb4dcd-63e7-4811-862e-99da31e488e4"

headers = {
  'Authorization': ACCESS_TOKEN
}

response = requests.request("DELETE", url, headers=headers)

print(response.text.encode('utf8'))

Response

204 No Content

No response body provided.