Pipelines
A pipeline defines a set of runnable actions composed from datacomponents to complete a set of tasks - for example, ELT. Pipelines are run as jobs, either manually or on a predetermined schedule. Only a single pipeline can be run at any given time.
Objects
Pipeline
Path | Type | Format | Description |
---|---|---|---|
id |
String |
Version 4 UUID | The pipeline ID |
status |
String |
Pipeline Status | |
name |
String |
The pipeline name | |
schedule |
String |
Cron | The interval at which to launch a new job e.g. 0 0 9-17 * * MON-FRI launches a job on the hour nine-to-five weekdays |
timeout |
Integer |
Unsigned | The number of seconds after which the job will terminate - if set to 0 , an implicit default value of 300 seconds is used |
script |
String |
Bash script | Custom script to execute during a job |
created |
String |
ISO 8601 timestamp | When the pipeline was created |
lastModified |
String |
ISO 8601 timestamp | When the pipeline was last modified |
properties |
Properties |
The pipeline properties, defined by the dataplugin settings of each datacomponent |
|
dataComponents |
Array of String |
Array of Datacomponent name s |
The pipeline datacomponent name s or create / update with dataplugin fullyQualifiedName |
actions |
Array of String |
Array of Datacomponent name s or commands |
The pipeline actions to run during a job |
{
"id" : "7437c296-93ad-470c-b14d-24afc965c711",
"status" : "READY",
"name" : "SIT-generated pipeline [2023-03-23T21:46:50.230490] (updated)",
"timeout" : 0,
"created" : "2023-03-23T21:46:50.593501",
"lastModified" : "2023-03-23T21:46:50.593502",
"properties" : {
"tap-google-analytics.view_id" : "1234567890",
"tap-google-analytics.oauth_credentials.client_secret" : "client_secret",
"tap-google-analytics.reports" : "reports",
"tap-google-analytics.oauth_credentials.refresh_token" : "refresh_token",
"tap-google-analytics.start_date" : "2023-02-23T21:45:53.021107Z",
"tap-google-analytics.end_date" : "2023-03-23T21:45:53.021218Z",
"tap-google-analytics.oauth_credentials.client_id" : "client_id",
"tap-google-analytics.oauth_credentials.access_token" : "access_token"
},
"dataComponents" : [ "tap-google-analytics", "Warehouse", "dbt" ],
"actions" : [ ],
"_embedded" : {
"dataComponents" : [ {
"id" : "5be95c7e-4ef6-495d-b312-2e89c8cefaba",
"created" : "2023-03-23T21:45:52.26399",
"lastModified" : "2023-03-23T21:45:52.263991",
"name" : "tap-google-analytics",
"properties" : { },
"commands" : { },
"dataPlugin" : "extractors/tap-google-analytics--matatika",
"_embedded" : {
"dataplugin" : {
"id" : "d9e8ffbf-375a-4915-b629-2549c4c4060b",
"repositoryPath" : "plugins/extractors/tap-google-analytics--matatika.lock",
"pluginType" : "EXTRACTOR",
"name" : "tap-google-analytics",
"namespace" : "tap_google_analytics",
"variant" : "matatika",
"label" : "Google Analytics",
"description" : "App and website analytics platform hosted by Google\n\n## Prerequisites\n\nBefore importing data from Google Analytics you or your webmaster will need to have performed some initial setup to grant\nyourself access to the Google Analytics for your website.\n\n[Google Analytics Prerequisites](https://www.matatika.com/docs/data-sources/tap-google-analytics/prerequisites)\n\n## Data source settings\n### View ID\n\nOnce you have connected with your google account, the `View ID` drop down will allow you to choose which view you want to get your data from.\n\n### Start Date\n\nProviding a start date allows you to set how much historical data you wish to import into you workspace.\n\nThis date has to be in the ISO-8601 date format, see the example below.\n\nFormat: `YYYY-MM-DDTHH:MM:SSZ`\n\nExample: `2020-10-01T13:12:11Z`\n\n### End Date\n\nProviding an end date allows you to limit how much data you wish to import into you workspace.\n\nThis date has to be in the ISO-8601 date format, see the example below.\n\nFormat: `YYYY-MM-DDTHH:MM:SSZ`\n\nExample: `2020-10-01T13:12:11Z`\n\n### Reports\n\nProviding the path to a reports json file allows you to supply the tap with a user defined definition of tables and information you want to sync from Google Analytics.\nThis does require you to commit a file to your workspace repository so that Google Analytics can find it during run time, and may also break the dbt transforms that are required to have our included datasets work.\nFor more information on how to define a custom reports file see the [Meltano Hub Documentation](https://hub.meltano.com/extractors/google-analytics#reports).\n\n## Learn more\n\n[Matatika Google Analytics Data Source Documentation](https://www.matatika.com/docs/data-sources/tap-google-analytics)\n\n[Meltano Hub Google Analytics Documentation](https://hub.meltano.com/extractors/google-analytics)",
"logoUrl" : "https://app.matatika.com/assets/images/datasource/tap-google-analytics.svg",
"hidden" : false,
"docs" : "https://www.matatika.com/docs/data-sources/tap-google-analytics/",
"pipUrl" : "git+https://github.com/Matatika/[email protected]",
"repo" : "https://github.com/Matatika/tap-google-analytics",
"capabilities" : [ "catalog", "state", "discover" ],
"select" : [ ],
"update" : { },
"vars" : { },
"settings" : [ {
"name" : "oauth_credentials.authorization_url",
"aliases" : [ ],
"label" : "OAuth identity provider authorization endpoint used create and refresh tokens",
"value" : "https://oauth2.googleapis.com/token",
"kind" : "HIDDEN",
"required" : "true",
"protected" : false
}, {
"name" : "oauth_credentials.scope",
"aliases" : [ ],
"label" : "OAuth scopes we need to request access to",
"value" : "profile email https://www.googleapis.com/auth/analytics.readonly",
"kind" : "HIDDEN",
"required" : "true",
"protected" : false
}, {
"name" : "oauth_credentials.access_token",
"aliases" : [ ],
"label" : "Access Token",
"kind" : "HIDDEN",
"required" : "true",
"protected" : false
}, {
"name" : "oauth_credentials.refresh_token",
"aliases" : [ ],
"label" : "OAuth Refresh Token",
"kind" : "HIDDEN",
"required" : "true",
"protected" : false
}, {
"name" : "oauth_credentials.refresh_proxy_url",
"aliases" : [ ],
"label" : "Optional - will be called with 'oauth_credentials.refresh_token' to refresh the access token",
"kind" : "HIDDEN",
"protected" : false
}, {
"name" : "oauth_credentials.refresh_proxy_url_auth",
"aliases" : [ ],
"label" : "Optional - Sets Authorization header on 'oauth_credentials.refresh_url' request",
"kind" : "HIDDEN",
"protected" : false
}, {
"name" : "oauth_credentials.client_id",
"aliases" : [ ],
"label" : "Optional - OAuth Client ID used if refresh_proxy_url not supplied",
"kind" : "HIDDEN",
"protected" : false
}, {
"name" : "oauth_credentials.client_secret",
"aliases" : [ ],
"label" : "Optional - OAuth Client Secret used if refresh_proxy_url not supplied",
"kind" : "HIDDEN",
"protected" : false
}, {
"name" : "view_id",
"aliases" : [ ],
"label" : "View ID",
"placeholder" : "Ex. 198343027",
"kind" : "STRING",
"required" : "true",
"protected" : false
}, {
"name" : "reports",
"aliases" : [ ],
"label" : "Reports",
"placeholder" : "Ex. my_report_definition.json",
"kind" : "STRING",
"protected" : false
}, {
"name" : "start_date",
"aliases" : [ ],
"kind" : "DATE_ISO8601",
"protected" : false
}, {
"name" : "end_date",
"aliases" : [ ],
"kind" : "DATE_ISO8601",
"protected" : false
} ],
"variants" : [ ],
"commands" : { },
"matatikaHidden" : false,
"_links" : {
"self" : {
"href" : "https://catalog.matatika.com/api/dataplugins/d9e8ffbf-375a-4915-b629-2549c4c4060b"
},
"update dataplugin" : {
"href" : "https://catalog.matatika.com/api/workspaces/d2dad12c-4533-49c3-9d4a-cf03cf7aaaca/dataplugins/d9e8ffbf-375a-4915-b629-2549c4c4060b",
"type" : "PUT"
}
}
}
},
"managed" : false,
"draft" : false,
"_links" : {
"self" : {
"href" : "https://catalog.matatika.com/api/datacomponents/5be95c7e-4ef6-495d-b312-2e89c8cefaba"
},
"update datacomponent" : {
"href" : "https://catalog.matatika.com/api/datacomponents/5be95c7e-4ef6-495d-b312-2e89c8cefaba"
},
"delete datacomponent" : {
"href" : "https://catalog.matatika.com/api/datacomponents/5be95c7e-4ef6-495d-b312-2e89c8cefaba"
}
}
}, {
"id" : "a3de91eb-39a0-4ea8-a6a2-615d5ff721e3",
"created" : "2023-03-23T21:45:24.806929",
"lastModified" : "2023-03-23T21:45:24.80693",
"name" : "Warehouse",
"properties" : {
"password" : "Ptm5kD04_3ngU1Y6E_YeVd43da",
"dbname" : "ugdwwoy",
"default_target_schema" : "analytics",
"port" : "5432",
"host" : "nutty-ugli.db.elephantsql.com",
"user" : "ugdwwoy"
},
"commands" : { },
"dataPlugin" : "loaders/target-postgres--transferwise",
"_embedded" : {
"dataplugin" : {
"id" : "bed88248-835a-468d-a57c-b5b4ac24002c",
"repositoryPath" : "plugins/loaders/target-postgres--transferwise.lock",
"pluginType" : "LOADER",
"name" : "target-postgres",
"namespace" : "postgres_transferwise",
"variant" : "transferwise",
"label" : "Postgres Warehouse",
"description" : "Loader for Postgres Data Warehouse\n\nThe World's Most Advanced Open Source Relational Database\n\n## Settings\n\n### User\nPostgres user\n\n### Password\nPostgres password\n\n### Host\nPostgres host\nExample: `test-host.db.elephantsql.com`\n\n### Port\nPostgres port\nExample: `5432`\n\n### Database Name\nPostgres database name\n\n### Default Target Schema\nName of the schema where tables will be created (no database prefix)",
"logoUrl" : "https://app.matatika.com/assets/logos/loaders/postgres.png",
"hidden" : false,
"pipUrl" : "git+https://github.com/Matatika/[email protected]",
"repo" : "git+https://github.com/Matatika/[email protected]",
"capabilities" : [ ],
"select" : [ ],
"update" : { },
"vars" : { },
"settings" : [ {
"name" : "user",
"aliases" : [ "username" ],
"label" : "User",
"kind" : "STRING",
"description" : "PostgreSQL user",
"required" : "true",
"protected" : false
}, {
"name" : "password",
"aliases" : [ ],
"label" : "Password",
"kind" : "PASSWORD",
"description" : "PostgreSQL password",
"required" : "true",
"protected" : false
}, {
"name" : "host",
"aliases" : [ "address" ],
"label" : "Host",
"kind" : "STRING",
"description" : "PostgreSQL host",
"required" : "true",
"protected" : false
}, {
"name" : "port",
"aliases" : [ ],
"label" : "Port",
"value" : "5432",
"kind" : "INTEGER",
"description" : "PostgreSQL port",
"required" : "true",
"protected" : false
}, {
"name" : "dbname",
"aliases" : [ "database" ],
"label" : "Database Name",
"kind" : "STRING",
"description" : "PostgreSQL database name",
"required" : "true",
"protected" : false
}, {
"name" : "default_target_schema",
"aliases" : [ ],
"label" : "Default Target Schema",
"value" : "analytics",
"kind" : "STRING",
"description" : "Name of the schema where the tables will be created.",
"required" : "true",
"protected" : false
}, {
"name" : "ssl",
"aliases" : [ ],
"label" : "SSL",
"value" : "false",
"kind" : "HIDDEN",
"protected" : false,
"value_post_processor" : "STRINGIFY"
}, {
"name" : "batch_size_rows",
"aliases" : [ ],
"label" : "Batch Size Rows",
"value" : "100000",
"kind" : "INTEGER",
"description" : "Maximum number of rows in each batch. At the end of each batch, the rows in the batch are loaded into Postgres.",
"protected" : false
}, {
"name" : "underscore_camel_case_fields",
"aliases" : [ ],
"label" : "Underscore Camel Case Fields",
"value" : "true",
"kind" : "HIDDEN",
"description" : "Enable underscoring camel case field names, make the resulting postgres column names the same as from the Meltano variant of target-postgres.",
"protected" : false
}, {
"name" : "flush_all_streams",
"aliases" : [ ],
"label" : "Flush All Streams",
"value" : "false",
"kind" : "HIDDEN",
"description" : "Flush and load every stream into Postgres when one batch is full. Warning: This may trigger the COPY command to use files with low number of records.",
"protected" : false
}, {
"name" : "parallelism",
"aliases" : [ ],
"label" : "Parallelism",
"value" : "0",
"kind" : "HIDDEN",
"description" : "The number of threads used to flush tables. 0 will create a thread for each stream, up to parallelism_max. -1 will create a thread for each CPU core. Any other positive number will create that number of threads, up to parallelism_max.",
"protected" : false
}, {
"name" : "parallelism_max",
"aliases" : [ ],
"label" : "Max Parallelism",
"value" : "16",
"kind" : "HIDDEN",
"description" : "Max number of parallel threads to use when flushing tables.",
"protected" : false
}, {
"name" : "default_target_schema_select_permission",
"aliases" : [ ],
"label" : "Default Target Schema Select Permission",
"kind" : "HIDDEN",
"description" : "Grant USAGE privilege on newly created schemas and grant SELECT privilege on newly created tables to a specific role or a list of roles. If `schema_mapping` is not defined then every stream sent by the tap is granted accordingly.",
"protected" : false
}, {
"name" : "schema_mapping",
"aliases" : [ ],
"label" : "Schema Mapping",
"kind" : "HIDDEN",
"description" : "Useful if you want to load multiple streams from one tap to multiple Postgres schemas.\n\nIf the tap sends the `stream_id` in `<schema_name>-<table_name>` format then this option overwrites the `default_target_schema` value.\nNote, that using `schema_mapping` you can overwrite the `default_target_schema_select_permission` value to grant SELECT permissions to different groups per schemas or optionally you can create indices automatically for the replicated tables.\n",
"protected" : false
}, {
"name" : "add_metadata_columns",
"aliases" : [ ],
"label" : "Add Metadata Columns",
"value" : "true",
"kind" : "HIDDEN",
"description" : "Metadata columns add extra row level information about data ingestions, (i.e. when was the row read in source, when was inserted or deleted in postgres etc.) Metadata columns are creating automatically by adding extra columns to the tables with a column prefix `_SDC_`. The column names are following the stitch naming conventions documented at https://www.stitchdata.com/docs/data-structure/integration-schemas#sdc-columns. Enabling metadata columns will flag the deleted rows by setting the `_SDC_DELETED_AT` metadata column. Without the `add_metadata_columns` option the deleted rows from singer taps will not be recongisable in Postgres.",
"protected" : false
}, {
"name" : "hard_delete",
"aliases" : [ ],
"label" : "Hard Delete",
"value" : "false",
"kind" : "HIDDEN",
"description" : "When `hard_delete` option is true then DELETE SQL commands will be performed in Postgres to delete rows in tables. It's achieved by continuously checking the `_SDC_DELETED_AT` metadata column sent by the singer tap. Due to deleting rows requires metadata columns, `hard_delete` option automatically enables the `add_metadata_columns` option as well.",
"protected" : false
}, {
"name" : "data_flattening_max_level",
"aliases" : [ ],
"label" : "Data Flattening Max Level",
"value" : "10",
"kind" : "HIDDEN",
"description" : "Object type RECORD items from taps can be transformed to flattened columns by creating columns automatically. When value is 0 (default) then flattening functionality is turned off.",
"protected" : false
}, {
"name" : "primary_key_required",
"aliases" : [ ],
"label" : "Primary Key Required",
"value" : "false",
"kind" : "BOOLEAN",
"description" : "Log based and Incremental replications on tables with no Primary Key cause duplicates when merging UPDATE events. When set to true, stop loading data if no Primary Key is defined.",
"protected" : false
}, {
"name" : "validate_records",
"aliases" : [ ],
"label" : "Validate Records",
"value" : "false",
"kind" : "BOOLEAN",
"description" : "Validate every single record message to the corresponding JSON schema. This option is disabled by default and invalid RECORD messages will fail only at load time by Postgres. Enabling this option will detect invalid records earlier but could cause performance degradation.",
"protected" : false
}, {
"name" : "temp_dir",
"aliases" : [ ],
"label" : "Temporary Directory",
"kind" : "HIDDEN",
"description" : "(Default: platform-dependent) Directory of temporary CSV files with RECORD messages.",
"protected" : false
} ],
"variants" : [ ],
"commands" : { },
"matatikaHidden" : false,
"_links" : {
"self" : {
"href" : "https://catalog.matatika.com/api/dataplugins/bed88248-835a-468d-a57c-b5b4ac24002c"
},
"update dataplugin" : {
"href" : "https://catalog.matatika.com/api/workspaces/d2dad12c-4533-49c3-9d4a-cf03cf7aaaca/dataplugins/bed88248-835a-468d-a57c-b5b4ac24002c",
"type" : "PUT"
}
}
}
},
"managed" : true,
"draft" : false,
"_links" : {
"self" : {
"href" : "https://catalog.matatika.com/api/datacomponents/a3de91eb-39a0-4ea8-a6a2-615d5ff721e3"
},
"update datacomponent" : {
"href" : "https://catalog.matatika.com/api/datacomponents/a3de91eb-39a0-4ea8-a6a2-615d5ff721e3"
},
"delete datacomponent" : {
"href" : "https://catalog.matatika.com/api/datacomponents/a3de91eb-39a0-4ea8-a6a2-615d5ff721e3"
}
}
}, {
"id" : "166fc471-51cc-449f-b9cc-58b890eea347",
"created" : "2023-03-23T21:46:46.377154",
"lastModified" : "2023-03-23T21:46:46.377169",
"name" : "dbt",
"properties" : { },
"commands" : {
"compile" : {
"args" : "compile",
"description" : "Generates executable SQL from source model, test, and analysis files. Compiled SQL files are written to the target/ directory."
},
"seed" : {
"args" : "seed",
"description" : "Load data from csv files into your data warehouse."
},
"test" : {
"args" : "test",
"description" : "Runs tests on data in deployed models."
},
"deps" : {
"args" : "deps",
"description" : "Pull the most recent version of the dependencies listed in packages.yml"
},
"run" : {
"args" : "run",
"description" : "Compile SQL and execute against the current target database."
},
"clean" : {
"args" : "clean",
"description" : "Delete all folders in the clean-targets list (usually the dbt_modules and target directories.)"
},
"snapshot" : {
"args" : "snapshot",
"description" : "Execute snapshots defined in your project."
}
},
"dataPlugin" : "transformers/dbt--dbt-labs",
"_embedded" : {
"dataplugin" : {
"id" : "35a04646-0982-473d-9632-4d6bcc18d7cf",
"repositoryPath" : "plugins/transformers/dbt--dbt-labs.lock",
"pluginType" : "TRANSFORMER",
"name" : "dbt",
"namespace" : "dbt",
"variant" : "dbt-labs",
"label" : "dbt",
"logoUrl" : "https://app.matatika.com/assets/images/transformer/dbt.png",
"hidden" : false,
"docs" : "https://meltano.com/docs/transforms.html",
"pipUrl" : "dbt-core~=1.3.0 dbt-postgres~=1.3.0 dbt-snowflake~=1.3.0\n",
"repo" : "https://github.com/dbt-labs/dbt-core",
"capabilities" : [ ],
"select" : [ ],
"update" : { },
"vars" : { },
"settings" : [ {
"name" : "project_dir",
"aliases" : [ ],
"value" : "$MELTANO_PROJECT_ROOT/transform",
"kind" : "STRING",
"protected" : false
}, {
"name" : "profiles_dir",
"aliases" : [ ],
"value" : "$MELTANO_PROJECT_ROOT/transform/profile",
"kind" : "STRING",
"env" : "DBT_PROFILES_DIR",
"protected" : false
}, {
"name" : "target",
"aliases" : [ ],
"value" : "$MELTANO_LOAD__DIALECT",
"kind" : "STRING",
"protected" : false
}, {
"name" : "source_schema",
"aliases" : [ ],
"value" : "$MELTANO_LOAD__TARGET_SCHEMA",
"kind" : "STRING",
"protected" : false
}, {
"name" : "target_schema",
"aliases" : [ ],
"value" : "analytics",
"kind" : "STRING",
"protected" : false
}, {
"name" : "models",
"aliases" : [ ],
"value" : "$MELTANO_TRANSFORM__PACKAGE_NAME $MELTANO_EXTRACTOR_NAMESPACE my_meltano_project",
"kind" : "STRING",
"protected" : false
} ],
"variants" : [ ],
"commands" : {
"compile" : {
"args" : "compile",
"description" : "Generates executable SQL from source model, test, and analysis files. Compiled SQL files are written to the target/ directory."
},
"seed" : {
"args" : "seed",
"description" : "Load data from csv files into your data warehouse."
},
"test" : {
"args" : "test",
"description" : "Runs tests on data in deployed models."
},
"deps" : {
"args" : "deps",
"description" : "Pull the most recent version of the dependencies listed in packages.yml"
},
"run" : {
"args" : "run",
"description" : "Compile SQL and execute against the current target database."
},
"clean" : {
"args" : "clean",
"description" : "Delete all folders in the clean-targets list (usually the dbt_modules and target directories.)"
},
"snapshot" : {
"args" : "snapshot",
"description" : "Execute snapshots defined in your project."
}
},
"matatikaHidden" : false,
"_links" : {
"self" : {
"href" : "https://catalog.matatika.com/api/dataplugins/35a04646-0982-473d-9632-4d6bcc18d7cf"
},
"update dataplugin" : {
"href" : "https://catalog.matatika.com/api/workspaces/d2dad12c-4533-49c3-9d4a-cf03cf7aaaca/dataplugins/35a04646-0982-473d-9632-4d6bcc18d7cf",
"type" : "PUT"
}
}
}
},
"managed" : false,
"draft" : false,
"_links" : {
"self" : {
"href" : "https://catalog.matatika.com/api/datacomponents/166fc471-51cc-449f-b9cc-58b890eea347"
},
"update datacomponent" : {
"href" : "https://catalog.matatika.com/api/datacomponents/166fc471-51cc-449f-b9cc-58b890eea347"
},
"delete datacomponent" : {
"href" : "https://catalog.matatika.com/api/datacomponents/166fc471-51cc-449f-b9cc-58b890eea347"
}
}
} ],
"latest job" : {
"id" : "9cd1a03b-7749-42f0-a975-1364feeb0b45",
"created" : "2023-03-23T21:46:50.998498",
"type" : "WORKSPACE_CONFIG",
"exitCode" : 0,
"status" : "COMPLETE",
"startTime" : "2023-03-23T21:47:00.734",
"endTime" : "2023-03-23T21:47:14.715",
"_embedded" : {
"pipeline" : {
"id" : "7437c296-93ad-470c-b14d-24afc965c711",
"status" : "READY",
"name" : "SIT-generated pipeline [2023-03-23T21:46:50.230490] (updated)",
"timeout" : 0,
"created" : "2023-03-23T21:46:50.593501",
"lastModified" : "2023-03-23T21:46:50.593502",
"properties" : {
"tap-google-analytics.view_id" : "1234567890",
"tap-google-analytics.oauth_credentials.client_secret" : "client_secret",
"tap-google-analytics.reports" : "reports",
"tap-google-analytics.oauth_credentials.refresh_token" : "refresh_token",
"tap-google-analytics.start_date" : "2023-02-23T21:45:53.021107Z",
"tap-google-analytics.end_date" : "2023-03-23T21:45:53.021218Z",
"tap-google-analytics.oauth_credentials.client_id" : "client_id",
"tap-google-analytics.oauth_credentials.access_token" : "access_token"
},
"dataComponents" : [ "tap-google-analytics", "Warehouse", "dbt" ],
"actions" : [ ]
}
},
"_links" : {
"self" : {
"href" : "https://catalog.matatika.com/api/jobs/9cd1a03b-7749-42f0-a975-1364feeb0b45"
},
"delete job" : {
"href" : "https://catalog.matatika.com/api/jobs/9cd1a03b-7749-42f0-a975-1364feeb0b45",
"type" : "DELETE"
},
"logs" : {
"href" : "https://catalog.matatika.com/api/jobs/9cd1a03b-7749-42f0-a975-1364feeb0b45/logs?sequence=0",
"type" : "GET"
}
}
}
},
"_links" : {
"update pipeline" : {
"href" : "https://catalog.matatika.com/api/workspaces/d2dad12c-4533-49c3-9d4a-cf03cf7aaaca/pipelines/7437c296-93ad-470c-b14d-24afc965c711",
"type" : "PUT"
},
"delete pipeline" : {
"href" : "https://catalog.matatika.com/api/pipelines/7437c296-93ad-470c-b14d-24afc965c711",
"type" : "DELETE"
},
"draft pipeline" : {
"href" : "https://catalog.matatika.com/api/workspaces/d2dad12c-4533-49c3-9d4a-cf03cf7aaaca/pipelines/7437c296-93ad-470c-b14d-24afc965c711/draft",
"type" : "PUT"
},
"self" : {
"href" : "https://catalog.matatika.com/api/pipelines/7437c296-93ad-470c-b14d-24afc965c711"
},
"environment" : {
"href" : "https://catalog.matatika.com/api/pipelines/7437c296-93ad-470c-b14d-24afc965c711/environment"
},
"jobs" : {
"href" : "https://catalog.matatika.com/api/pipelines/7437c296-93ad-470c-b14d-24afc965c711/jobs",
"type" : "GET"
},
"add subscription" : {
"href" : "https://catalog.matatika.com/api/pipelines/7437c296-93ad-470c-b14d-24afc965c711/subscriptions"
},
"verify pipeline" : {
"href" : "https://catalog.matatika.com/api/pipelines/7437c296-93ad-470c-b14d-24afc965c711/verification",
"type" : "POST"
},
"create job" : {
"href" : "https://catalog.matatika.com/api/pipelines/7437c296-93ad-470c-b14d-24afc965c711/jobs",
"type" : "POST"
},
"latest job" : {
"href" : "https://catalog.matatika.com/api/jobs/9cd1a03b-7749-42f0-a975-1364feeb0b45"
}
}
}
Properties
For each setting s
in the datacomponents’ dataplugin settings
for each
Path | Type | Description |
---|---|---|
s.name |
s.kind |
Refer to s.description |
- Any required settings not satisfied by a datacomponent property must be provided as a pipeline property
- Any settings that are already satisfied by a datacomponent property can be overridden by a pipeline property
Formats
Pipeline Status
Value | Description |
---|---|
READY |
The pipeline completed processing resource changes |
PROVISIONING |
The pipeline is processing resource changes |
FAILED |
The pipeline failed to process resource changes |
Requests
- View all pipelines in a workspace
- View a pipeline
- Initialise a pipeline in a workspace
- Create or update a pipeline in a workspace
- Create or update a pipeline as a draft
- Validate a pipeline configuration in a workspace
- Verify a pipeline
- Delete a pipeline
See Also
- View all running or completed jobs for a pipeline
- Create a job from a pipeline
- Subscribe to a pipeline
View all pipelines in a workspace
GET
/api/workspaces/{workspace-id}/pipelines
Returns all configured pipelines in the workspace {workspace-id}
.
Prerequisites
- Workspace
{workspace-id}
must exist
Request
Example Snippets
- cURL
curl -H "Authorization: Bearer $ACCESS_TOKEN" 'https://catalog.matatika.com:443/api/workspaces/d2dad12c-4533-49c3-9d4a-cf03cf7aaaca/pipelines' -i -X GET \
-H 'Accept: application/json, application/javascript, text/javascript, text/json' \
-H 'Content-Type: application/json'
- Python (requests)
import requests
url = "https://catalog.matatika.com:443/api/workspaces/d2dad12c-4533-49c3-9d4a-cf03cf7aaaca/pipelines"
headers = {
'Authorization': ACCESS_TOKEN
}
response = requests.request("GET", url, headers=headers)
print(response.text.encode('utf8'))
Response
200 OK
Pipeline collection with HAL links.
{
"_embedded" : {
"pipelines" : [ {
"id" : "48acbb57-1837-47d0-a230-ebb041a0fb39",
"status" : "DRAFT",
"name" : "SIT-generated pipeline [2023-03-23T21:45:52.793220]",
"schedule" : "0 0 0 25 12 ?",
"timeout" : 0,
"created" : "2023-03-23T21:45:53.448327",
"lastModified" : "2023-03-23T21:45:53.448328",
"properties" : { },
"dataComponents" : [ "tap-google-analytics", "Warehouse", "dbt" ],
"actions" : [ ],
"_embedded" : {
"dataComponents" : [ {
"id" : "5be95c7e-4ef6-495d-b312-2e89c8cefaba",
"created" : "2023-03-23T21:45:52.26399",
"lastModified" : "2023-03-23T21:45:52.263991",
"name" : "tap-google-analytics",
"properties" : { },
"commands" : { },
"dataPlugin" : "extractors/tap-google-analytics--matatika",
"_embedded" : {
"dataplugin" : {
"id" : "d9e8ffbf-375a-4915-b629-2549c4c4060b",
"repositoryPath" : "plugins/extractors/tap-google-analytics--matatika.lock",
"pluginType" : "EXTRACTOR",
"name" : "tap-google-analytics",
"namespace" : "tap_google_analytics",
"variant" : "matatika",
"label" : "Google Analytics",
"description" : "App and website analytics platform hosted by Google\n\n## Prerequisites\n\nBefore importing data from Google Analytics you or your webmaster will need to have performed some initial setup to grant\nyourself access to the Google Analytics for your website.\n\n[Google Analytics Prerequisites](https://www.matatika.com/docs/data-sources/tap-google-analytics/prerequisites)\n\n## Data source settings\n### View ID\n\nOnce you have connected with your google account, the `View ID` drop down will allow you to choose which view you want to get your data from.\n\n### Start Date\n\nProviding a start date allows you to set how much historical data you wish to import into you workspace.\n\nThis date has to be in the ISO-8601 date format, see the example below.\n\nFormat: `YYYY-MM-DDTHH:MM:SSZ`\n\nExample: `2020-10-01T13:12:11Z`\n\n### End Date\n\nProviding an end date allows you to limit how much data you wish to import into you workspace.\n\nThis date has to be in the ISO-8601 date format, see the example below.\n\nFormat: `YYYY-MM-DDTHH:MM:SSZ`\n\nExample: `2020-10-01T13:12:11Z`\n\n### Reports\n\nProviding the path to a reports json file allows you to supply the tap with a user defined definition of tables and information you want to sync from Google Analytics.\nThis does require you to commit a file to your workspace repository so that Google Analytics can find it during run time, and may also break the dbt transforms that are required to have our included datasets work.\nFor more information on how to define a custom reports file see the [Meltano Hub Documentation](https://hub.meltano.com/extractors/google-analytics#reports).\n\n## Learn more\n\n[Matatika Google Analytics Data Source Documentation](https://www.matatika.com/docs/data-sources/tap-google-analytics)\n\n[Meltano Hub Google Analytics Documentation](https://hub.meltano.com/extractors/google-analytics)",
"logoUrl" : "https://app.matatika.com/assets/images/datasource/tap-google-analytics.svg",
"hidden" : false,
"docs" : "https://www.matatika.com/docs/data-sources/tap-google-analytics/",
"pipUrl" : "git+https://github.com/Matatika/[email protected]",
"repo" : "https://github.com/Matatika/tap-google-analytics",
"capabilities" : [ "catalog", "state", "discover" ],
"select" : [ ],
"update" : { },
"vars" : { },
"settings" : [ {
"name" : "oauth_credentials.authorization_url",
"aliases" : [ ],
"label" : "OAuth identity provider authorization endpoint used create and refresh tokens",
"value" : "https://oauth2.googleapis.com/token",
"kind" : "HIDDEN",
"required" : "true",
"protected" : false
}, {
"name" : "oauth_credentials.scope",
"aliases" : [ ],
"label" : "OAuth scopes we need to request access to",
"value" : "profile email https://www.googleapis.com/auth/analytics.readonly",
"kind" : "HIDDEN",
"required" : "true",
"protected" : false
}, {
"name" : "oauth_credentials.access_token",
"aliases" : [ ],
"label" : "Access Token",
"kind" : "HIDDEN",
"required" : "true",
"protected" : false
}, {
"name" : "oauth_credentials.refresh_token",
"aliases" : [ ],
"label" : "OAuth Refresh Token",
"kind" : "HIDDEN",
"required" : "true",
"protected" : false
}, {
"name" : "oauth_credentials.refresh_proxy_url",
"aliases" : [ ],
"label" : "Optional - will be called with 'oauth_credentials.refresh_token' to refresh the access token",
"kind" : "HIDDEN",
"protected" : false
}, {
"name" : "oauth_credentials.refresh_proxy_url_auth",
"aliases" : [ ],
"label" : "Optional - Sets Authorization header on 'oauth_credentials.refresh_url' request",
"kind" : "HIDDEN",
"protected" : false
}, {
"name" : "oauth_credentials.client_id",
"aliases" : [ ],
"label" : "Optional - OAuth Client ID used if refresh_proxy_url not supplied",
"kind" : "HIDDEN",
"protected" : false
}, {
"name" : "oauth_credentials.client_secret",
"aliases" : [ ],
"label" : "Optional - OAuth Client Secret used if refresh_proxy_url not supplied",
"kind" : "HIDDEN",
"protected" : false
}, {
"name" : "view_id",
"aliases" : [ ],
"label" : "View ID",
"placeholder" : "Ex. 198343027",
"kind" : "STRING",
"required" : "true",
"protected" : false
}, {
"name" : "reports",
"aliases" : [ ],
"label" : "Reports",
"placeholder" : "Ex. my_report_definition.json",
"kind" : "STRING",
"protected" : false
}, {
"name" : "start_date",
"aliases" : [ ],
"kind" : "DATE_ISO8601",
"protected" : false
}, {
"name" : "end_date",
"aliases" : [ ],
"kind" : "DATE_ISO8601",
"protected" : false
} ],
"variants" : [ ],
"commands" : { },
"matatikaHidden" : false,
"_links" : {
"self" : {
"href" : "https://catalog.matatika.com/api/dataplugins/d9e8ffbf-375a-4915-b629-2549c4c4060b"
},
"update dataplugin" : {
"href" : "https://catalog.matatika.com/api/workspaces/d2dad12c-4533-49c3-9d4a-cf03cf7aaaca/dataplugins/d9e8ffbf-375a-4915-b629-2549c4c4060b",
"type" : "PUT"
}
}
}
},
"managed" : false,
"draft" : false,
"_links" : {
"self" : {
"href" : "https://catalog.matatika.com/api/datacomponents/5be95c7e-4ef6-495d-b312-2e89c8cefaba"
},
"update datacomponent" : {
"href" : "https://catalog.matatika.com/api/datacomponents/5be95c7e-4ef6-495d-b312-2e89c8cefaba"
},
"delete datacomponent" : {
"href" : "https://catalog.matatika.com/api/datacomponents/5be95c7e-4ef6-495d-b312-2e89c8cefaba"
}
}
}, {
"id" : "a3de91eb-39a0-4ea8-a6a2-615d5ff721e3",
"created" : "2023-03-23T21:45:24.806929",
"lastModified" : "2023-03-23T21:45:24.80693",
"name" : "Warehouse",
"properties" : {
"password" : "Ptm5kD04_3ngU1Y6E_YeVd43da",
"dbname" : "ugdwwoy",
"default_target_schema" : "analytics",
"port" : "5432",
"host" : "nutty-ugli.db.elephantsql.com",
"user" : "ugdwwoy"
},
"commands" : { },
"dataPlugin" : "loaders/target-postgres--transferwise",
"_embedded" : {
"dataplugin" : {
"id" : "bed88248-835a-468d-a57c-b5b4ac24002c",
"repositoryPath" : "plugins/loaders/target-postgres--transferwise.lock",
"pluginType" : "LOADER",
"name" : "target-postgres",
"namespace" : "postgres_transferwise",
"variant" : "transferwise",
"label" : "Postgres Warehouse",
"description" : "Loader for Postgres Data Warehouse\n\nThe World's Most Advanced Open Source Relational Database\n\n## Settings\n\n### User\nPostgres user\n\n### Password\nPostgres password\n\n### Host\nPostgres host\nExample: `test-host.db.elephantsql.com`\n\n### Port\nPostgres port\nExample: `5432`\n\n### Database Name\nPostgres database name\n\n### Default Target Schema\nName of the schema where tables will be created (no database prefix)",
"logoUrl" : "https://app.matatika.com/assets/logos/loaders/postgres.png",
"hidden" : false,
"pipUrl" : "git+https://github.com/Matatika/[email protected]",
"repo" : "git+https://github.com/Matatika/[email protected]",
"capabilities" : [ ],
"select" : [ ],
"update" : { },
"vars" : { },
"settings" : [ {
"name" : "user",
"aliases" : [ "username" ],
"label" : "User",
"kind" : "STRING",
"description" : "PostgreSQL user",
"required" : "true",
"protected" : false
}, {
"name" : "password",
"aliases" : [ ],
"label" : "Password",
"kind" : "PASSWORD",
"description" : "PostgreSQL password",
"required" : "true",
"protected" : false
}, {
"name" : "host",
"aliases" : [ "address" ],
"label" : "Host",
"kind" : "STRING",
"description" : "PostgreSQL host",
"required" : "true",
"protected" : false
}, {
"name" : "port",
"aliases" : [ ],
"label" : "Port",
"value" : "5432",
"kind" : "INTEGER",
"description" : "PostgreSQL port",
"required" : "true",
"protected" : false
}, {
"name" : "dbname",
"aliases" : [ "database" ],
"label" : "Database Name",
"kind" : "STRING",
"description" : "PostgreSQL database name",
"required" : "true",
"protected" : false
}, {
"name" : "default_target_schema",
"aliases" : [ ],
"label" : "Default Target Schema",
"value" : "analytics",
"kind" : "STRING",
"description" : "Name of the schema where the tables will be created.",
"required" : "true",
"protected" : false
}, {
"name" : "ssl",
"aliases" : [ ],
"label" : "SSL",
"value" : "false",
"kind" : "HIDDEN",
"protected" : false,
"value_post_processor" : "STRINGIFY"
}, {
"name" : "batch_size_rows",
"aliases" : [ ],
"label" : "Batch Size Rows",
"value" : "100000",
"kind" : "INTEGER",
"description" : "Maximum number of rows in each batch. At the end of each batch, the rows in the batch are loaded into Postgres.",
"protected" : false
}, {
"name" : "underscore_camel_case_fields",
"aliases" : [ ],
"label" : "Underscore Camel Case Fields",
"value" : "true",
"kind" : "HIDDEN",
"description" : "Enable underscoring camel case field names, make the resulting postgres column names the same as from the Meltano variant of target-postgres.",
"protected" : false
}, {
"name" : "flush_all_streams",
"aliases" : [ ],
"label" : "Flush All Streams",
"value" : "false",
"kind" : "HIDDEN",
"description" : "Flush and load every stream into Postgres when one batch is full. Warning: This may trigger the COPY command to use files with low number of records.",
"protected" : false
}, {
"name" : "parallelism",
"aliases" : [ ],
"label" : "Parallelism",
"value" : "0",
"kind" : "HIDDEN",
"description" : "The number of threads used to flush tables. 0 will create a thread for each stream, up to parallelism_max. -1 will create a thread for each CPU core. Any other positive number will create that number of threads, up to parallelism_max.",
"protected" : false
}, {
"name" : "parallelism_max",
"aliases" : [ ],
"label" : "Max Parallelism",
"value" : "16",
"kind" : "HIDDEN",
"description" : "Max number of parallel threads to use when flushing tables.",
"protected" : false
}, {
"name" : "default_target_schema_select_permission",
"aliases" : [ ],
"label" : "Default Target Schema Select Permission",
"kind" : "HIDDEN",
"description" : "Grant USAGE privilege on newly created schemas and grant SELECT privilege on newly created tables to a specific role or a list of roles. If `schema_mapping` is not defined then every stream sent by the tap is granted accordingly.",
"protected" : false
}, {
"name" : "schema_mapping",
"aliases" : [ ],
"label" : "Schema Mapping",
"kind" : "HIDDEN",
"description" : "Useful if you want to load multiple streams from one tap to multiple Postgres schemas.\n\nIf the tap sends the `stream_id` in `<schema_name>-<table_name>` format then this option overwrites the `default_target_schema` value.\nNote, that using `schema_mapping` you can overwrite the `default_target_schema_select_permission` value to grant SELECT permissions to different groups per schemas or optionally you can create indices automatically for the replicated tables.\n",
"protected" : false
}, {
"name" : "add_metadata_columns",
"aliases" : [ ],
"label" : "Add Metadata Columns",
"value" : "true",
"kind" : "HIDDEN",
"description" : "Metadata columns add extra row level information about data ingestions, (i.e. when was the row read in source, when was inserted or deleted in postgres etc.) Metadata columns are creating automatically by adding extra columns to the tables with a column prefix `_SDC_`. The column names are following the stitch naming conventions documented at https://www.stitchdata.com/docs/data-structure/integration-schemas#sdc-columns. Enabling metadata columns will flag the deleted rows by setting the `_SDC_DELETED_AT` metadata column. Without the `add_metadata_columns` option the deleted rows from singer taps will not be recongisable in Postgres.",
"protected" : false
}, {
"name" : "hard_delete",
"aliases" : [ ],
"label" : "Hard Delete",
"value" : "false",
"kind" : "HIDDEN",
"description" : "When `hard_delete` option is true then DELETE SQL commands will be performed in Postgres to delete rows in tables. It's achieved by continuously checking the `_SDC_DELETED_AT` metadata column sent by the singer tap. Due to deleting rows requires metadata columns, `hard_delete` option automatically enables the `add_metadata_columns` option as well.",
"protected" : false
}, {
"name" : "data_flattening_max_level",
"aliases" : [ ],
"label" : "Data Flattening Max Level",
"value" : "10",
"kind" : "HIDDEN",
"description" : "Object type RECORD items from taps can be transformed to flattened columns by creating columns automatically. When value is 0 (default) then flattening functionality is turned off.",
"protected" : false
}, {
"name" : "primary_key_required",
"aliases" : [ ],
"label" : "Primary Key Required",
"value" : "false",
"kind" : "BOOLEAN",
"description" : "Log based and Incremental replications on tables with no Primary Key cause duplicates when merging UPDATE events. When set to true, stop loading data if no Primary Key is defined.",
"protected" : false
}, {
"name" : "validate_records",
"aliases" : [ ],
"label" : "Validate Records",
"value" : "false",
"kind" : "BOOLEAN",
"description" : "Validate every single record message to the corresponding JSON schema. This option is disabled by default and invalid RECORD messages will fail only at load time by Postgres. Enabling this option will detect invalid records earlier but could cause performance degradation.",
"protected" : false
}, {
"name" : "temp_dir",
"aliases" : [ ],
"label" : "Temporary Directory",
"kind" : "HIDDEN",
"description" : "(Default: platform-dependent) Directory of temporary CSV files with RECORD messages.",
"protected" : false
} ],
"variants" : [ ],
"commands" : { },
"matatikaHidden" : false,
"_links" : {
"self" : {
"href" : "https://catalog.matatika.com/api/dataplugins/bed88248-835a-468d-a57c-b5b4ac24002c"
},
"update dataplugin" : {
"href" : "https://catalog.matatika.com/api/workspaces/d2dad12c-4533-49c3-9d4a-cf03cf7aaaca/dataplugins/bed88248-835a-468d-a57c-b5b4ac24002c",
"type" : "PUT"
}
}
}
},
"managed" : true,
"draft" : false,
"_links" : {
"self" : {
"href" : "https://catalog.matatika.com/api/datacomponents/a3de91eb-39a0-4ea8-a6a2-615d5ff721e3"
},
"update datacomponent" : {
"href" : "https://catalog.matatika.com/api/datacomponents/a3de91eb-39a0-4ea8-a6a2-615d5ff721e3"
},
"delete datacomponent" : {
"href" : "https://catalog.matatika.com/api/datacomponents/a3de91eb-39a0-4ea8-a6a2-615d5ff721e3"
}
}
}, {
"id" : "166fc471-51cc-449f-b9cc-58b890eea347",
"created" : "2023-03-23T21:46:46.377154",
"lastModified" : "2023-03-23T21:46:46.377169",
"name" : "dbt",
"properties" : { },
"commands" : {
"compile" : {
"args" : "compile",
"description" : "Generates executable SQL from source model, test, and analysis files. Compiled SQL files are written to the target/ directory."
},
"seed" : {
"args" : "seed",
"description" : "Load data from csv files into your data warehouse."
},
"test" : {
"args" : "test",
"description" : "Runs tests on data in deployed models."
},
"deps" : {
"args" : "deps",
"description" : "Pull the most recent version of the dependencies listed in packages.yml"
},
"run" : {
"args" : "run",
"description" : "Compile SQL and execute against the current target database."
},
"clean" : {
"args" : "clean",
"description" : "Delete all folders in the clean-targets list (usually the dbt_modules and target directories.)"
},
"snapshot" : {
"args" : "snapshot",
"description" : "Execute snapshots defined in your project."
}
},
"dataPlugin" : "transformers/dbt--dbt-labs",
"_embedded" : {
"dataplugin" : {
"id" : "35a04646-0982-473d-9632-4d6bcc18d7cf",
"repositoryPath" : "plugins/transformers/dbt--dbt-labs.lock",
"pluginType" : "TRANSFORMER",
"name" : "dbt",
"namespace" : "dbt",
"variant" : "dbt-labs",
"label" : "dbt",
"logoUrl" : "https://app.matatika.com/assets/images/transformer/dbt.png",
"hidden" : false,
"docs" : "https://meltano.com/docs/transforms.html",
"pipUrl" : "dbt-core~=1.3.0 dbt-postgres~=1.3.0 dbt-snowflake~=1.3.0\n",
"repo" : "https://github.com/dbt-labs/dbt-core",
"capabilities" : [ ],
"select" : [ ],
"update" : { },
"vars" : { },
"settings" : [ {
"name" : "project_dir",
"aliases" : [ ],
"value" : "$MELTANO_PROJECT_ROOT/transform",
"kind" : "STRING",
"protected" : false
}, {
"name" : "profiles_dir",
"aliases" : [ ],
"value" : "$MELTANO_PROJECT_ROOT/transform/profile",
"kind" : "STRING",
"env" : "DBT_PROFILES_DIR",
"protected" : false
}, {
"name" : "target",
"aliases" : [ ],
"value" : "$MELTANO_LOAD__DIALECT",
"kind" : "STRING",
"protected" : false
}, {
"name" : "source_schema",
"aliases" : [ ],
"value" : "$MELTANO_LOAD__TARGET_SCHEMA",
"kind" : "STRING",
"protected" : false
}, {
"name" : "target_schema",
"aliases" : [ ],
"value" : "analytics",
"kind" : "STRING",
"protected" : false
}, {
"name" : "models",
"aliases" : [ ],
"value" : "$MELTANO_TRANSFORM__PACKAGE_NAME $MELTANO_EXTRACTOR_NAMESPACE my_meltano_project",
"kind" : "STRING",
"protected" : false
} ],
"variants" : [ ],
"commands" : {
"compile" : {
"args" : "compile",
"description" : "Generates executable SQL from source model, test, and analysis files. Compiled SQL files are written to the target/ directory."
},
"seed" : {
"args" : "seed",
"description" : "Load data from csv files into your data warehouse."
},
"test" : {
"args" : "test",
"description" : "Runs tests on data in deployed models."
},
"deps" : {
"args" : "deps",
"description" : "Pull the most recent version of the dependencies listed in packages.yml"
},
"run" : {
"args" : "run",
"description" : "Compile SQL and execute against the current target database."
},
"clean" : {
"args" : "clean",
"description" : "Delete all folders in the clean-targets list (usually the dbt_modules and target directories.)"
},
"snapshot" : {
"args" : "snapshot",
"description" : "Execute snapshots defined in your project."
}
},
"matatikaHidden" : false,
"_links" : {
"self" : {
"href" : "https://catalog.matatika.com/api/dataplugins/35a04646-0982-473d-9632-4d6bcc18d7cf"
},
"update dataplugin" : {
"href" : "https://catalog.matatika.com/api/workspaces/d2dad12c-4533-49c3-9d4a-cf03cf7aaaca/dataplugins/35a04646-0982-473d-9632-4d6bcc18d7cf",
"type" : "PUT"
}
}
}
},
"managed" : false,
"draft" : false,
"_links" : {
"self" : {
"href" : "https://catalog.matatika.com/api/datacomponents/166fc471-51cc-449f-b9cc-58b890eea347"
},
"update datacomponent" : {
"href" : "https://catalog.matatika.com/api/datacomponents/166fc471-51cc-449f-b9cc-58b890eea347"
},
"delete datacomponent" : {
"href" : "https://catalog.matatika.com/api/datacomponents/166fc471-51cc-449f-b9cc-58b890eea347"
}
}
} ],
"latest job" : {
"id" : "85a3256d-3fce-48cc-b953-ca196d8504d7",
"created" : "2023-03-23T21:45:54.068873",
"type" : "WORKSPACE_CONFIG",
"exitCode" : 0,
"status" : "COMPLETE",
"startTime" : "2023-03-23T21:46:06.06",
"endTime" : "2023-03-23T21:46:47.497"
}
},
"_links" : {
"update pipeline" : {
"href" : "https://catalog.matatika.com/api/workspaces/d2dad12c-4533-49c3-9d4a-cf03cf7aaaca/pipelines/48acbb57-1837-47d0-a230-ebb041a0fb39",
"type" : "PUT"
},
"delete pipeline" : {
"href" : "https://catalog.matatika.com/api/pipelines/48acbb57-1837-47d0-a230-ebb041a0fb39",
"type" : "DELETE"
},
"draft pipeline" : {
"href" : "https://catalog.matatika.com/api/workspaces/d2dad12c-4533-49c3-9d4a-cf03cf7aaaca/pipelines/48acbb57-1837-47d0-a230-ebb041a0fb39/draft",
"type" : "PUT"
},
"self" : {
"href" : "https://catalog.matatika.com/api/pipelines/48acbb57-1837-47d0-a230-ebb041a0fb39"
},
"environment" : {
"href" : "https://catalog.matatika.com/api/pipelines/48acbb57-1837-47d0-a230-ebb041a0fb39/environment"
},
"jobs" : {
"href" : "https://catalog.matatika.com/api/pipelines/48acbb57-1837-47d0-a230-ebb041a0fb39/jobs",
"type" : "GET"
},
"add subscription" : {
"href" : "https://catalog.matatika.com/api/pipelines/48acbb57-1837-47d0-a230-ebb041a0fb39/subscriptions"
},
"verify pipeline" : {
"href" : "https://catalog.matatika.com/api/pipelines/48acbb57-1837-47d0-a230-ebb041a0fb39/verification",
"type" : "POST"
},
"latest job" : {
"href" : "https://catalog.matatika.com/api/jobs/85a3256d-3fce-48cc-b953-ca196d8504d7"
}
}
}, {
"id" : "7437c296-93ad-470c-b14d-24afc965c711",
"status" : "READY",
"name" : "SIT-generated pipeline [2023-03-23T21:46:50.230490] (updated)",
"timeout" : 0,
"created" : "2023-03-23T21:46:50.593501",
"lastModified" : "2023-03-23T21:46:50.593502",
"properties" : { },
"dataComponents" : [ "tap-google-analytics", "Warehouse", "dbt" ],
"actions" : [ ],
"_embedded" : {
"dataComponents" : [ {
"id" : "5be95c7e-4ef6-495d-b312-2e89c8cefaba",
"created" : "2023-03-23T21:45:52.26399",
"lastModified" : "2023-03-23T21:45:52.263991",
"name" : "tap-google-analytics",
"properties" : { },
"commands" : { },
"dataPlugin" : "extractors/tap-google-analytics--matatika",
"_embedded" : {
"dataplugin" : {
"id" : "d9e8ffbf-375a-4915-b629-2549c4c4060b",
"repositoryPath" : "plugins/extractors/tap-google-analytics--matatika.lock",
"pluginType" : "EXTRACTOR",
"name" : "tap-google-analytics",
"namespace" : "tap_google_analytics",
"variant" : "matatika",
"label" : "Google Analytics",
"description" : "App and website analytics platform hosted by Google\n\n## Prerequisites\n\nBefore importing data from Google Analytics you or your webmaster will need to have performed some initial setup to grant\nyourself access to the Google Analytics for your website.\n\n[Google Analytics Prerequisites](https://www.matatika.com/docs/data-sources/tap-google-analytics/prerequisites)\n\n## Data source settings\n### View ID\n\nOnce you have connected with your google account, the `View ID` drop down will allow you to choose which view you want to get your data from.\n\n### Start Date\n\nProviding a start date allows you to set how much historical data you wish to import into you workspace.\n\nThis date has to be in the ISO-8601 date format, see the example below.\n\nFormat: `YYYY-MM-DDTHH:MM:SSZ`\n\nExample: `2020-10-01T13:12:11Z`\n\n### End Date\n\nProviding an end date allows you to limit how much data you wish to import into you workspace.\n\nThis date has to be in the ISO-8601 date format, see the example below.\n\nFormat: `YYYY-MM-DDTHH:MM:SSZ`\n\nExample: `2020-10-01T13:12:11Z`\n\n### Reports\n\nProviding the path to a reports json file allows you to supply the tap with a user defined definition of tables and information you want to sync from Google Analytics.\nThis does require you to commit a file to your workspace repository so that Google Analytics can find it during run time, and may also break the dbt transforms that are required to have our included datasets work.\nFor more information on how to define a custom reports file see the [Meltano Hub Documentation](https://hub.meltano.com/extractors/google-analytics#reports).\n\n## Learn more\n\n[Matatika Google Analytics Data Source Documentation](https://www.matatika.com/docs/data-sources/tap-google-analytics)\n\n[Meltano Hub Google Analytics Documentation](https://hub.meltano.com/extractors/google-analytics)",
"logoUrl" : "https://app.matatika.com/assets/images/datasource/tap-google-analytics.svg",
"hidden" : false,
"docs" : "https://www.matatika.com/docs/data-sources/tap-google-analytics/",
"pipUrl" : "git+https://github.com/Matatika/[email protected]",
"repo" : "https://github.com/Matatika/tap-google-analytics",
"capabilities" : [ "catalog", "state", "discover" ],
"select" : [ ],
"update" : { },
"vars" : { },
"settings" : [ {
"name" : "oauth_credentials.authorization_url",
"aliases" : [ ],
"label" : "OAuth identity provider authorization endpoint used create and refresh tokens",
"value" : "https://oauth2.googleapis.com/token",
"kind" : "HIDDEN",
"required" : "true",
"protected" : false
}, {
"name" : "oauth_credentials.scope",
"aliases" : [ ],
"label" : "OAuth scopes we need to request access to",
"value" : "profile email https://www.googleapis.com/auth/analytics.readonly",
"kind" : "HIDDEN",
"required" : "true",
"protected" : false
}, {
"name" : "oauth_credentials.access_token",
"aliases" : [ ],
"label" : "Access Token",
"kind" : "HIDDEN",
"required" : "true",
"protected" : false
}, {
"name" : "oauth_credentials.refresh_token",
"aliases" : [ ],
"label" : "OAuth Refresh Token",
"kind" : "HIDDEN",
"required" : "true",
"protected" : false
}, {
"name" : "oauth_credentials.refresh_proxy_url",
"aliases" : [ ],
"label" : "Optional - will be called with 'oauth_credentials.refresh_token' to refresh the access token",
"kind" : "HIDDEN",
"protected" : false
}, {
"name" : "oauth_credentials.refresh_proxy_url_auth",
"aliases" : [ ],
"label" : "Optional - Sets Authorization header on 'oauth_credentials.refresh_url' request",
"kind" : "HIDDEN",
"protected" : false
}, {
"name" : "oauth_credentials.client_id",
"aliases" : [ ],
"label" : "Optional - OAuth Client ID used if refresh_proxy_url not supplied",
"kind" : "HIDDEN",
"protected" : false
}, {
"name" : "oauth_credentials.client_secret",
"aliases" : [ ],
"label" : "Optional - OAuth Client Secret used if refresh_proxy_url not supplied",
"kind" : "HIDDEN",
"protected" : false
}, {
"name" : "view_id",
"aliases" : [ ],
"label" : "View ID",
"placeholder" : "Ex. 198343027",
"kind" : "STRING",
"required" : "true",
"protected" : false
}, {
"name" : "reports",
"aliases" : [ ],
"label" : "Reports",
"placeholder" : "Ex. my_report_definition.json",
"kind" : "STRING",
"protected" : false
}, {
"name" : "start_date",
"aliases" : [ ],
"kind" : "DATE_ISO8601",
"protected" : false
}, {
"name" : "end_date",
"aliases" : [ ],
"kind" : "DATE_ISO8601",
"protected" : false
} ],
"variants" : [ ],
"commands" : { },
"matatikaHidden" : false,
"_links" : {
"self" : {
"href" : "https://catalog.matatika.com/api/dataplugins/d9e8ffbf-375a-4915-b629-2549c4c4060b"
},
"update dataplugin" : {
"href" : "https://catalog.matatika.com/api/workspaces/d2dad12c-4533-49c3-9d4a-cf03cf7aaaca/dataplugins/d9e8ffbf-375a-4915-b629-2549c4c4060b",
"type" : "PUT"
}
}
}
},
"managed" : false,
"draft" : false,
"_links" : {
"self" : {
"href" : "https://catalog.matatika.com/api/datacomponents/5be95c7e-4ef6-495d-b312-2e89c8cefaba"
},
"update datacomponent" : {
"href" : "https://catalog.matatika.com/api/datacomponents/5be95c7e-4ef6-495d-b312-2e89c8cefaba"
},
"delete datacomponent" : {
"href" : "https://catalog.matatika.com/api/datacomponents/5be95c7e-4ef6-495d-b312-2e89c8cefaba"
}
}
}, {
"id" : "a3de91eb-39a0-4ea8-a6a2-615d5ff721e3",
"created" : "2023-03-23T21:45:24.806929",
"lastModified" : "2023-03-23T21:45:24.80693",
"name" : "Warehouse",
"properties" : {
"password" : "Ptm5kD04_3ngU1Y6E_YeVd43da",
"dbname" : "ugdwwoy",
"default_target_schema" : "analytics",
"port" : "5432",
"host" : "nutty-ugli.db.elephantsql.com",
"user" : "ugdwwoy"
},
"commands" : { },
"dataPlugin" : "loaders/target-postgres--transferwise",
"_embedded" : {
"dataplugin" : {
"id" : "bed88248-835a-468d-a57c-b5b4ac24002c",
"repositoryPath" : "plugins/loaders/target-postgres--transferwise.lock",
"pluginType" : "LOADER",
"name" : "target-postgres",
"namespace" : "postgres_transferwise",
"variant" : "transferwise",
"label" : "Postgres Warehouse",
"description" : "Loader for Postgres Data Warehouse\n\nThe World's Most Advanced Open Source Relational Database\n\n## Settings\n\n### User\nPostgres user\n\n### Password\nPostgres password\n\n### Host\nPostgres host\nExample: `test-host.db.elephantsql.com`\n\n### Port\nPostgres port\nExample: `5432`\n\n### Database Name\nPostgres database name\n\n### Default Target Schema\nName of the schema where tables will be created (no database prefix)",
"logoUrl" : "https://app.matatika.com/assets/logos/loaders/postgres.png",
"hidden" : false,
"pipUrl" : "git+https://github.com/Matatika/[email protected]",
"repo" : "git+https://github.com/Matatika/[email protected]",
"capabilities" : [ ],
"select" : [ ],
"update" : { },
"vars" : { },
"settings" : [ {
"name" : "user",
"aliases" : [ "username" ],
"label" : "User",
"kind" : "STRING",
"description" : "PostgreSQL user",
"required" : "true",
"protected" : false
}, {
"name" : "password",
"aliases" : [ ],
"label" : "Password",
"kind" : "PASSWORD",
"description" : "PostgreSQL password",
"required" : "true",
"protected" : false
}, {
"name" : "host",
"aliases" : [ "address" ],
"label" : "Host",
"kind" : "STRING",
"description" : "PostgreSQL host",
"required" : "true",
"protected" : false
}, {
"name" : "port",
"aliases" : [ ],
"label" : "Port",
"value" : "5432",
"kind" : "INTEGER",
"description" : "PostgreSQL port",
"required" : "true",
"protected" : false
}, {
"name" : "dbname",
"aliases" : [ "database" ],
"label" : "Database Name",
"kind" : "STRING",
"description" : "PostgreSQL database name",
"required" : "true",
"protected" : false
}, {
"name" : "default_target_schema",
"aliases" : [ ],
"label" : "Default Target Schema",
"value" : "analytics",
"kind" : "STRING",
"description" : "Name of the schema where the tables will be created.",
"required" : "true",
"protected" : false
}, {
"name" : "ssl",
"aliases" : [ ],
"label" : "SSL",
"value" : "false",
"kind" : "HIDDEN",
"protected" : false,
"value_post_processor" : "STRINGIFY"
}, {
"name" : "batch_size_rows",
"aliases" : [ ],
"label" : "Batch Size Rows",
"value" : "100000",
"kind" : "INTEGER",
"description" : "Maximum number of rows in each batch. At the end of each batch, the rows in the batch are loaded into Postgres.",
"protected" : false
}, {
"name" : "underscore_camel_case_fields",
"aliases" : [ ],
"label" : "Underscore Camel Case Fields",
"value" : "true",
"kind" : "HIDDEN",
"description" : "Enable underscoring camel case field names, make the resulting postgres column names the same as from the Meltano variant of target-postgres.",
"protected" : false
}, {
"name" : "flush_all_streams",
"aliases" : [ ],
"label" : "Flush All Streams",
"value" : "false",
"kind" : "HIDDEN",
"description" : "Flush and load every stream into Postgres when one batch is full. Warning: This may trigger the COPY command to use files with low number of records.",
"protected" : false
}, {
"name" : "parallelism",
"aliases" : [ ],
"label" : "Parallelism",
"value" : "0",
"kind" : "HIDDEN",
"description" : "The number of threads used to flush tables. 0 will create a thread for each stream, up to parallelism_max. -1 will create a thread for each CPU core. Any other positive number will create that number of threads, up to parallelism_max.",
"protected" : false
}, {
"name" : "parallelism_max",
"aliases" : [ ],
"label" : "Max Parallelism",
"value" : "16",
"kind" : "HIDDEN",
"description" : "Max number of parallel threads to use when flushing tables.",
"protected" : false
}, {
"name" : "default_target_schema_select_permission",
"aliases" : [ ],
"label" : "Default Target Schema Select Permission",
"kind" : "HIDDEN",
"description" : "Grant USAGE privilege on newly created schemas and grant SELECT privilege on newly created tables to a specific role or a list of roles. If `schema_mapping` is not defined then every stream sent by the tap is granted accordingly.",
"protected" : false
}, {
"name" : "schema_mapping",
"aliases" : [ ],
"label" : "Schema Mapping",
"kind" : "HIDDEN",
"description" : "Useful if you want to load multiple streams from one tap to multiple Postgres schemas.\n\nIf the tap sends the `stream_id` in `<schema_name>-<table_name>` format then this option overwrites the `default_target_schema` value.\nNote, that using `schema_mapping` you can overwrite the `default_target_schema_select_permission` value to grant SELECT permissions to different groups per schemas or optionally you can create indices automatically for the replicated tables.\n",
"protected" : false
}, {
"name" : "add_metadata_columns",
"aliases" : [ ],
"label" : "Add Metadata Columns",
"value" : "true",
"kind" : "HIDDEN",
"description" : "Metadata columns add extra row level information about data ingestions, (i.e. when was the row read in source, when was inserted or deleted in postgres etc.) Metadata columns are creating automatically by adding extra columns to the tables with a column prefix `_SDC_`. The column names are following the stitch naming conventions documented at https://www.stitchdata.com/docs/data-structure/integration-schemas#sdc-columns. Enabling metadata columns will flag the deleted rows by setting the `_SDC_DELETED_AT` metadata column. Without the `add_metadata_columns` option the deleted rows from singer taps will not be recongisable in Postgres.",
"protected" : false
}, {
"name" : "hard_delete",
"aliases" : [ ],
"label" : "Hard Delete",
"value" : "false",
"kind" : "HIDDEN",
"description" : "When `hard_delete` option is true then DELETE SQL commands will be performed in Postgres to delete rows in tables. It's achieved by continuously checking the `_SDC_DELETED_AT` metadata column sent by the singer tap. Due to deleting rows requires metadata columns, `hard_delete` option automatically enables the `add_metadata_columns` option as well.",
"protected" : false
}, {
"name" : "data_flattening_max_level",
"aliases" : [ ],
"label" : "Data Flattening Max Level",
"value" : "10",
"kind" : "HIDDEN",
"description" : "Object type RECORD items from taps can be transformed to flattened columns by creating columns automatically. When value is 0 (default) then flattening functionality is turned off.",
"protected" : false
}, {
"name" : "primary_key_required",
"aliases" : [ ],
"label" : "Primary Key Required",
"value" : "false",
"kind" : "BOOLEAN",
"description" : "Log based and Incremental replications on tables with no Primary Key cause duplicates when merging UPDATE events. When set to true, stop loading data if no Primary Key is defined.",
"protected" : false
}, {
"name" : "validate_records",
"aliases" : [ ],
"label" : "Validate Records",
"value" : "false",
"kind" : "BOOLEAN",
"description" : "Validate every single record message to the corresponding JSON schema. This option is disabled by default and invalid RECORD messages will fail only at load time by Postgres. Enabling this option will detect invalid records earlier but could cause performance degradation.",
"protected" : false
}, {
"name" : "temp_dir",
"aliases" : [ ],
"label" : "Temporary Directory",
"kind" : "HIDDEN",
"description" : "(Default: platform-dependent) Directory of temporary CSV files with RECORD messages.",
"protected" : false
} ],
"variants" : [ ],
"commands" : { },
"matatikaHidden" : false,
"_links" : {
"self" : {
"href" : "https://catalog.matatika.com/api/dataplugins/bed88248-835a-468d-a57c-b5b4ac24002c"
},
"update dataplugin" : {
"href" : "https://catalog.matatika.com/api/workspaces/d2dad12c-4533-49c3-9d4a-cf03cf7aaaca/dataplugins/bed88248-835a-468d-a57c-b5b4ac24002c",
"type" : "PUT"
}
}
}
},
"managed" : true,
"draft" : false,
"_links" : {
"self" : {
"href" : "https://catalog.matatika.com/api/datacomponents/a3de91eb-39a0-4ea8-a6a2-615d5ff721e3"
},
"update datacomponent" : {
"href" : "https://catalog.matatika.com/api/datacomponents/a3de91eb-39a0-4ea8-a6a2-615d5ff721e3"
},
"delete datacomponent" : {
"href" : "https://catalog.matatika.com/api/datacomponents/a3de91eb-39a0-4ea8-a6a2-615d5ff721e3"
}
}
}, {
"id" : "166fc471-51cc-449f-b9cc-58b890eea347",
"created" : "2023-03-23T21:46:46.377154",
"lastModified" : "2023-03-23T21:46:46.377169",
"name" : "dbt",
"properties" : { },
"commands" : {
"compile" : {
"args" : "compile",
"description" : "Generates executable SQL from source model, test, and analysis files. Compiled SQL files are written to the target/ directory."
},
"seed" : {
"args" : "seed",
"description" : "Load data from csv files into your data warehouse."
},
"test" : {
"args" : "test",
"description" : "Runs tests on data in deployed models."
},
"deps" : {
"args" : "deps",
"description" : "Pull the most recent version of the dependencies listed in packages.yml"
},
"run" : {
"args" : "run",
"description" : "Compile SQL and execute against the current target database."
},
"clean" : {
"args" : "clean",
"description" : "Delete all folders in the clean-targets list (usually the dbt_modules and target directories.)"
},
"snapshot" : {
"args" : "snapshot",
"description" : "Execute snapshots defined in your project."
}
},
"dataPlugin" : "transformers/dbt--dbt-labs",
"_embedded" : {
"dataplugin" : {
"id" : "35a04646-0982-473d-9632-4d6bcc18d7cf",
"repositoryPath" : "plugins/transformers/dbt--dbt-labs.lock",
"pluginType" : "TRANSFORMER",
"name" : "dbt",
"namespace" : "dbt",
"variant" : "dbt-labs",
"label" : "dbt",
"logoUrl" : "https://app.matatika.com/assets/images/transformer/dbt.png",
"hidden" : false,
"docs" : "https://meltano.com/docs/transforms.html",
"pipUrl" : "dbt-core~=1.3.0 dbt-postgres~=1.3.0 dbt-snowflake~=1.3.0\n",
"repo" : "https://github.com/dbt-labs/dbt-core",
"capabilities" : [ ],
"select" : [ ],
"update" : { },
"vars" : { },
"settings" : [ {
"name" : "project_dir",
"aliases" : [ ],
"value" : "$MELTANO_PROJECT_ROOT/transform",
"kind" : "STRING",
"protected" : false
}, {
"name" : "profiles_dir",
"aliases" : [ ],
"value" : "$MELTANO_PROJECT_ROOT/transform/profile",
"kind" : "STRING",
"env" : "DBT_PROFILES_DIR",
"protected" : false
}, {
"name" : "target",
"aliases" : [ ],
"value" : "$MELTANO_LOAD__DIALECT",
"kind" : "STRING",
"protected" : false
}, {
"name" : "source_schema",
"aliases" : [ ],
"value" : "$MELTANO_LOAD__TARGET_SCHEMA",
"kind" : "STRING",
"protected" : false
}, {
"name" : "target_schema",
"aliases" : [ ],
"value" : "analytics",
"kind" : "STRING",
"protected" : false
}, {
"name" : "models",
"aliases" : [ ],
"value" : "$MELTANO_TRANSFORM__PACKAGE_NAME $MELTANO_EXTRACTOR_NAMESPACE my_meltano_project",
"kind" : "STRING",
"protected" : false
} ],
"variants" : [ ],
"commands" : {
"compile" : {
"args" : "compile",
"description" : "Generates executable SQL from source model, test, and analysis files. Compiled SQL files are written to the target/ directory."
},
"seed" : {
"args" : "seed",
"description" : "Load data from csv files into your data warehouse."
},
"test" : {
"args" : "test",
"description" : "Runs tests on data in deployed models."
},
"deps" : {
"args" : "deps",
"description" : "Pull the most recent version of the dependencies listed in packages.yml"
},
"run" : {
"args" : "run",
"description" : "Compile SQL and execute against the current target database."
},
"clean" : {
"args" : "clean",
"description" : "Delete all folders in the clean-targets list (usually the dbt_modules and target directories.)"
},
"snapshot" : {
"args" : "snapshot",
"description" : "Execute snapshots defined in your project."
}
},
"matatikaHidden" : false,
"_links" : {
"self" : {
"href" : "https://catalog.matatika.com/api/dataplugins/35a04646-0982-473d-9632-4d6bcc18d7cf"
},
"update dataplugin" : {
"href" : "https://catalog.matatika.com/api/workspaces/d2dad12c-4533-49c3-9d4a-cf03cf7aaaca/dataplugins/35a04646-0982-473d-9632-4d6bcc18d7cf",
"type" : "PUT"
}
}
}
},
"managed" : false,
"draft" : false,
"_links" : {
"self" : {
"href" : "https://catalog.matatika.com/api/datacomponents/166fc471-51cc-449f-b9cc-58b890eea347"
},
"update datacomponent" : {
"href" : "https://catalog.matatika.com/api/datacomponents/166fc471-51cc-449f-b9cc-58b890eea347"
},
"delete datacomponent" : {
"href" : "https://catalog.matatika.com/api/datacomponents/166fc471-51cc-449f-b9cc-58b890eea347"
}
}
} ],
"latest job" : {
"id" : "9cd1a03b-7749-42f0-a975-1364feeb0b45",
"created" : "2023-03-23T21:46:50.998498",
"type" : "WORKSPACE_CONFIG",
"exitCode" : 0,
"status" : "COMPLETE",
"startTime" : "2023-03-23T21:47:00.734",
"endTime" : "2023-03-23T21:47:14.715"
}
},
"_links" : {
"update pipeline" : {
"href" : "https://catalog.matatika.com/api/workspaces/d2dad12c-4533-49c3-9d4a-cf03cf7aaaca/pipelines/7437c296-93ad-470c-b14d-24afc965c711",
"type" : "PUT"
},
"delete pipeline" : {
"href" : "https://catalog.matatika.com/api/pipelines/7437c296-93ad-470c-b14d-24afc965c711",
"type" : "DELETE"
},
"draft pipeline" : {
"href" : "https://catalog.matatika.com/api/workspaces/d2dad12c-4533-49c3-9d4a-cf03cf7aaaca/pipelines/7437c296-93ad-470c-b14d-24afc965c711/draft",
"type" : "PUT"
},
"self" : {
"href" : "https://catalog.matatika.com/api/pipelines/7437c296-93ad-470c-b14d-24afc965c711"
},
"environment" : {
"href" : "https://catalog.matatika.com/api/pipelines/7437c296-93ad-470c-b14d-24afc965c711/environment"
},
"jobs" : {
"href" : "https://catalog.matatika.com/api/pipelines/7437c296-93ad-470c-b14d-24afc965c711/jobs",
"type" : "GET"
},
"add subscription" : {
"href" : "https://catalog.matatika.com/api/pipelines/7437c296-93ad-470c-b14d-24afc965c711/subscriptions"
},
"verify pipeline" : {
"href" : "https://catalog.matatika.com/api/pipelines/7437c296-93ad-470c-b14d-24afc965c711/verification",
"type" : "POST"
},
"create job" : {
"href" : "https://catalog.matatika.com/api/pipelines/7437c296-93ad-470c-b14d-24afc965c711/jobs",
"type" : "POST"
},
"latest job" : {
"href" : "https://catalog.matatika.com/api/jobs/9cd1a03b-7749-42f0-a975-1364feeb0b45"
}
}
} ]
},
"_links" : {
"self" : {
"href" : "https://catalog.matatika.com/api/workspaces/d2dad12c-4533-49c3-9d4a-cf03cf7aaaca/pipelines?page=0&size=20&sort=name,asc"
}
},
"page" : {
"size" : 20,
"totalElements" : 2,
"totalPages" : 1,
"number" : 0
}
}
View a pipeline
GET
/api/pipelines/{pipeline-id}
Returns the pipeline {pipeline-id}
.
Prerequisites
- Pipeline
{pipeline-id}
must exist
Request
Example Snippets
- cURL
curl -H "Authorization: Bearer $ACCESS_TOKEN" 'https://catalog.matatika.com:443/api/pipelines/7437c296-93ad-470c-b14d-24afc965c711' -i -X GET \
-H 'Accept: application/json, application/javascript, text/javascript, text/json' \
-H 'Content-Type: application/json'
- Python (requests)
import requests
url = "https://catalog.matatika.com:443/api/pipelines/7437c296-93ad-470c-b14d-24afc965c711"
headers = {
'Authorization': ACCESS_TOKEN
}
response = requests.request("GET", url, headers=headers)
print(response.text.encode('utf8'))
Response
200 OK
Pipeline with HAL links.
{
"id" : "7437c296-93ad-470c-b14d-24afc965c711",
"status" : "READY",
"name" : "SIT-generated pipeline [2023-03-23T21:46:50.230490] (updated)",
"timeout" : 0,
"created" : "2023-03-23T21:46:50.593501",
"lastModified" : "2023-03-23T21:46:50.593502",
"properties" : {
"tap-google-analytics.view_id" : "1234567890",
"tap-google-analytics.oauth_credentials.client_secret" : "client_secret",
"tap-google-analytics.reports" : "reports",
"tap-google-analytics.oauth_credentials.refresh_token" : "refresh_token",
"tap-google-analytics.start_date" : "2023-02-23T21:45:53.021107Z",
"tap-google-analytics.end_date" : "2023-03-23T21:45:53.021218Z",
"tap-google-analytics.oauth_credentials.client_id" : "client_id",
"tap-google-analytics.oauth_credentials.access_token" : "access_token"
},
"dataComponents" : [ "tap-google-analytics", "Warehouse", "dbt" ],
"actions" : [ ],
"_embedded" : {
"dataComponents" : [ {
"id" : "5be95c7e-4ef6-495d-b312-2e89c8cefaba",
"created" : "2023-03-23T21:45:52.26399",
"lastModified" : "2023-03-23T21:45:52.263991",
"name" : "tap-google-analytics",
"properties" : { },
"commands" : { },
"dataPlugin" : "extractors/tap-google-analytics--matatika",
"_embedded" : {
"dataplugin" : {
"id" : "d9e8ffbf-375a-4915-b629-2549c4c4060b",
"repositoryPath" : "plugins/extractors/tap-google-analytics--matatika.lock",
"pluginType" : "EXTRACTOR",
"name" : "tap-google-analytics",
"namespace" : "tap_google_analytics",
"variant" : "matatika",
"label" : "Google Analytics",
"description" : "App and website analytics platform hosted by Google\n\n## Prerequisites\n\nBefore importing data from Google Analytics you or your webmaster will need to have performed some initial setup to grant\nyourself access to the Google Analytics for your website.\n\n[Google Analytics Prerequisites](https://www.matatika.com/docs/data-sources/tap-google-analytics/prerequisites)\n\n## Data source settings\n### View ID\n\nOnce you have connected with your google account, the `View ID` drop down will allow you to choose which view you want to get your data from.\n\n### Start Date\n\nProviding a start date allows you to set how much historical data you wish to import into you workspace.\n\nThis date has to be in the ISO-8601 date format, see the example below.\n\nFormat: `YYYY-MM-DDTHH:MM:SSZ`\n\nExample: `2020-10-01T13:12:11Z`\n\n### End Date\n\nProviding an end date allows you to limit how much data you wish to import into you workspace.\n\nThis date has to be in the ISO-8601 date format, see the example below.\n\nFormat: `YYYY-MM-DDTHH:MM:SSZ`\n\nExample: `2020-10-01T13:12:11Z`\n\n### Reports\n\nProviding the path to a reports json file allows you to supply the tap with a user defined definition of tables and information you want to sync from Google Analytics.\nThis does require you to commit a file to your workspace repository so that Google Analytics can find it during run time, and may also break the dbt transforms that are required to have our included datasets work.\nFor more information on how to define a custom reports file see the [Meltano Hub Documentation](https://hub.meltano.com/extractors/google-analytics#reports).\n\n## Learn more\n\n[Matatika Google Analytics Data Source Documentation](https://www.matatika.com/docs/data-sources/tap-google-analytics)\n\n[Meltano Hub Google Analytics Documentation](https://hub.meltano.com/extractors/google-analytics)",
"logoUrl" : "https://app.matatika.com/assets/images/datasource/tap-google-analytics.svg",
"hidden" : false,
"docs" : "https://www.matatika.com/docs/data-sources/tap-google-analytics/",
"pipUrl" : "git+https://github.com/Matatika/[email protected]",
"repo" : "https://github.com/Matatika/tap-google-analytics",
"capabilities" : [ "catalog", "state", "discover" ],
"select" : [ ],
"update" : { },
"vars" : { },
"settings" : [ {
"name" : "oauth_credentials.authorization_url",
"aliases" : [ ],
"label" : "OAuth identity provider authorization endpoint used create and refresh tokens",
"value" : "https://oauth2.googleapis.com/token",
"kind" : "HIDDEN",
"required" : "true",
"protected" : false
}, {
"name" : "oauth_credentials.scope",
"aliases" : [ ],
"label" : "OAuth scopes we need to request access to",
"value" : "profile email https://www.googleapis.com/auth/analytics.readonly",
"kind" : "HIDDEN",
"required" : "true",
"protected" : false
}, {
"name" : "oauth_credentials.access_token",
"aliases" : [ ],
"label" : "Access Token",
"kind" : "HIDDEN",
"required" : "true",
"protected" : false
}, {
"name" : "oauth_credentials.refresh_token",
"aliases" : [ ],
"label" : "OAuth Refresh Token",
"kind" : "HIDDEN",
"required" : "true",
"protected" : false
}, {
"name" : "oauth_credentials.refresh_proxy_url",
"aliases" : [ ],
"label" : "Optional - will be called with 'oauth_credentials.refresh_token' to refresh the access token",
"kind" : "HIDDEN",
"protected" : false
}, {
"name" : "oauth_credentials.refresh_proxy_url_auth",
"aliases" : [ ],
"label" : "Optional - Sets Authorization header on 'oauth_credentials.refresh_url' request",
"kind" : "HIDDEN",
"protected" : false
}, {
"name" : "oauth_credentials.client_id",
"aliases" : [ ],
"label" : "Optional - OAuth Client ID used if refresh_proxy_url not supplied",
"kind" : "HIDDEN",
"protected" : false
}, {
"name" : "oauth_credentials.client_secret",
"aliases" : [ ],
"label" : "Optional - OAuth Client Secret used if refresh_proxy_url not supplied",
"kind" : "HIDDEN",
"protected" : false
}, {
"name" : "view_id",
"aliases" : [ ],
"label" : "View ID",
"placeholder" : "Ex. 198343027",
"kind" : "STRING",
"required" : "true",
"protected" : false
}, {
"name" : "reports",
"aliases" : [ ],
"label" : "Reports",
"placeholder" : "Ex. my_report_definition.json",
"kind" : "STRING",
"protected" : false
}, {
"name" : "start_date",
"aliases" : [ ],
"kind" : "DATE_ISO8601",
"protected" : false
}, {
"name" : "end_date",
"aliases" : [ ],
"kind" : "DATE_ISO8601",
"protected" : false
} ],
"variants" : [ ],
"commands" : { },
"matatikaHidden" : false,
"_links" : {
"self" : {
"href" : "https://catalog.matatika.com/api/dataplugins/d9e8ffbf-375a-4915-b629-2549c4c4060b"
},
"update dataplugin" : {
"href" : "https://catalog.matatika.com/api/workspaces/d2dad12c-4533-49c3-9d4a-cf03cf7aaaca/dataplugins/d9e8ffbf-375a-4915-b629-2549c4c4060b",
"type" : "PUT"
}
}
}
},
"managed" : false,
"draft" : false,
"_links" : {
"self" : {
"href" : "https://catalog.matatika.com/api/datacomponents/5be95c7e-4ef6-495d-b312-2e89c8cefaba"
},
"update datacomponent" : {
"href" : "https://catalog.matatika.com/api/datacomponents/5be95c7e-4ef6-495d-b312-2e89c8cefaba"
},
"delete datacomponent" : {
"href" : "https://catalog.matatika.com/api/datacomponents/5be95c7e-4ef6-495d-b312-2e89c8cefaba"
}
}
}, {
"id" : "a3de91eb-39a0-4ea8-a6a2-615d5ff721e3",
"created" : "2023-03-23T21:45:24.806929",
"lastModified" : "2023-03-23T21:45:24.80693",
"name" : "Warehouse",
"properties" : {
"password" : "Ptm5kD04_3ngU1Y6E_YeVd43da",
"dbname" : "ugdwwoy",
"default_target_schema" : "analytics",
"port" : "5432",
"host" : "nutty-ugli.db.elephantsql.com",
"user" : "ugdwwoy"
},
"commands" : { },
"dataPlugin" : "loaders/target-postgres--transferwise",
"_embedded" : {
"dataplugin" : {
"id" : "bed88248-835a-468d-a57c-b5b4ac24002c",
"repositoryPath" : "plugins/loaders/target-postgres--transferwise.lock",
"pluginType" : "LOADER",
"name" : "target-postgres",
"namespace" : "postgres_transferwise",
"variant" : "transferwise",
"label" : "Postgres Warehouse",
"description" : "Loader for Postgres Data Warehouse\n\nThe World's Most Advanced Open Source Relational Database\n\n## Settings\n\n### User\nPostgres user\n\n### Password\nPostgres password\n\n### Host\nPostgres host\nExample: `test-host.db.elephantsql.com`\n\n### Port\nPostgres port\nExample: `5432`\n\n### Database Name\nPostgres database name\n\n### Default Target Schema\nName of the schema where tables will be created (no database prefix)",
"logoUrl" : "https://app.matatika.com/assets/logos/loaders/postgres.png",
"hidden" : false,
"pipUrl" : "git+https://github.com/Matatika/[email protected]",
"repo" : "git+https://github.com/Matatika/[email protected]",
"capabilities" : [ ],
"select" : [ ],
"update" : { },
"vars" : { },
"settings" : [ {
"name" : "user",
"aliases" : [ "username" ],
"label" : "User",
"kind" : "STRING",
"description" : "PostgreSQL user",
"required" : "true",
"protected" : false
}, {
"name" : "password",
"aliases" : [ ],
"label" : "Password",
"kind" : "PASSWORD",
"description" : "PostgreSQL password",
"required" : "true",
"protected" : false
}, {
"name" : "host",
"aliases" : [ "address" ],
"label" : "Host",
"kind" : "STRING",
"description" : "PostgreSQL host",
"required" : "true",
"protected" : false
}, {
"name" : "port",
"aliases" : [ ],
"label" : "Port",
"value" : "5432",
"kind" : "INTEGER",
"description" : "PostgreSQL port",
"required" : "true",
"protected" : false
}, {
"name" : "dbname",
"aliases" : [ "database" ],
"label" : "Database Name",
"kind" : "STRING",
"description" : "PostgreSQL database name",
"required" : "true",
"protected" : false
}, {
"name" : "default_target_schema",
"aliases" : [ ],
"label" : "Default Target Schema",
"value" : "analytics",
"kind" : "STRING",
"description" : "Name of the schema where the tables will be created.",
"required" : "true",
"protected" : false
}, {
"name" : "ssl",
"aliases" : [ ],
"label" : "SSL",
"value" : "false",
"kind" : "HIDDEN",
"protected" : false,
"value_post_processor" : "STRINGIFY"
}, {
"name" : "batch_size_rows",
"aliases" : [ ],
"label" : "Batch Size Rows",
"value" : "100000",
"kind" : "INTEGER",
"description" : "Maximum number of rows in each batch. At the end of each batch, the rows in the batch are loaded into Postgres.",
"protected" : false
}, {
"name" : "underscore_camel_case_fields",
"aliases" : [ ],
"label" : "Underscore Camel Case Fields",
"value" : "true",
"kind" : "HIDDEN",
"description" : "Enable underscoring camel case field names, make the resulting postgres column names the same as from the Meltano variant of target-postgres.",
"protected" : false
}, {
"name" : "flush_all_streams",
"aliases" : [ ],
"label" : "Flush All Streams",
"value" : "false",
"kind" : "HIDDEN",
"description" : "Flush and load every stream into Postgres when one batch is full. Warning: This may trigger the COPY command to use files with low number of records.",
"protected" : false
}, {
"name" : "parallelism",
"aliases" : [ ],
"label" : "Parallelism",
"value" : "0",
"kind" : "HIDDEN",
"description" : "The number of threads used to flush tables. 0 will create a thread for each stream, up to parallelism_max. -1 will create a thread for each CPU core. Any other positive number will create that number of threads, up to parallelism_max.",
"protected" : false
}, {
"name" : "parallelism_max",
"aliases" : [ ],
"label" : "Max Parallelism",
"value" : "16",
"kind" : "HIDDEN",
"description" : "Max number of parallel threads to use when flushing tables.",
"protected" : false
}, {
"name" : "default_target_schema_select_permission",
"aliases" : [ ],
"label" : "Default Target Schema Select Permission",
"kind" : "HIDDEN",
"description" : "Grant USAGE privilege on newly created schemas and grant SELECT privilege on newly created tables to a specific role or a list of roles. If `schema_mapping` is not defined then every stream sent by the tap is granted accordingly.",
"protected" : false
}, {
"name" : "schema_mapping",
"aliases" : [ ],
"label" : "Schema Mapping",
"kind" : "HIDDEN",
"description" : "Useful if you want to load multiple streams from one tap to multiple Postgres schemas.\n\nIf the tap sends the `stream_id` in `<schema_name>-<table_name>` format then this option overwrites the `default_target_schema` value.\nNote, that using `schema_mapping` you can overwrite the `default_target_schema_select_permission` value to grant SELECT permissions to different groups per schemas or optionally you can create indices automatically for the replicated tables.\n",
"protected" : false
}, {
"name" : "add_metadata_columns",
"aliases" : [ ],
"label" : "Add Metadata Columns",
"value" : "true",
"kind" : "HIDDEN",
"description" : "Metadata columns add extra row level information about data ingestions, (i.e. when was the row read in source, when was inserted or deleted in postgres etc.) Metadata columns are creating automatically by adding extra columns to the tables with a column prefix `_SDC_`. The column names are following the stitch naming conventions documented at https://www.stitchdata.com/docs/data-structure/integration-schemas#sdc-columns. Enabling metadata columns will flag the deleted rows by setting the `_SDC_DELETED_AT` metadata column. Without the `add_metadata_columns` option the deleted rows from singer taps will not be recongisable in Postgres.",
"protected" : false
}, {
"name" : "hard_delete",
"aliases" : [ ],
"label" : "Hard Delete",
"value" : "false",
"kind" : "HIDDEN",
"description" : "When `hard_delete` option is true then DELETE SQL commands will be performed in Postgres to delete rows in tables. It's achieved by continuously checking the `_SDC_DELETED_AT` metadata column sent by the singer tap. Due to deleting rows requires metadata columns, `hard_delete` option automatically enables the `add_metadata_columns` option as well.",
"protected" : false
}, {
"name" : "data_flattening_max_level",
"aliases" : [ ],
"label" : "Data Flattening Max Level",
"value" : "10",
"kind" : "HIDDEN",
"description" : "Object type RECORD items from taps can be transformed to flattened columns by creating columns automatically. When value is 0 (default) then flattening functionality is turned off.",
"protected" : false
}, {
"name" : "primary_key_required",
"aliases" : [ ],
"label" : "Primary Key Required",
"value" : "false",
"kind" : "BOOLEAN",
"description" : "Log based and Incremental replications on tables with no Primary Key cause duplicates when merging UPDATE events. When set to true, stop loading data if no Primary Key is defined.",
"protected" : false
}, {
"name" : "validate_records",
"aliases" : [ ],
"label" : "Validate Records",
"value" : "false",
"kind" : "BOOLEAN",
"description" : "Validate every single record message to the corresponding JSON schema. This option is disabled by default and invalid RECORD messages will fail only at load time by Postgres. Enabling this option will detect invalid records earlier but could cause performance degradation.",
"protected" : false
}, {
"name" : "temp_dir",
"aliases" : [ ],
"label" : "Temporary Directory",
"kind" : "HIDDEN",
"description" : "(Default: platform-dependent) Directory of temporary CSV files with RECORD messages.",
"protected" : false
} ],
"variants" : [ ],
"commands" : { },
"matatikaHidden" : false,
"_links" : {
"self" : {
"href" : "https://catalog.matatika.com/api/dataplugins/bed88248-835a-468d-a57c-b5b4ac24002c"
},
"update dataplugin" : {
"href" : "https://catalog.matatika.com/api/workspaces/d2dad12c-4533-49c3-9d4a-cf03cf7aaaca/dataplugins/bed88248-835a-468d-a57c-b5b4ac24002c",
"type" : "PUT"
}
}
}
},
"managed" : true,
"draft" : false,
"_links" : {
"self" : {
"href" : "https://catalog.matatika.com/api/datacomponents/a3de91eb-39a0-4ea8-a6a2-615d5ff721e3"
},
"update datacomponent" : {
"href" : "https://catalog.matatika.com/api/datacomponents/a3de91eb-39a0-4ea8-a6a2-615d5ff721e3"
},
"delete datacomponent" : {
"href" : "https://catalog.matatika.com/api/datacomponents/a3de91eb-39a0-4ea8-a6a2-615d5ff721e3"
}
}
}, {
"id" : "166fc471-51cc-449f-b9cc-58b890eea347",
"created" : "2023-03-23T21:46:46.377154",
"lastModified" : "2023-03-23T21:46:46.377169",
"name" : "dbt",
"properties" : { },
"commands" : {
"compile" : {
"args" : "compile",
"description" : "Generates executable SQL from source model, test, and analysis files. Compiled SQL files are written to the target/ directory."
},
"seed" : {
"args" : "seed",
"description" : "Load data from csv files into your data warehouse."
},
"test" : {
"args" : "test",
"description" : "Runs tests on data in deployed models."
},
"deps" : {
"args" : "deps",
"description" : "Pull the most recent version of the dependencies listed in packages.yml"
},
"run" : {
"args" : "run",
"description" : "Compile SQL and execute against the current target database."
},
"clean" : {
"args" : "clean",
"description" : "Delete all folders in the clean-targets list (usually the dbt_modules and target directories.)"
},
"snapshot" : {
"args" : "snapshot",
"description" : "Execute snapshots defined in your project."
}
},
"dataPlugin" : "transformers/dbt--dbt-labs",
"_embedded" : {
"dataplugin" : {
"id" : "35a04646-0982-473d-9632-4d6bcc18d7cf",
"repositoryPath" : "plugins/transformers/dbt--dbt-labs.lock",
"pluginType" : "TRANSFORMER",
"name" : "dbt",
"namespace" : "dbt",
"variant" : "dbt-labs",
"label" : "dbt",
"logoUrl" : "https://app.matatika.com/assets/images/transformer/dbt.png",
"hidden" : false,
"docs" : "https://meltano.com/docs/transforms.html",
"pipUrl" : "dbt-core~=1.3.0 dbt-postgres~=1.3.0 dbt-snowflake~=1.3.0\n",
"repo" : "https://github.com/dbt-labs/dbt-core",
"capabilities" : [ ],
"select" : [ ],
"update" : { },
"vars" : { },
"settings" : [ {
"name" : "project_dir",
"aliases" : [ ],
"value" : "$MELTANO_PROJECT_ROOT/transform",
"kind" : "STRING",
"protected" : false
}, {
"name" : "profiles_dir",
"aliases" : [ ],
"value" : "$MELTANO_PROJECT_ROOT/transform/profile",
"kind" : "STRING",
"env" : "DBT_PROFILES_DIR",
"protected" : false
}, {
"name" : "target",
"aliases" : [ ],
"value" : "$MELTANO_LOAD__DIALECT",
"kind" : "STRING",
"protected" : false
}, {
"name" : "source_schema",
"aliases" : [ ],
"value" : "$MELTANO_LOAD__TARGET_SCHEMA",
"kind" : "STRING",
"protected" : false
}, {
"name" : "target_schema",
"aliases" : [ ],
"value" : "analytics",
"kind" : "STRING",
"protected" : false
}, {
"name" : "models",
"aliases" : [ ],
"value" : "$MELTANO_TRANSFORM__PACKAGE_NAME $MELTANO_EXTRACTOR_NAMESPACE my_meltano_project",
"kind" : "STRING",
"protected" : false
} ],
"variants" : [ ],
"commands" : {
"compile" : {
"args" : "compile",
"description" : "Generates executable SQL from source model, test, and analysis files. Compiled SQL files are written to the target/ directory."
},
"seed" : {
"args" : "seed",
"description" : "Load data from csv files into your data warehouse."
},
"test" : {
"args" : "test",
"description" : "Runs tests on data in deployed models."
},
"deps" : {
"args" : "deps",
"description" : "Pull the most recent version of the dependencies listed in packages.yml"
},
"run" : {
"args" : "run",
"description" : "Compile SQL and execute against the current target database."
},
"clean" : {
"args" : "clean",
"description" : "Delete all folders in the clean-targets list (usually the dbt_modules and target directories.)"
},
"snapshot" : {
"args" : "snapshot",
"description" : "Execute snapshots defined in your project."
}
},
"matatikaHidden" : false,
"_links" : {
"self" : {
"href" : "https://catalog.matatika.com/api/dataplugins/35a04646-0982-473d-9632-4d6bcc18d7cf"
},
"update dataplugin" : {
"href" : "https://catalog.matatika.com/api/workspaces/d2dad12c-4533-49c3-9d4a-cf03cf7aaaca/dataplugins/35a04646-0982-473d-9632-4d6bcc18d7cf",
"type" : "PUT"
}
}
}
},
"managed" : false,
"draft" : false,
"_links" : {
"self" : {
"href" : "https://catalog.matatika.com/api/datacomponents/166fc471-51cc-449f-b9cc-58b890eea347"
},
"update datacomponent" : {
"href" : "https://catalog.matatika.com/api/datacomponents/166fc471-51cc-449f-b9cc-58b890eea347"
},
"delete datacomponent" : {
"href" : "https://catalog.matatika.com/api/datacomponents/166fc471-51cc-449f-b9cc-58b890eea347"
}
}
} ],
"latest job" : {
"id" : "9cd1a03b-7749-42f0-a975-1364feeb0b45",
"created" : "2023-03-23T21:46:50.998498",
"type" : "WORKSPACE_CONFIG",
"exitCode" : 0,
"status" : "COMPLETE",
"startTime" : "2023-03-23T21:47:00.734",
"endTime" : "2023-03-23T21:47:14.715",
"_embedded" : {
"pipeline" : {
"id" : "7437c296-93ad-470c-b14d-24afc965c711",
"status" : "READY",
"name" : "SIT-generated pipeline [2023-03-23T21:46:50.230490] (updated)",
"timeout" : 0,
"created" : "2023-03-23T21:46:50.593501",
"lastModified" : "2023-03-23T21:46:50.593502",
"properties" : {
"tap-google-analytics.view_id" : "1234567890",
"tap-google-analytics.oauth_credentials.client_secret" : "client_secret",
"tap-google-analytics.reports" : "reports",
"tap-google-analytics.oauth_credentials.refresh_token" : "refresh_token",
"tap-google-analytics.start_date" : "2023-02-23T21:45:53.021107Z",
"tap-google-analytics.end_date" : "2023-03-23T21:45:53.021218Z",
"tap-google-analytics.oauth_credentials.client_id" : "client_id",
"tap-google-analytics.oauth_credentials.access_token" : "access_token"
},
"dataComponents" : [ "tap-google-analytics", "Warehouse", "dbt" ],
"actions" : [ ]
}
},
"_links" : {
"self" : {
"href" : "https://catalog.matatika.com/api/jobs/9cd1a03b-7749-42f0-a975-1364feeb0b45"
},
"delete job" : {
"href" : "https://catalog.matatika.com/api/jobs/9cd1a03b-7749-42f0-a975-1364feeb0b45",
"type" : "DELETE"
},
"logs" : {
"href" : "https://catalog.matatika.com/api/jobs/9cd1a03b-7749-42f0-a975-1364feeb0b45/logs?sequence=0",
"type" : "GET"
}
}
}
},
"_links" : {
"update pipeline" : {
"href" : "https://catalog.matatika.com/api/workspaces/d2dad12c-4533-49c3-9d4a-cf03cf7aaaca/pipelines/7437c296-93ad-470c-b14d-24afc965c711",
"type" : "PUT"
},
"delete pipeline" : {
"href" : "https://catalog.matatika.com/api/pipelines/7437c296-93ad-470c-b14d-24afc965c711",
"type" : "DELETE"
},
"draft pipeline" : {
"href" : "https://catalog.matatika.com/api/workspaces/d2dad12c-4533-49c3-9d4a-cf03cf7aaaca/pipelines/7437c296-93ad-470c-b14d-24afc965c711/draft",
"type" : "PUT"
},
"self" : {
"href" : "https://catalog.matatika.com/api/pipelines/7437c296-93ad-470c-b14d-24afc965c711"
},
"environment" : {
"href" : "https://catalog.matatika.com/api/pipelines/7437c296-93ad-470c-b14d-24afc965c711/environment"
},
"jobs" : {
"href" : "https://catalog.matatika.com/api/pipelines/7437c296-93ad-470c-b14d-24afc965c711/jobs",
"type" : "GET"
},
"add subscription" : {
"href" : "https://catalog.matatika.com/api/pipelines/7437c296-93ad-470c-b14d-24afc965c711/subscriptions"
},
"verify pipeline" : {
"href" : "https://catalog.matatika.com/api/pipelines/7437c296-93ad-470c-b14d-24afc965c711/verification",
"type" : "POST"
},
"create job" : {
"href" : "https://catalog.matatika.com/api/pipelines/7437c296-93ad-470c-b14d-24afc965c711/jobs",
"type" : "POST"
},
"latest job" : {
"href" : "https://catalog.matatika.com/api/jobs/9cd1a03b-7749-42f0-a975-1364feeb0b45"
}
}
}
Initialise a pipeline in a workspace
POST
/api/workspaces/{workspace-id}/pipelines
Initialises a new pipeline in the workspace {workspace-id}
.
Prerequisites
- Workspace
{workspace-id}
must exist
Request
Example Snippets
- cURL
curl -H "Authorization: Bearer $ACCESS_TOKEN" 'https://catalog.matatika.com:443/api/workspaces/d2dad12c-4533-49c3-9d4a-cf03cf7aaaca/pipelines' -i -X POST \
-H 'Accept: application/json, application/javascript, text/javascript, text/json' \
-H 'Content-Type: application/json'
- Python (requests)
import requests
url = "https://catalog.matatika.com:443/api/workspaces/d2dad12c-4533-49c3-9d4a-cf03cf7aaaca/pipelines"
headers = {
'Authorization': ACCESS_TOKEN
}
response = requests.request("POST", url, headers=headers)
print(response.text.encode('utf8'))
Response
200 OK
Pipeline with HAL links.
{
"id" : "48acbb57-1837-47d0-a230-ebb041a0fb39",
"status" : "PROVISIONING",
"timeout" : 0,
"created" : "2023-03-23T21:45:51.777087835",
"lastModified" : "2023-03-23T21:45:51.777088035",
"properties" : { },
"dataComponents" : [ ],
"actions" : [ ],
"_links" : {
"create pipeline" : {
"href" : "https://catalog.matatika.com/api/workspaces/d2dad12c-4533-49c3-9d4a-cf03cf7aaaca/pipelines/48acbb57-1837-47d0-a230-ebb041a0fb39",
"type" : "PUT"
},
"draft pipeline" : {
"href" : "https://catalog.matatika.com/api/workspaces/d2dad12c-4533-49c3-9d4a-cf03cf7aaaca/pipelines/48acbb57-1837-47d0-a230-ebb041a0fb39/draft",
"type" : "PUT"
},
"validate pipeline" : {
"href" : "https://catalog.matatika.com/api/workspaces/d2dad12c-4533-49c3-9d4a-cf03cf7aaaca/pipelines/validation",
"type" : "POST"
}
}
}
Create or update a pipeline in a workspace
PUT
/api/workspaces/{workspace-id}/pipelines/{pipeline-id}
Creates or updates the pipeline {pipeline-id}
in the workspace {workspace-id}
.
Prerequisites
- Workspace
{workspace-id}
must exist
Request
Body
Pipeline resource.
{
"name" : "SIT-generated pipeline [2023-03-23T21:45:52.793220]",
"dataComponents" : [ "tap-google-analytics", "Warehouse", "dbt" ],
"schedule" : "0 0 0 25 12 ?",
"properties" : {
"tap-google-analytics.view_id" : "1234567890",
"tap-google-analytics.oauth_credentials.client_secret" : "client_secret",
"tap-google-analytics.reports" : "reports",
"tap-google-analytics.oauth_credentials.refresh_token" : "refresh_token",
"tap-google-analytics.start_date" : "2023-02-23T21:45:53.021107Z",
"tap-google-analytics.end_date" : "2023-03-23T21:45:53.021218Z",
"tap-google-analytics.oauth_credentials.client_id" : "client_id",
"tap-google-analytics.oauth_credentials.access_token" : "access_token"
}
}
Example Snippets
- cURL
curl -H "Authorization: Bearer $ACCESS_TOKEN" 'https://catalog.matatika.com:443/api/workspaces/d2dad12c-4533-49c3-9d4a-cf03cf7aaaca/pipelines/48acbb57-1837-47d0-a230-ebb041a0fb39' -i -X PUT \
-H 'Accept: application/json, application/javascript, text/javascript, text/json' \
-H 'Content-Type: application/json' \
-d '{
"name" : "SIT-generated pipeline [2023-03-23T21:45:52.793220]",
"dataComponents" : [ "tap-google-analytics", "Warehouse", "dbt" ],
"schedule" : "0 0 0 25 12 ?",
"properties" : {
"tap-google-analytics.view_id" : "1234567890",
"tap-google-analytics.oauth_credentials.client_secret" : "client_secret",
"tap-google-analytics.reports" : "reports",
"tap-google-analytics.oauth_credentials.refresh_token" : "refresh_token",
"tap-google-analytics.start_date" : "2023-02-23T21:45:53.021107Z",
"tap-google-analytics.end_date" : "2023-03-23T21:45:53.021218Z",
"tap-google-analytics.oauth_credentials.client_id" : "client_id",
"tap-google-analytics.oauth_credentials.access_token" : "access_token"
}
}'
- Python (requests)
import requests
url = "https://catalog.matatika.com:443/api/workspaces/d2dad12c-4533-49c3-9d4a-cf03cf7aaaca/pipelines/48acbb57-1837-47d0-a230-ebb041a0fb39"
data = {
"name" : "SIT-generated pipeline [2023-03-23T21:45:52.793220]",
"dataComponents" : [ "tap-google-analytics", "Warehouse", "dbt" ],
"schedule" : "0 0 0 25 12 ?",
"properties" : {
"tap-google-analytics.view_id" : "1234567890",
"tap-google-analytics.oauth_credentials.client_secret" : "client_secret",
"tap-google-analytics.reports" : "reports",
"tap-google-analytics.oauth_credentials.refresh_token" : "refresh_token",
"tap-google-analytics.start_date" : "2023-02-23T21:45:53.021107Z",
"tap-google-analytics.end_date" : "2023-03-23T21:45:53.021218Z",
"tap-google-analytics.oauth_credentials.client_id" : "client_id",
"tap-google-analytics.oauth_credentials.access_token" : "access_token"
}
}
headers = {
'Authorization': ACCESS_TOKEN
}
response = requests.request("PUT", url, headers=headers, data=data)
print(response.text.encode('utf8'))
Response
200 OK / 201 Created
Pipeline with HAL links.
{
"id" : "48acbb57-1837-47d0-a230-ebb041a0fb39",
"status" : "PROVISIONING",
"name" : "SIT-generated pipeline [2023-03-23T21:45:52.793220]",
"schedule" : "0 0 0 25 12 ?",
"timeout" : 0,
"created" : "2023-03-23T21:45:53.448327",
"lastModified" : "2023-03-23T21:45:53.448328",
"properties" : {
"tap-google-analytics.view_id" : "1234567890",
"tap-google-analytics.oauth_credentials.client_secret" : "client_secret",
"tap-google-analytics.reports" : "reports",
"tap-google-analytics.oauth_credentials.refresh_token" : "refresh_token",
"tap-google-analytics.start_date" : "2023-02-23T21:45:53.021107Z",
"tap-google-analytics.end_date" : "2023-03-23T21:45:53.021218Z",
"tap-google-analytics.oauth_credentials.client_id" : "client_id",
"tap-google-analytics.oauth_credentials.access_token" : "access_token"
},
"dataComponents" : [ "tap-google-analytics", "Warehouse", "dbt" ],
"actions" : [ ],
"_embedded" : {
"dataComponents" : [ {
"id" : "5be95c7e-4ef6-495d-b312-2e89c8cefaba",
"created" : "2023-03-23T21:45:52.26399",
"lastModified" : "2023-03-23T21:45:52.263991",
"name" : "tap-google-analytics",
"properties" : { },
"commands" : { },
"dataPlugin" : "extractors/tap-google-analytics--matatika",
"_embedded" : {
"dataplugin" : {
"id" : "d9e8ffbf-375a-4915-b629-2549c4c4060b",
"pluginType" : "EXTRACTOR",
"name" : "tap-google-analytics",
"namespace" : "tap_google_analytics",
"variant" : "matatika",
"label" : "Google Analytics",
"description" : "App and website analytics platform hosted by Google\n\n## Prerequisites\n\nBefore importing data from Google Analytics you or your webmaster will need to have performed some initial setup to grant\nyourself access to the Google Analytics for your website.\n\n[Google Analytics Prerequisites](https://www.matatika.com/docs/data-sources/tap-google-analytics/prerequisites)\n\n## Data source settings\n### View ID\n\nOnce you have connected with your google account, the `View ID` drop down will allow you to choose which view you want to get your data from.\n\n### Start Date\n\nProviding a start date allows you to set how much historical data you wish to import into you workspace.\n\nThis date has to be in the ISO-8601 date format, see the example below.\n\nFormat: `YYYY-MM-DDTHH:MM:SSZ`\n\nExample: `2020-10-01T13:12:11Z`\n\n### End Date\n\nProviding an end date allows you to limit how much data you wish to import into you workspace.\n\nThis date has to be in the ISO-8601 date format, see the example below.\n\nFormat: `YYYY-MM-DDTHH:MM:SSZ`\n\nExample: `2020-10-01T13:12:11Z`\n\n### Reports\n\nProviding the path to a reports json file allows you to supply the tap with a user defined definition of tables and information you want to sync from Google Analytics.\nThis does require you to commit a file to your workspace repository so that Google Analytics can find it during run time, and may also break the dbt transforms that are required to have our included datasets work.\nFor more information on how to define a custom reports file see the [Meltano Hub Documentation](https://hub.meltano.com/extractors/google-analytics#reports).\n\n## Learn more\n\n[Matatika Google Analytics Data Source Documentation](https://www.matatika.com/docs/data-sources/tap-google-analytics)\n\n[Meltano Hub Google Analytics Documentation](https://hub.meltano.com/extractors/google-analytics)",
"logoUrl" : "/assets/images/datasource/tap-google-analytics.svg",
"hidden" : false,
"docs" : "https://www.matatika.com/docs/data-sources/tap-google-analytics/",
"pipUrl" : "git+https://github.com/Matatika/[email protected]",
"repo" : "https://github.com/Matatika/tap-google-analytics",
"capabilities" : [ "catalog", "state", "discover" ],
"select" : [ ],
"update" : { },
"vars" : { },
"settings" : [ {
"name" : "oauth_credentials.authorization_url",
"aliases" : [ ],
"label" : "OAuth identity provider authorization endpoint used create and refresh tokens",
"value" : "https://oauth2.googleapis.com/token",
"kind" : "HIDDEN",
"required" : "true",
"protected" : false
}, {
"name" : "oauth_credentials.scope",
"aliases" : [ ],
"label" : "OAuth scopes we need to request access to",
"value" : "profile email https://www.googleapis.com/auth/analytics.readonly",
"kind" : "HIDDEN",
"required" : "true",
"protected" : false
}, {
"name" : "oauth_credentials.access_token",
"aliases" : [ ],
"label" : "Access Token",
"kind" : "HIDDEN",
"required" : "true",
"protected" : false
}, {
"name" : "oauth_credentials.refresh_token",
"aliases" : [ ],
"label" : "OAuth Refresh Token",
"kind" : "HIDDEN",
"required" : "true",
"protected" : false
}, {
"name" : "oauth_credentials.refresh_proxy_url",
"aliases" : [ ],
"label" : "Optional - will be called with 'oauth_credentials.refresh_token' to refresh the access token",
"kind" : "HIDDEN",
"protected" : false
}, {
"name" : "oauth_credentials.refresh_proxy_url_auth",
"aliases" : [ ],
"label" : "Optional - Sets Authorization header on 'oauth_credentials.refresh_url' request",
"kind" : "HIDDEN",
"protected" : false
}, {
"name" : "oauth_credentials.client_id",
"aliases" : [ ],
"label" : "Optional - OAuth Client ID used if refresh_proxy_url not supplied",
"kind" : "HIDDEN",
"protected" : false
}, {
"name" : "oauth_credentials.client_secret",
"aliases" : [ ],
"label" : "Optional - OAuth Client Secret used if refresh_proxy_url not supplied",
"kind" : "HIDDEN",
"protected" : false
}, {
"name" : "view_id",
"aliases" : [ ],
"label" : "View ID",
"placeholder" : "Ex. 198343027",
"kind" : "STRING",
"required" : "true",
"protected" : false
}, {
"name" : "reports",
"aliases" : [ ],
"label" : "Reports",
"placeholder" : "Ex. my_report_definition.json",
"kind" : "STRING",
"protected" : false
}, {
"name" : "start_date",
"aliases" : [ ],
"kind" : "DATE_ISO8601",
"protected" : false
}, {
"name" : "end_date",
"aliases" : [ ],
"kind" : "DATE_ISO8601",
"protected" : false
} ],
"variants" : [ ],
"commands" : { },
"matatikaHidden" : false,
"_links" : {
"self" : {
"href" : "https://catalog.matatika.com/api/dataplugins/d9e8ffbf-375a-4915-b629-2549c4c4060b"
},
"update dataplugin" : {
"href" : "https://catalog.matatika.com/api/workspaces/d2dad12c-4533-49c3-9d4a-cf03cf7aaaca/dataplugins/d9e8ffbf-375a-4915-b629-2549c4c4060b",
"type" : "PUT"
}
}
}
},
"managed" : false,
"draft" : false,
"_links" : {
"self" : {
"href" : "https://catalog.matatika.com/api/datacomponents/5be95c7e-4ef6-495d-b312-2e89c8cefaba"
},
"update datacomponent" : {
"href" : "https://catalog.matatika.com/api/datacomponents/5be95c7e-4ef6-495d-b312-2e89c8cefaba"
},
"delete datacomponent" : {
"href" : "https://catalog.matatika.com/api/datacomponents/5be95c7e-4ef6-495d-b312-2e89c8cefaba"
}
}
}, {
"id" : "a3de91eb-39a0-4ea8-a6a2-615d5ff721e3",
"created" : "2023-03-23T21:45:24.806929",
"lastModified" : "2023-03-23T21:45:24.80693",
"name" : "Warehouse",
"properties" : {
"password" : "Ptm5kD04_3ngU1Y6E_YeVd43da",
"dbname" : "ugdwwoy",
"default_target_schema" : "analytics",
"port" : "5432",
"host" : "nutty-ugli.db.elephantsql.com",
"user" : "ugdwwoy"
},
"commands" : { },
"dataPlugin" : "loaders/target-postgres--transferwise",
"_embedded" : {
"dataplugin" : {
"id" : "bed88248-835a-468d-a57c-b5b4ac24002c",
"pluginType" : "LOADER",
"name" : "target-postgres",
"namespace" : "postgres_transferwise",
"variant" : "transferwise",
"label" : "Postgres Warehouse",
"description" : "Loader for Postgres Data Warehouse\n\nThe World's Most Advanced Open Source Relational Database\n\n## Settings\n\n### User\nPostgres user\n\n### Password\nPostgres password\n\n### Host\nPostgres host\nExample: `test-host.db.elephantsql.com`\n\n### Port\nPostgres port\nExample: `5432`\n\n### Database Name\nPostgres database name\n\n### Default Target Schema\nName of the schema where tables will be created (no database prefix)",
"logoUrl" : "/assets/logos/loaders/postgres.png",
"hidden" : false,
"pipUrl" : "git+https://github.com/Matatika/[email protected]",
"repo" : "git+https://github.com/Matatika/[email protected]",
"capabilities" : [ ],
"select" : [ ],
"update" : { },
"vars" : { },
"settings" : [ {
"name" : "user",
"aliases" : [ "username" ],
"label" : "User",
"kind" : "STRING",
"description" : "PostgreSQL user",
"required" : "true",
"protected" : false
}, {
"name" : "password",
"aliases" : [ ],
"label" : "Password",
"kind" : "PASSWORD",
"description" : "PostgreSQL password",
"required" : "true",
"protected" : false
}, {
"name" : "host",
"aliases" : [ "address" ],
"label" : "Host",
"kind" : "STRING",
"description" : "PostgreSQL host",
"required" : "true",
"protected" : false
}, {
"name" : "port",
"aliases" : [ ],
"label" : "Port",
"value" : "5432",
"kind" : "INTEGER",
"description" : "PostgreSQL port",
"required" : "true",
"protected" : false
}, {
"name" : "dbname",
"aliases" : [ "database" ],
"label" : "Database Name",
"kind" : "STRING",
"description" : "PostgreSQL database name",
"required" : "true",
"protected" : false
}, {
"name" : "default_target_schema",
"aliases" : [ ],
"label" : "Default Target Schema",
"value" : "analytics",
"kind" : "STRING",
"description" : "Name of the schema where the tables will be created.",
"required" : "true",
"protected" : false
}, {
"name" : "ssl",
"aliases" : [ ],
"label" : "SSL",
"value" : "false",
"kind" : "HIDDEN",
"protected" : false,
"value_post_processor" : "STRINGIFY"
}, {
"name" : "batch_size_rows",
"aliases" : [ ],
"label" : "Batch Size Rows",
"value" : "100000",
"kind" : "INTEGER",
"description" : "Maximum number of rows in each batch. At the end of each batch, the rows in the batch are loaded into Postgres.",
"protected" : false
}, {
"name" : "underscore_camel_case_fields",
"aliases" : [ ],
"label" : "Underscore Camel Case Fields",
"value" : "true",
"kind" : "HIDDEN",
"description" : "Enable underscoring camel case field names, make the resulting postgres column names the same as from the Meltano variant of target-postgres.",
"protected" : false
}, {
"name" : "flush_all_streams",
"aliases" : [ ],
"label" : "Flush All Streams",
"value" : "false",
"kind" : "HIDDEN",
"description" : "Flush and load every stream into Postgres when one batch is full. Warning: This may trigger the COPY command to use files with low number of records.",
"protected" : false
}, {
"name" : "parallelism",
"aliases" : [ ],
"label" : "Parallelism",
"value" : "0",
"kind" : "HIDDEN",
"description" : "The number of threads used to flush tables. 0 will create a thread for each stream, up to parallelism_max. -1 will create a thread for each CPU core. Any other positive number will create that number of threads, up to parallelism_max.",
"protected" : false
}, {
"name" : "parallelism_max",
"aliases" : [ ],
"label" : "Max Parallelism",
"value" : "16",
"kind" : "HIDDEN",
"description" : "Max number of parallel threads to use when flushing tables.",
"protected" : false
}, {
"name" : "default_target_schema_select_permission",
"aliases" : [ ],
"label" : "Default Target Schema Select Permission",
"kind" : "HIDDEN",
"description" : "Grant USAGE privilege on newly created schemas and grant SELECT privilege on newly created tables to a specific role or a list of roles. If `schema_mapping` is not defined then every stream sent by the tap is granted accordingly.",
"protected" : false
}, {
"name" : "schema_mapping",
"aliases" : [ ],
"label" : "Schema Mapping",
"kind" : "HIDDEN",
"description" : "Useful if you want to load multiple streams from one tap to multiple Postgres schemas.\n\nIf the tap sends the `stream_id` in `<schema_name>-<table_name>` format then this option overwrites the `default_target_schema` value.\nNote, that using `schema_mapping` you can overwrite the `default_target_schema_select_permission` value to grant SELECT permissions to different groups per schemas or optionally you can create indices automatically for the replicated tables.\n",
"protected" : false
}, {
"name" : "add_metadata_columns",
"aliases" : [ ],
"label" : "Add Metadata Columns",
"value" : "true",
"kind" : "HIDDEN",
"description" : "Metadata columns add extra row level information about data ingestions, (i.e. when was the row read in source, when was inserted or deleted in postgres etc.) Metadata columns are creating automatically by adding extra columns to the tables with a column prefix `_SDC_`. The column names are following the stitch naming conventions documented at https://www.stitchdata.com/docs/data-structure/integration-schemas#sdc-columns. Enabling metadata columns will flag the deleted rows by setting the `_SDC_DELETED_AT` metadata column. Without the `add_metadata_columns` option the deleted rows from singer taps will not be recongisable in Postgres.",
"protected" : false
}, {
"name" : "hard_delete",
"aliases" : [ ],
"label" : "Hard Delete",
"value" : "false",
"kind" : "HIDDEN",
"description" : "When `hard_delete` option is true then DELETE SQL commands will be performed in Postgres to delete rows in tables. It's achieved by continuously checking the `_SDC_DELETED_AT` metadata column sent by the singer tap. Due to deleting rows requires metadata columns, `hard_delete` option automatically enables the `add_metadata_columns` option as well.",
"protected" : false
}, {
"name" : "data_flattening_max_level",
"aliases" : [ ],
"label" : "Data Flattening Max Level",
"value" : "10",
"kind" : "HIDDEN",
"description" : "Object type RECORD items from taps can be transformed to flattened columns by creating columns automatically. When value is 0 (default) then flattening functionality is turned off.",
"protected" : false
}, {
"name" : "primary_key_required",
"aliases" : [ ],
"label" : "Primary Key Required",
"value" : "false",
"kind" : "BOOLEAN",
"description" : "Log based and Incremental replications on tables with no Primary Key cause duplicates when merging UPDATE events. When set to true, stop loading data if no Primary Key is defined.",
"protected" : false
}, {
"name" : "validate_records",
"aliases" : [ ],
"label" : "Validate Records",
"value" : "false",
"kind" : "BOOLEAN",
"description" : "Validate every single record message to the corresponding JSON schema. This option is disabled by default and invalid RECORD messages will fail only at load time by Postgres. Enabling this option will detect invalid records earlier but could cause performance degradation.",
"protected" : false
}, {
"name" : "temp_dir",
"aliases" : [ ],
"label" : "Temporary Directory",
"kind" : "HIDDEN",
"description" : "(Default: platform-dependent) Directory of temporary CSV files with RECORD messages.",
"protected" : false
} ],
"variants" : [ ],
"commands" : { },
"matatikaHidden" : false,
"_links" : {
"self" : {
"href" : "https://catalog.matatika.com/api/dataplugins/bed88248-835a-468d-a57c-b5b4ac24002c"
},
"update dataplugin" : {
"href" : "https://catalog.matatika.com/api/workspaces/d2dad12c-4533-49c3-9d4a-cf03cf7aaaca/dataplugins/bed88248-835a-468d-a57c-b5b4ac24002c",
"type" : "PUT"
}
}
}
},
"managed" : true,
"draft" : false,
"_links" : {
"self" : {
"href" : "https://catalog.matatika.com/api/datacomponents/a3de91eb-39a0-4ea8-a6a2-615d5ff721e3"
},
"update datacomponent" : {
"href" : "https://catalog.matatika.com/api/datacomponents/a3de91eb-39a0-4ea8-a6a2-615d5ff721e3"
},
"delete datacomponent" : {
"href" : "https://catalog.matatika.com/api/datacomponents/a3de91eb-39a0-4ea8-a6a2-615d5ff721e3"
}
}
}, {
"id" : "f7092d4c-9f85-4bf5-b81c-d32bd69ba30c",
"created" : "2023-03-23T21:45:24.926621",
"lastModified" : "2023-03-23T21:45:24.926622",
"name" : "dbt",
"properties" : { },
"commands" : {
"compile" : {
"args" : "compile",
"description" : "Generates executable SQL from source model, test, and analysis files. Compiled SQL files are written to the target/ directory."
},
"seed" : {
"args" : "seed",
"description" : "Load data from csv files into your data warehouse."
},
"test" : {
"args" : "test",
"description" : "Runs tests on data in deployed models."
},
"deps" : {
"args" : "deps",
"description" : "Pull the most recent version of the dependencies listed in packages.yml"
},
"run" : {
"args" : "run",
"description" : "Compile SQL and execute against the current target database."
},
"clean" : {
"args" : "clean",
"description" : "Delete all folders in the clean-targets list (usually the dbt_modules and target directories.)"
},
"snapshot" : {
"args" : "snapshot",
"description" : "Execute snapshots defined in your project."
}
},
"dataPlugin" : "transformers/dbt--dbt-labs",
"_embedded" : {
"dataplugin" : {
"id" : "35a04646-0982-473d-9632-4d6bcc18d7cf",
"pluginType" : "TRANSFORMER",
"name" : "dbt",
"namespace" : "dbt",
"variant" : "dbt-labs",
"label" : "dbt",
"logoUrl" : "/assets/images/transformer/dbt.png",
"hidden" : false,
"docs" : "https://meltano.com/docs/transforms.html",
"pipUrl" : "dbt-core~=1.3.0 dbt-postgres~=1.3.0 dbt-snowflake~=1.3.0\n",
"repo" : "https://github.com/dbt-labs/dbt-core",
"capabilities" : [ ],
"select" : [ ],
"update" : { },
"vars" : { },
"settings" : [ {
"name" : "project_dir",
"aliases" : [ ],
"value" : "$MELTANO_PROJECT_ROOT/transform",
"kind" : "STRING",
"protected" : false
}, {
"name" : "profiles_dir",
"aliases" : [ ],
"value" : "$MELTANO_PROJECT_ROOT/transform/profile",
"kind" : "STRING",
"env" : "DBT_PROFILES_DIR",
"protected" : false
}, {
"name" : "target",
"aliases" : [ ],
"value" : "$MELTANO_LOAD__DIALECT",
"kind" : "STRING",
"protected" : false
}, {
"name" : "source_schema",
"aliases" : [ ],
"value" : "$MELTANO_LOAD__TARGET_SCHEMA",
"kind" : "STRING",
"protected" : false
}, {
"name" : "target_schema",
"aliases" : [ ],
"value" : "analytics",
"kind" : "STRING",
"protected" : false
}, {
"name" : "models",
"aliases" : [ ],
"value" : "$MELTANO_TRANSFORM__PACKAGE_NAME $MELTANO_EXTRACTOR_NAMESPACE my_meltano_project",
"kind" : "STRING",
"protected" : false
} ],
"variants" : [ ],
"commands" : {
"compile" : {
"args" : "compile",
"description" : "Generates executable SQL from source model, test, and analysis files. Compiled SQL files are written to the target/ directory."
},
"seed" : {
"args" : "seed",
"description" : "Load data from csv files into your data warehouse."
},
"test" : {
"args" : "test",
"description" : "Runs tests on data in deployed models."
},
"deps" : {
"args" : "deps",
"description" : "Pull the most recent version of the dependencies listed in packages.yml"
},
"run" : {
"args" : "run",
"description" : "Compile SQL and execute against the current target database."
},
"clean" : {
"args" : "clean",
"description" : "Delete all folders in the clean-targets list (usually the dbt_modules and target directories.)"
},
"snapshot" : {
"args" : "snapshot",
"description" : "Execute snapshots defined in your project."
}
},
"matatikaHidden" : false,
"_links" : {
"self" : {
"href" : "https://catalog.matatika.com/api/dataplugins/35a04646-0982-473d-9632-4d6bcc18d7cf"
},
"update dataplugin" : {
"href" : "https://catalog.matatika.com/api/workspaces/d2dad12c-4533-49c3-9d4a-cf03cf7aaaca/dataplugins/35a04646-0982-473d-9632-4d6bcc18d7cf",
"type" : "PUT"
}
}
}
},
"managed" : true,
"draft" : false,
"_links" : {
"self" : {
"href" : "https://catalog.matatika.com/api/datacomponents/f7092d4c-9f85-4bf5-b81c-d32bd69ba30c"
},
"update datacomponent" : {
"href" : "https://catalog.matatika.com/api/datacomponents/f7092d4c-9f85-4bf5-b81c-d32bd69ba30c"
},
"delete datacomponent" : {
"href" : "https://catalog.matatika.com/api/datacomponents/f7092d4c-9f85-4bf5-b81c-d32bd69ba30c"
}
}
} ],
"latest job" : {
"id" : "85a3256d-3fce-48cc-b953-ca196d8504d7",
"created" : "2023-03-23T21:45:54.068873",
"type" : "WORKSPACE_CONFIG",
"status" : "QUEUED",
"_embedded" : {
"pipeline" : {
"id" : "48acbb57-1837-47d0-a230-ebb041a0fb39",
"status" : "PROVISIONING",
"name" : "SIT-generated pipeline [2023-03-23T21:45:52.793220]",
"schedule" : "0 0 0 25 12 ?",
"timeout" : 0,
"created" : "2023-03-23T21:45:53.448327",
"lastModified" : "2023-03-23T21:45:53.448328",
"properties" : {
"tap-google-analytics.view_id" : "1234567890",
"tap-google-analytics.oauth_credentials.client_secret" : "client_secret",
"tap-google-analytics.reports" : "reports",
"tap-google-analytics.oauth_credentials.refresh_token" : "refresh_token",
"tap-google-analytics.start_date" : "2023-02-23T21:45:53.021107Z",
"tap-google-analytics.end_date" : "2023-03-23T21:45:53.021218Z",
"tap-google-analytics.oauth_credentials.client_id" : "client_id",
"tap-google-analytics.oauth_credentials.access_token" : "access_token"
},
"dataComponents" : [ "tap-google-analytics", "Warehouse", "dbt" ],
"actions" : [ ]
}
},
"_links" : {
"self" : {
"href" : "https://catalog.matatika.com/api/jobs/85a3256d-3fce-48cc-b953-ca196d8504d7"
},
"delete job" : {
"href" : "https://catalog.matatika.com/api/jobs/85a3256d-3fce-48cc-b953-ca196d8504d7",
"type" : "DELETE"
},
"logs" : {
"href" : "https://catalog.matatika.com/api/jobs/85a3256d-3fce-48cc-b953-ca196d8504d7/logs?sequence=0",
"type" : "GET"
},
"withdraw job" : {
"href" : "https://catalog.matatika.com/api/jobs/85a3256d-3fce-48cc-b953-ca196d8504d7/stopped",
"type" : "PUT"
}
}
}
},
"_links" : {
"update pipeline" : {
"href" : "https://catalog.matatika.com/api/workspaces/d2dad12c-4533-49c3-9d4a-cf03cf7aaaca/pipelines/48acbb57-1837-47d0-a230-ebb041a0fb39",
"type" : "PUT"
},
"delete pipeline" : {
"href" : "https://catalog.matatika.com/api/pipelines/48acbb57-1837-47d0-a230-ebb041a0fb39",
"type" : "DELETE"
},
"draft pipeline" : {
"href" : "https://catalog.matatika.com/api/workspaces/d2dad12c-4533-49c3-9d4a-cf03cf7aaaca/pipelines/48acbb57-1837-47d0-a230-ebb041a0fb39/draft",
"type" : "PUT"
},
"self" : {
"href" : "https://catalog.matatika.com/api/pipelines/48acbb57-1837-47d0-a230-ebb041a0fb39"
},
"environment" : {
"href" : "https://catalog.matatika.com/api/pipelines/48acbb57-1837-47d0-a230-ebb041a0fb39/environment"
},
"jobs" : {
"href" : "https://catalog.matatika.com/api/pipelines/48acbb57-1837-47d0-a230-ebb041a0fb39/jobs",
"type" : "GET"
},
"add subscription" : {
"href" : "https://catalog.matatika.com/api/pipelines/48acbb57-1837-47d0-a230-ebb041a0fb39/subscriptions"
},
"withdraw job" : {
"href" : "https://catalog.matatika.com/api/jobs/85a3256d-3fce-48cc-b953-ca196d8504d7/stopped",
"type" : "PUT"
},
"latest job" : {
"href" : "https://catalog.matatika.com/api/jobs/85a3256d-3fce-48cc-b953-ca196d8504d7"
}
}
}
Create or update a pipeline as a draft
PUT
/api/workspaces/{workspace-id}/pipelines/{pipeline-id}/draft
Creates or updates the pipeline {pipeline-id}
in the workspace {workspace-id}
as a draft.
Prerequisites
- Workspace
{workspace-id}
must exist
Request
Body
Pipeline resource.
{
"name" : "SIT-generated pipeline [2023-03-23T21:46:50.230490]",
"dataComponents" : [ "tap-google-analytics", "Warehouse", "dbt" ]
}
Example Snippets
- cURL
curl -H "Authorization: Bearer $ACCESS_TOKEN" 'https://catalog.matatika.com:443/api/workspaces/d2dad12c-4533-49c3-9d4a-cf03cf7aaaca/pipelines/7437c296-93ad-470c-b14d-24afc965c711/draft' -i -X PUT \
-H 'Accept: application/json, application/javascript, text/javascript, text/json' \
-H 'Content-Type: application/json' \
-d '{
"name" : "SIT-generated pipeline [2023-03-23T21:46:50.230490]",
"dataComponents" : [ "tap-google-analytics", "Warehouse", "dbt" ]
}'
- Python (requests)
import requests
url = "https://catalog.matatika.com:443/api/workspaces/d2dad12c-4533-49c3-9d4a-cf03cf7aaaca/pipelines/7437c296-93ad-470c-b14d-24afc965c711/draft"
data = {
"name" : "SIT-generated pipeline [2023-03-23T21:46:50.230490]",
"dataComponents" : [ "tap-google-analytics", "Warehouse", "dbt" ]
}
headers = {
'Authorization': ACCESS_TOKEN
}
response = requests.request("PUT", url, headers=headers, data=data)
print(response.text.encode('utf8'))
Response
200 OK / 201 Created
Pipeline with HAL links.
{
"id" : "7437c296-93ad-470c-b14d-24afc965c711",
"status" : "PROVISIONING",
"name" : "SIT-generated pipeline [2023-03-23T21:46:50.230490]",
"timeout" : 0,
"created" : "2023-03-23T21:46:50.593501",
"lastModified" : "2023-03-23T21:46:50.593502",
"properties" : { },
"dataComponents" : [ "tap-google-analytics", "Warehouse", "dbt" ],
"actions" : [ ],
"_embedded" : {
"dataComponents" : [ {
"id" : "5be95c7e-4ef6-495d-b312-2e89c8cefaba",
"created" : "2023-03-23T21:45:52.26399",
"lastModified" : "2023-03-23T21:45:52.263991",
"name" : "tap-google-analytics",
"properties" : { },
"commands" : { },
"dataPlugin" : "extractors/tap-google-analytics--matatika",
"_embedded" : {
"dataplugin" : {
"id" : "d9e8ffbf-375a-4915-b629-2549c4c4060b",
"repositoryPath" : "plugins/extractors/tap-google-analytics--matatika.lock",
"pluginType" : "EXTRACTOR",
"name" : "tap-google-analytics",
"namespace" : "tap_google_analytics",
"variant" : "matatika",
"label" : "Google Analytics",
"description" : "App and website analytics platform hosted by Google\n\n## Prerequisites\n\nBefore importing data from Google Analytics you or your webmaster will need to have performed some initial setup to grant\nyourself access to the Google Analytics for your website.\n\n[Google Analytics Prerequisites](https://www.matatika.com/docs/data-sources/tap-google-analytics/prerequisites)\n\n## Data source settings\n### View ID\n\nOnce you have connected with your google account, the `View ID` drop down will allow you to choose which view you want to get your data from.\n\n### Start Date\n\nProviding a start date allows you to set how much historical data you wish to import into you workspace.\n\nThis date has to be in the ISO-8601 date format, see the example below.\n\nFormat: `YYYY-MM-DDTHH:MM:SSZ`\n\nExample: `2020-10-01T13:12:11Z`\n\n### End Date\n\nProviding an end date allows you to limit how much data you wish to import into you workspace.\n\nThis date has to be in the ISO-8601 date format, see the example below.\n\nFormat: `YYYY-MM-DDTHH:MM:SSZ`\n\nExample: `2020-10-01T13:12:11Z`\n\n### Reports\n\nProviding the path to a reports json file allows you to supply the tap with a user defined definition of tables and information you want to sync from Google Analytics.\nThis does require you to commit a file to your workspace repository so that Google Analytics can find it during run time, and may also break the dbt transforms that are required to have our included datasets work.\nFor more information on how to define a custom reports file see the [Meltano Hub Documentation](https://hub.meltano.com/extractors/google-analytics#reports).\n\n## Learn more\n\n[Matatika Google Analytics Data Source Documentation](https://www.matatika.com/docs/data-sources/tap-google-analytics)\n\n[Meltano Hub Google Analytics Documentation](https://hub.meltano.com/extractors/google-analytics)",
"logoUrl" : "https://app.matatika.com/assets/images/datasource/tap-google-analytics.svg",
"hidden" : false,
"docs" : "https://www.matatika.com/docs/data-sources/tap-google-analytics/",
"pipUrl" : "git+https://github.com/Matatika/[email protected]",
"repo" : "https://github.com/Matatika/tap-google-analytics",
"capabilities" : [ "catalog", "state", "discover" ],
"select" : [ ],
"update" : { },
"vars" : { },
"settings" : [ {
"name" : "oauth_credentials.authorization_url",
"aliases" : [ ],
"label" : "OAuth identity provider authorization endpoint used create and refresh tokens",
"value" : "https://oauth2.googleapis.com/token",
"kind" : "HIDDEN",
"required" : "true",
"protected" : false
}, {
"name" : "oauth_credentials.scope",
"aliases" : [ ],
"label" : "OAuth scopes we need to request access to",
"value" : "profile email https://www.googleapis.com/auth/analytics.readonly",
"kind" : "HIDDEN",
"required" : "true",
"protected" : false
}, {
"name" : "oauth_credentials.access_token",
"aliases" : [ ],
"label" : "Access Token",
"kind" : "HIDDEN",
"required" : "true",
"protected" : false
}, {
"name" : "oauth_credentials.refresh_token",
"aliases" : [ ],
"label" : "OAuth Refresh Token",
"kind" : "HIDDEN",
"required" : "true",
"protected" : false
}, {
"name" : "oauth_credentials.refresh_proxy_url",
"aliases" : [ ],
"label" : "Optional - will be called with 'oauth_credentials.refresh_token' to refresh the access token",
"kind" : "HIDDEN",
"protected" : false
}, {
"name" : "oauth_credentials.refresh_proxy_url_auth",
"aliases" : [ ],
"label" : "Optional - Sets Authorization header on 'oauth_credentials.refresh_url' request",
"kind" : "HIDDEN",
"protected" : false
}, {
"name" : "oauth_credentials.client_id",
"aliases" : [ ],
"label" : "Optional - OAuth Client ID used if refresh_proxy_url not supplied",
"kind" : "HIDDEN",
"protected" : false
}, {
"name" : "oauth_credentials.client_secret",
"aliases" : [ ],
"label" : "Optional - OAuth Client Secret used if refresh_proxy_url not supplied",
"kind" : "HIDDEN",
"protected" : false
}, {
"name" : "view_id",
"aliases" : [ ],
"label" : "View ID",
"placeholder" : "Ex. 198343027",
"kind" : "STRING",
"required" : "true",
"protected" : false
}, {
"name" : "reports",
"aliases" : [ ],
"label" : "Reports",
"placeholder" : "Ex. my_report_definition.json",
"kind" : "STRING",
"protected" : false
}, {
"name" : "start_date",
"aliases" : [ ],
"kind" : "DATE_ISO8601",
"protected" : false
}, {
"name" : "end_date",
"aliases" : [ ],
"kind" : "DATE_ISO8601",
"protected" : false
} ],
"variants" : [ ],
"commands" : { },
"matatikaHidden" : false,
"_links" : {
"self" : {
"href" : "https://catalog.matatika.com/api/dataplugins/d9e8ffbf-375a-4915-b629-2549c4c4060b"
},
"update dataplugin" : {
"href" : "https://catalog.matatika.com/api/workspaces/d2dad12c-4533-49c3-9d4a-cf03cf7aaaca/dataplugins/d9e8ffbf-375a-4915-b629-2549c4c4060b",
"type" : "PUT"
}
}
}
},
"managed" : false,
"draft" : false,
"_links" : {
"self" : {
"href" : "https://catalog.matatika.com/api/datacomponents/5be95c7e-4ef6-495d-b312-2e89c8cefaba"
},
"update datacomponent" : {
"href" : "https://catalog.matatika.com/api/datacomponents/5be95c7e-4ef6-495d-b312-2e89c8cefaba"
},
"delete datacomponent" : {
"href" : "https://catalog.matatika.com/api/datacomponents/5be95c7e-4ef6-495d-b312-2e89c8cefaba"
}
}
}, {
"id" : "a3de91eb-39a0-4ea8-a6a2-615d5ff721e3",
"created" : "2023-03-23T21:45:24.806929",
"lastModified" : "2023-03-23T21:45:24.80693",
"name" : "Warehouse",
"properties" : {
"password" : "Ptm5kD04_3ngU1Y6E_YeVd43da",
"dbname" : "ugdwwoy",
"default_target_schema" : "analytics",
"port" : "5432",
"host" : "nutty-ugli.db.elephantsql.com",
"user" : "ugdwwoy"
},
"commands" : { },
"dataPlugin" : "loaders/target-postgres--transferwise",
"_embedded" : {
"dataplugin" : {
"id" : "bed88248-835a-468d-a57c-b5b4ac24002c",
"repositoryPath" : "plugins/loaders/target-postgres--transferwise.lock",
"pluginType" : "LOADER",
"name" : "target-postgres",
"namespace" : "postgres_transferwise",
"variant" : "transferwise",
"label" : "Postgres Warehouse",
"description" : "Loader for Postgres Data Warehouse\n\nThe World's Most Advanced Open Source Relational Database\n\n## Settings\n\n### User\nPostgres user\n\n### Password\nPostgres password\n\n### Host\nPostgres host\nExample: `test-host.db.elephantsql.com`\n\n### Port\nPostgres port\nExample: `5432`\n\n### Database Name\nPostgres database name\n\n### Default Target Schema\nName of the schema where tables will be created (no database prefix)",
"logoUrl" : "https://app.matatika.com/assets/logos/loaders/postgres.png",
"hidden" : false,
"pipUrl" : "git+https://github.com/Matatika/[email protected]",
"repo" : "git+https://github.com/Matatika/[email protected]",
"capabilities" : [ ],
"select" : [ ],
"update" : { },
"vars" : { },
"settings" : [ {
"name" : "user",
"aliases" : [ "username" ],
"label" : "User",
"kind" : "STRING",
"description" : "PostgreSQL user",
"required" : "true",
"protected" : false
}, {
"name" : "password",
"aliases" : [ ],
"label" : "Password",
"kind" : "PASSWORD",
"description" : "PostgreSQL password",
"required" : "true",
"protected" : false
}, {
"name" : "host",
"aliases" : [ "address" ],
"label" : "Host",
"kind" : "STRING",
"description" : "PostgreSQL host",
"required" : "true",
"protected" : false
}, {
"name" : "port",
"aliases" : [ ],
"label" : "Port",
"value" : "5432",
"kind" : "INTEGER",
"description" : "PostgreSQL port",
"required" : "true",
"protected" : false
}, {
"name" : "dbname",
"aliases" : [ "database" ],
"label" : "Database Name",
"kind" : "STRING",
"description" : "PostgreSQL database name",
"required" : "true",
"protected" : false
}, {
"name" : "default_target_schema",
"aliases" : [ ],
"label" : "Default Target Schema",
"value" : "analytics",
"kind" : "STRING",
"description" : "Name of the schema where the tables will be created.",
"required" : "true",
"protected" : false
}, {
"name" : "ssl",
"aliases" : [ ],
"label" : "SSL",
"value" : "false",
"kind" : "HIDDEN",
"protected" : false,
"value_post_processor" : "STRINGIFY"
}, {
"name" : "batch_size_rows",
"aliases" : [ ],
"label" : "Batch Size Rows",
"value" : "100000",
"kind" : "INTEGER",
"description" : "Maximum number of rows in each batch. At the end of each batch, the rows in the batch are loaded into Postgres.",
"protected" : false
}, {
"name" : "underscore_camel_case_fields",
"aliases" : [ ],
"label" : "Underscore Camel Case Fields",
"value" : "true",
"kind" : "HIDDEN",
"description" : "Enable underscoring camel case field names, make the resulting postgres column names the same as from the Meltano variant of target-postgres.",
"protected" : false
}, {
"name" : "flush_all_streams",
"aliases" : [ ],
"label" : "Flush All Streams",
"value" : "false",
"kind" : "HIDDEN",
"description" : "Flush and load every stream into Postgres when one batch is full. Warning: This may trigger the COPY command to use files with low number of records.",
"protected" : false
}, {
"name" : "parallelism",
"aliases" : [ ],
"label" : "Parallelism",
"value" : "0",
"kind" : "HIDDEN",
"description" : "The number of threads used to flush tables. 0 will create a thread for each stream, up to parallelism_max. -1 will create a thread for each CPU core. Any other positive number will create that number of threads, up to parallelism_max.",
"protected" : false
}, {
"name" : "parallelism_max",
"aliases" : [ ],
"label" : "Max Parallelism",
"value" : "16",
"kind" : "HIDDEN",
"description" : "Max number of parallel threads to use when flushing tables.",
"protected" : false
}, {
"name" : "default_target_schema_select_permission",
"aliases" : [ ],
"label" : "Default Target Schema Select Permission",
"kind" : "HIDDEN",
"description" : "Grant USAGE privilege on newly created schemas and grant SELECT privilege on newly created tables to a specific role or a list of roles. If `schema_mapping` is not defined then every stream sent by the tap is granted accordingly.",
"protected" : false
}, {
"name" : "schema_mapping",
"aliases" : [ ],
"label" : "Schema Mapping",
"kind" : "HIDDEN",
"description" : "Useful if you want to load multiple streams from one tap to multiple Postgres schemas.\n\nIf the tap sends the `stream_id` in `<schema_name>-<table_name>` format then this option overwrites the `default_target_schema` value.\nNote, that using `schema_mapping` you can overwrite the `default_target_schema_select_permission` value to grant SELECT permissions to different groups per schemas or optionally you can create indices automatically for the replicated tables.\n",
"protected" : false
}, {
"name" : "add_metadata_columns",
"aliases" : [ ],
"label" : "Add Metadata Columns",
"value" : "true",
"kind" : "HIDDEN",
"description" : "Metadata columns add extra row level information about data ingestions, (i.e. when was the row read in source, when was inserted or deleted in postgres etc.) Metadata columns are creating automatically by adding extra columns to the tables with a column prefix `_SDC_`. The column names are following the stitch naming conventions documented at https://www.stitchdata.com/docs/data-structure/integration-schemas#sdc-columns. Enabling metadata columns will flag the deleted rows by setting the `_SDC_DELETED_AT` metadata column. Without the `add_metadata_columns` option the deleted rows from singer taps will not be recongisable in Postgres.",
"protected" : false
}, {
"name" : "hard_delete",
"aliases" : [ ],
"label" : "Hard Delete",
"value" : "false",
"kind" : "HIDDEN",
"description" : "When `hard_delete` option is true then DELETE SQL commands will be performed in Postgres to delete rows in tables. It's achieved by continuously checking the `_SDC_DELETED_AT` metadata column sent by the singer tap. Due to deleting rows requires metadata columns, `hard_delete` option automatically enables the `add_metadata_columns` option as well.",
"protected" : false
}, {
"name" : "data_flattening_max_level",
"aliases" : [ ],
"label" : "Data Flattening Max Level",
"value" : "10",
"kind" : "HIDDEN",
"description" : "Object type RECORD items from taps can be transformed to flattened columns by creating columns automatically. When value is 0 (default) then flattening functionality is turned off.",
"protected" : false
}, {
"name" : "primary_key_required",
"aliases" : [ ],
"label" : "Primary Key Required",
"value" : "false",
"kind" : "BOOLEAN",
"description" : "Log based and Incremental replications on tables with no Primary Key cause duplicates when merging UPDATE events. When set to true, stop loading data if no Primary Key is defined.",
"protected" : false
}, {
"name" : "validate_records",
"aliases" : [ ],
"label" : "Validate Records",
"value" : "false",
"kind" : "BOOLEAN",
"description" : "Validate every single record message to the corresponding JSON schema. This option is disabled by default and invalid RECORD messages will fail only at load time by Postgres. Enabling this option will detect invalid records earlier but could cause performance degradation.",
"protected" : false
}, {
"name" : "temp_dir",
"aliases" : [ ],
"label" : "Temporary Directory",
"kind" : "HIDDEN",
"description" : "(Default: platform-dependent) Directory of temporary CSV files with RECORD messages.",
"protected" : false
} ],
"variants" : [ ],
"commands" : { },
"matatikaHidden" : false,
"_links" : {
"self" : {
"href" : "https://catalog.matatika.com/api/dataplugins/bed88248-835a-468d-a57c-b5b4ac24002c"
},
"update dataplugin" : {
"href" : "https://catalog.matatika.com/api/workspaces/d2dad12c-4533-49c3-9d4a-cf03cf7aaaca/dataplugins/bed88248-835a-468d-a57c-b5b4ac24002c",
"type" : "PUT"
}
}
}
},
"managed" : true,
"draft" : false,
"_links" : {
"self" : {
"href" : "https://catalog.matatika.com/api/datacomponents/a3de91eb-39a0-4ea8-a6a2-615d5ff721e3"
},
"update datacomponent" : {
"href" : "https://catalog.matatika.com/api/datacomponents/a3de91eb-39a0-4ea8-a6a2-615d5ff721e3"
},
"delete datacomponent" : {
"href" : "https://catalog.matatika.com/api/datacomponents/a3de91eb-39a0-4ea8-a6a2-615d5ff721e3"
}
}
}, {
"id" : "166fc471-51cc-449f-b9cc-58b890eea347",
"created" : "2023-03-23T21:46:46.377154",
"lastModified" : "2023-03-23T21:46:46.377169",
"name" : "dbt",
"properties" : { },
"commands" : {
"compile" : {
"args" : "compile",
"description" : "Generates executable SQL from source model, test, and analysis files. Compiled SQL files are written to the target/ directory."
},
"seed" : {
"args" : "seed",
"description" : "Load data from csv files into your data warehouse."
},
"test" : {
"args" : "test",
"description" : "Runs tests on data in deployed models."
},
"deps" : {
"args" : "deps",
"description" : "Pull the most recent version of the dependencies listed in packages.yml"
},
"run" : {
"args" : "run",
"description" : "Compile SQL and execute against the current target database."
},
"clean" : {
"args" : "clean",
"description" : "Delete all folders in the clean-targets list (usually the dbt_modules and target directories.)"
},
"snapshot" : {
"args" : "snapshot",
"description" : "Execute snapshots defined in your project."
}
},
"dataPlugin" : "transformers/dbt--dbt-labs",
"_embedded" : {
"dataplugin" : {
"id" : "35a04646-0982-473d-9632-4d6bcc18d7cf",
"repositoryPath" : "plugins/transformers/dbt--dbt-labs.lock",
"pluginType" : "TRANSFORMER",
"name" : "dbt",
"namespace" : "dbt",
"variant" : "dbt-labs",
"label" : "dbt",
"logoUrl" : "https://app.matatika.com/assets/images/transformer/dbt.png",
"hidden" : false,
"docs" : "https://meltano.com/docs/transforms.html",
"pipUrl" : "dbt-core~=1.3.0 dbt-postgres~=1.3.0 dbt-snowflake~=1.3.0\n",
"repo" : "https://github.com/dbt-labs/dbt-core",
"capabilities" : [ ],
"select" : [ ],
"update" : { },
"vars" : { },
"settings" : [ {
"name" : "project_dir",
"aliases" : [ ],
"value" : "$MELTANO_PROJECT_ROOT/transform",
"kind" : "STRING",
"protected" : false
}, {
"name" : "profiles_dir",
"aliases" : [ ],
"value" : "$MELTANO_PROJECT_ROOT/transform/profile",
"kind" : "STRING",
"env" : "DBT_PROFILES_DIR",
"protected" : false
}, {
"name" : "target",
"aliases" : [ ],
"value" : "$MELTANO_LOAD__DIALECT",
"kind" : "STRING",
"protected" : false
}, {
"name" : "source_schema",
"aliases" : [ ],
"value" : "$MELTANO_LOAD__TARGET_SCHEMA",
"kind" : "STRING",
"protected" : false
}, {
"name" : "target_schema",
"aliases" : [ ],
"value" : "analytics",
"kind" : "STRING",
"protected" : false
}, {
"name" : "models",
"aliases" : [ ],
"value" : "$MELTANO_TRANSFORM__PACKAGE_NAME $MELTANO_EXTRACTOR_NAMESPACE my_meltano_project",
"kind" : "STRING",
"protected" : false
} ],
"variants" : [ ],
"commands" : {
"compile" : {
"args" : "compile",
"description" : "Generates executable SQL from source model, test, and analysis files. Compiled SQL files are written to the target/ directory."
},
"seed" : {
"args" : "seed",
"description" : "Load data from csv files into your data warehouse."
},
"test" : {
"args" : "test",
"description" : "Runs tests on data in deployed models."
},
"deps" : {
"args" : "deps",
"description" : "Pull the most recent version of the dependencies listed in packages.yml"
},
"run" : {
"args" : "run",
"description" : "Compile SQL and execute against the current target database."
},
"clean" : {
"args" : "clean",
"description" : "Delete all folders in the clean-targets list (usually the dbt_modules and target directories.)"
},
"snapshot" : {
"args" : "snapshot",
"description" : "Execute snapshots defined in your project."
}
},
"matatikaHidden" : false,
"_links" : {
"self" : {
"href" : "https://catalog.matatika.com/api/dataplugins/35a04646-0982-473d-9632-4d6bcc18d7cf"
},
"update dataplugin" : {
"href" : "https://catalog.matatika.com/api/workspaces/d2dad12c-4533-49c3-9d4a-cf03cf7aaaca/dataplugins/35a04646-0982-473d-9632-4d6bcc18d7cf",
"type" : "PUT"
}
}
}
},
"managed" : false,
"draft" : false,
"_links" : {
"self" : {
"href" : "https://catalog.matatika.com/api/datacomponents/166fc471-51cc-449f-b9cc-58b890eea347"
},
"update datacomponent" : {
"href" : "https://catalog.matatika.com/api/datacomponents/166fc471-51cc-449f-b9cc-58b890eea347"
},
"delete datacomponent" : {
"href" : "https://catalog.matatika.com/api/datacomponents/166fc471-51cc-449f-b9cc-58b890eea347"
}
}
} ],
"latest job" : {
"id" : "9cd1a03b-7749-42f0-a975-1364feeb0b45",
"created" : "2023-03-23T21:46:50.998498",
"type" : "WORKSPACE_CONFIG",
"status" : "QUEUED",
"_embedded" : {
"pipeline" : {
"id" : "7437c296-93ad-470c-b14d-24afc965c711",
"status" : "PROVISIONING",
"name" : "SIT-generated pipeline [2023-03-23T21:46:50.230490]",
"timeout" : 0,
"created" : "2023-03-23T21:46:50.593501",
"lastModified" : "2023-03-23T21:46:50.593502",
"properties" : { },
"dataComponents" : [ "tap-google-analytics", "Warehouse", "dbt" ],
"actions" : [ ]
}
},
"_links" : {
"self" : {
"href" : "https://catalog.matatika.com/api/jobs/9cd1a03b-7749-42f0-a975-1364feeb0b45"
},
"delete job" : {
"href" : "https://catalog.matatika.com/api/jobs/9cd1a03b-7749-42f0-a975-1364feeb0b45",
"type" : "DELETE"
},
"logs" : {
"href" : "https://catalog.matatika.com/api/jobs/9cd1a03b-7749-42f0-a975-1364feeb0b45/logs?sequence=0",
"type" : "GET"
},
"withdraw job" : {
"href" : "https://catalog.matatika.com/api/jobs/9cd1a03b-7749-42f0-a975-1364feeb0b45/stopped",
"type" : "PUT"
}
}
}
},
"_links" : {
"update pipeline" : {
"href" : "https://catalog.matatika.com/api/workspaces/d2dad12c-4533-49c3-9d4a-cf03cf7aaaca/pipelines/7437c296-93ad-470c-b14d-24afc965c711",
"type" : "PUT"
},
"delete pipeline" : {
"href" : "https://catalog.matatika.com/api/pipelines/7437c296-93ad-470c-b14d-24afc965c711",
"type" : "DELETE"
},
"draft pipeline" : {
"href" : "https://catalog.matatika.com/api/workspaces/d2dad12c-4533-49c3-9d4a-cf03cf7aaaca/pipelines/7437c296-93ad-470c-b14d-24afc965c711/draft",
"type" : "PUT"
},
"self" : {
"href" : "https://catalog.matatika.com/api/pipelines/7437c296-93ad-470c-b14d-24afc965c711"
},
"environment" : {
"href" : "https://catalog.matatika.com/api/pipelines/7437c296-93ad-470c-b14d-24afc965c711/environment"
},
"jobs" : {
"href" : "https://catalog.matatika.com/api/pipelines/7437c296-93ad-470c-b14d-24afc965c711/jobs",
"type" : "GET"
},
"add subscription" : {
"href" : "https://catalog.matatika.com/api/pipelines/7437c296-93ad-470c-b14d-24afc965c711/subscriptions"
},
"withdraw job" : {
"href" : "https://catalog.matatika.com/api/jobs/9cd1a03b-7749-42f0-a975-1364feeb0b45/stopped",
"type" : "PUT"
},
"latest job" : {
"href" : "https://catalog.matatika.com/api/jobs/9cd1a03b-7749-42f0-a975-1364feeb0b45"
}
}
}
Validate a pipeline configuration in a workspace
POST
/api/workspaces/{workspace-id}/pipelines/validation
Validates a pipeline configuration in the workspace {workspace-id}
.
Prerequisites
- Workspace
{workspace-id}
must exist
Request
Body
Pipeline resource.
{
"name" : "SIT-generated pipeline [2023-03-23T21:45:52.793220]",
"dataComponents" : [ "tap-google-analytics", "Warehouse", "dbt" ]
}
Example Snippets
- cURL
curl -H "Authorization: Bearer $ACCESS_TOKEN" 'https://catalog.matatika.com:443/api/workspaces/d2dad12c-4533-49c3-9d4a-cf03cf7aaaca/pipelines/validation' -i -X POST \
-H 'Accept: application/json, application/javascript, text/javascript, text/json' \
-H 'Content-Type: application/json' \
-d '{
"name" : "SIT-generated pipeline [2023-03-23T21:45:52.793220]",
"dataComponents" : [ "tap-google-analytics", "Warehouse", "dbt" ]
}'
- Python (requests)
import requests
url = "https://catalog.matatika.com:443/api/workspaces/d2dad12c-4533-49c3-9d4a-cf03cf7aaaca/pipelines/validation"
data = {
"name" : "SIT-generated pipeline [2023-03-23T21:45:52.793220]",
"dataComponents" : [ "tap-google-analytics", "Warehouse", "dbt" ]
}
headers = {
'Authorization': ACCESS_TOKEN
}
response = requests.request("POST", url, headers=headers, data=data)
print(response.text.encode('utf8'))
Response
200 OK
No response body provided.
400 Bad Request
Pipeline property validation errors.
{
"timestamp" : "2023-03-23T21:45:52.953811676",
"status" : 400,
"error" : "Bad Request",
"message" : "3 validation errors from 'resource'",
"errors" : [ {
"codes" : [ "NotBlank.oauth_credentials.access_token", "NotBlank" ],
"defaultMessage" : "No value given for setting",
"objectName" : "resource",
"field" : "properties.tap-google-analytics.oauth_credentials.access_token",
"bindingFailure" : true,
"code" : "NotBlank"
}, {
"codes" : [ "NotBlank.oauth_credentials.refresh_token", "NotBlank" ],
"defaultMessage" : "No value given for setting",
"objectName" : "resource",
"field" : "properties.tap-google-analytics.oauth_credentials.refresh_token",
"bindingFailure" : true,
"code" : "NotBlank"
}, {
"codes" : [ "NotBlank.view_id", "NotBlank" ],
"defaultMessage" : "No value given for setting",
"objectName" : "resource",
"field" : "properties.tap-google-analytics.view_id",
"bindingFailure" : true,
"code" : "NotBlank"
} ],
"path" : "/api/workspaces/d2dad12c-4533-49c3-9d4a-cf03cf7aaaca/pipelines/validation"
}
Verify a pipeline
POST
/api/pipelines/{pipeline-id}/verification
Verifies the configuration of the pipeline {pipeline-id}
.
Prerequisites
- Pipeline
{pipeline-id}
must exist
Request
Example Snippets
- cURL
curl -H "Authorization: Bearer $ACCESS_TOKEN" 'https://catalog.matatika.com:443/api/pipelines/9c8e2db3-3015-4a2e-96a9-c8d7458b8523/verification' -i -X POST \
-H 'Accept: application/json, application/javascript, text/javascript, text/json' \
-H 'Content-Type: application/json'
- Python (requests)
import requests
url = "https://catalog.matatika.com:443/api/pipelines/9c8e2db3-3015-4a2e-96a9-c8d7458b8523/verification"
headers = {
'Authorization': ACCESS_TOKEN
}
response = requests.request("POST", url, headers=headers)
print(response.text.encode('utf8'))
Response
200 OK
Job with HAL links.
{
"id" : "e7aa5bf2-b4cb-4419-a4e9-8dd5f66035ac",
"created" : "2023-03-23T21:50:25.986164",
"type" : "PIPELINE_VERIFY",
"status" : "QUEUED",
"_embedded" : {
"pipeline" : {
"id" : "9c8e2db3-3015-4a2e-96a9-c8d7458b8523",
"status" : "READY",
"name" : "SIT-Generated Pipeline [2023-03-23T21:49:39.544676]",
"timeout" : 0,
"created" : "2023-03-23T21:49:39.971308",
"lastModified" : "2023-03-23T21:49:39.971308",
"properties" : { },
"dataComponents" : [ "tap-matatika-sit", "Warehouse", "dbt" ],
"actions" : [ ],
"repositoryPath" : "pipelines/SIT-Generated Pipeline [2023-03-23T21:49:39.544676].yml"
}
},
"_links" : {
"self" : {
"href" : "https://catalog.matatika.com/api/jobs/e7aa5bf2-b4cb-4419-a4e9-8dd5f66035ac"
},
"delete job" : {
"href" : "https://catalog.matatika.com/api/jobs/e7aa5bf2-b4cb-4419-a4e9-8dd5f66035ac",
"type" : "DELETE"
},
"logs" : {
"href" : "https://catalog.matatika.com/api/jobs/e7aa5bf2-b4cb-4419-a4e9-8dd5f66035ac/logs?sequence=0",
"type" : "GET"
},
"withdraw job" : {
"href" : "https://catalog.matatika.com/api/jobs/e7aa5bf2-b4cb-4419-a4e9-8dd5f66035ac/stopped",
"type" : "PUT"
}
}
}
Delete a pipeline
DELETE
/api/pipelines/{pipeline-id}
Deletes the pipeline {pipeline-id}
.
Prerequisites
- Pipeline
{pipeline-id}
must exist
Request
Example Snippets
- cURL
curl -H "Authorization: Bearer $ACCESS_TOKEN" 'https://catalog.matatika.com:443/api/pipelines/7437c296-93ad-470c-b14d-24afc965c711' -i -X DELETE \
-H 'Accept: application/json, application/javascript, text/javascript, text/json' \
-H 'Content-Type: application/json'
- Python (requests)
import requests
url = "https://catalog.matatika.com:443/api/pipelines/7437c296-93ad-470c-b14d-24afc965c711"
headers = {
'Authorization': ACCESS_TOKEN
}
response = requests.request("DELETE", url, headers=headers)
print(response.text.encode('utf8'))
Response
204 No Content
No response body provided.