Pipelines
A pipeline defines a set of runnable actions composed from datacomponents to complete a set of tasks - for example, ELT. Pipelines are run as jobs, either manually or on a predetermined schedule. Only a single pipeline can be run at any given time.
Objects
Pipeline
| Path | JSON Type | Format | Description |
|---|---|---|---|
id |
string |
Version 4 UUID | The pipeline ID |
status |
string |
Pipeline Status | |
name |
string |
The pipeline name | |
schedule |
string |
Cron | The interval at which to launch a new job e.g. 0 0 9-17 * * MON-FRI launches a job on the hour nine-to-five weekdays |
timeout |
number |
Unsigned integer | The number of seconds after which the job will terminate - if set to 0, an implicit default value of 300 seconds is used |
maxRetries |
number |
Unsigned integer | The maximum number of retries to attempt for a job ending with ERROR |
script |
string |
Bash script | Custom script to execute during a job |
created |
string |
ISO 8601 timestamp | When the pipeline was created |
lastModified |
string |
ISO 8601 timestamp | When the pipeline was last modified |
properties |
object |
Properties |
The pipeline properties, defined by the dataplugin settings of each datacomponent |
dataComponents |
string[] |
Array of datacomponent names |
The pipeline datacomponent names or create / update with dataplugin fullyQualifiedName |
actions |
string[] |
Array of datacomponent names or commands |
The pipeline actions to run during a job |
triggeredBy |
string[] |
Array of pipeline names or workspace task identifiers |
Pipelines or workspace tasks that will trigger the pipeline on successful completion Supported values for workspace tasks (case-insensitive):
|
{
"id" : "180a0df1-e833-4aae-b6ef-ee98f0f07b50",
"status" : "READY",
"name" : "SIT-generated pipeline [2025-06-19T11:19:48.350863] (updated)",
"timeout" : 0,
"maxRetries" : 0,
"created" : "2025-06-19T10:19:48.519054",
"lastModified" : "2025-06-19T10:19:48.519056",
"properties" : {
"tap-google-analytics.oauth_credentials.client_secret" : "client_secret",
"tap-google-analytics.view_id" : "1234567890",
"tap-google-analytics.reports" : "reports",
"tap-google-analytics.oauth_credentials.refresh_token" : "refresh_token",
"tap-google-analytics.start_date" : "2025-05-19T11:19:47.874302+01:00",
"tap-google-analytics.end_date" : "2025-06-19T11:19:47.874360+01:00",
"tap-google-analytics.oauth_credentials.access_token" : "access_token",
"tap-google-analytics.oauth_credentials.client_id" : "client_id"
},
"dataComponents" : [ "tap-google-analytics", "Warehouse", "dbt" ],
"actions" : [ ],
"triggeredBy" : [ ],
"_embedded" : {
"dataComponents" : [ {
"id" : "b6e29a0b-8836-4e3a-bbfd-f85163798009",
"created" : "2025-06-19T10:19:47.983989",
"lastModified" : "2025-06-19T10:19:47.98399",
"name" : "tap-google-analytics",
"properties" : { },
"commands" : { },
"dataPlugin" : "extractors/tap-google-analytics--matatika",
"_embedded" : {
"dataplugin" : {
"id" : "6b89da64-1538-4519-a188-5ff173208879",
"pluginType" : "EXTRACTOR",
"name" : "tap-google-analytics",
"namespace" : "tap_google_analytics",
"variant" : "matatika",
"label" : "Google Analytics",
"description" : "Google Analytics is a web analytics service that provides insights into website traffic and user behavior.\n\nGoogle Analytics allows website owners to track and analyze various metrics related to their website's performance, such as the number of visitors, pageviews, bounce rate, and average session duration. It also provides information on the demographics and interests of website visitors, as well as the sources of traffic, including organic search, paid search, social media, and referrals. This data can be used to optimize website content and marketing strategies, as well as to measure the effectiveness of advertising campaigns. Additionally, Google Analytics offers advanced features such as goal tracking, e-commerce tracking, and custom reporting, making it a powerful tool for businesses of all sizes.\n### Prerequisites\nTo obtain the OAuth identity provider authorization endpoint used to create and refresh tokens, you need to create a project in the Google API Console and enable the Google Analytics API. Then, you can create OAuth 2.0 credentials and configure the authorized redirect URIs. The authorization endpoint will be provided in the credentials.\n\nThe OAuth scopes you need to request access to depend on the specific data you want to access in Google Analytics. For example, if you want to read data from a specific view, you will need to request the \"https://www.googleapis.com/auth/analytics.readonly\" scope. You can find a list of available scopes in the Google Analytics API documentation.\n\nTo obtain the Access Token and OAuth Refresh Token, you need to authenticate the user and obtain their consent to access their Google Analytics data. This can be done using the Google Sign-In API or the OAuth 2.0 authorization flow. Once the user has granted access, you will receive an Access Token and a Refresh Token that you can use to make API requests.\n\nTo obtain the View ID, you need to log in to your Google Analytics account and navigate to the Admin section. From there, you can select the account, property, and view that you want to access and find the View ID in the View Settings.",
"logoUrl" : "/assets/images/datasource/tap-google-analytics.svg",
"hidden" : false,
"docs" : "https://www.matatika.com/docs/instant-insights/tap-google-analytics/",
"pipUrl" : "git+https://github.com/Matatika/[email protected]",
"repo" : "https://github.com/Matatika/tap-google-analytics",
"capabilities" : [ "DISCOVER", "CATALOG", "STATE" ],
"select" : [ ],
"update" : { },
"vars" : { },
"settings" : [ {
"name" : "oauth_credentials.authorization_url",
"aliases" : [ ],
"label" : "OAuth identity provider authorization endpoint used create and refresh tokens",
"value" : "https://oauth2.googleapis.com/token",
"options" : [ ],
"kind" : "STRING",
"description" : "The endpoint used to create and refresh OAuth tokens.",
"hidden" : true,
"sensitive" : false,
"required" : "true",
"protected" : false
}, {
"name" : "oauth_credentials.scope",
"aliases" : [ ],
"label" : "OAuth scopes we need to request access to",
"value" : "profile email https://www.googleapis.com/auth/analytics.readonly",
"options" : [ ],
"kind" : "STRING",
"description" : "The specific scopes we need to request access to in order to connect to Google Analytics.",
"hidden" : true,
"sensitive" : false,
"required" : "true",
"protected" : false
}, {
"name" : "oauth_credentials.access_token",
"aliases" : [ ],
"label" : "Access Token",
"options" : [ ],
"kind" : "STRING",
"description" : "The token used to authenticate and authorize API requests.",
"hidden" : true,
"sensitive" : false,
"required" : "true",
"protected" : false
}, {
"name" : "oauth_credentials.refresh_token",
"aliases" : [ ],
"label" : "OAuth Refresh Token",
"options" : [ ],
"kind" : "STRING",
"description" : "The token used to refresh the access token when it expires.",
"hidden" : true,
"sensitive" : false,
"required" : "true",
"protected" : false
}, {
"name" : "oauth_credentials.refresh_proxy_url",
"aliases" : [ ],
"label" : "Optional - will be called with 'oauth_credentials.refresh_token' to refresh the access token",
"options" : [ ],
"kind" : "STRING",
"description" : "An optional function that will be called to refresh the access token using the refresh token.",
"hidden" : true,
"sensitive" : false,
"protected" : false
}, {
"name" : "oauth_credentials.refresh_proxy_url_auth",
"aliases" : [ ],
"label" : "Optional - Sets Authorization header on 'oauth_credentials.refresh_url' request",
"options" : [ ],
"kind" : "STRING",
"description" : "An optional setting that sets the Authorization header on the refresh URL request.",
"hidden" : true,
"sensitive" : false,
"protected" : false
}, {
"name" : "oauth_credentials.client_id",
"aliases" : [ ],
"label" : "Optional - OAuth Client ID used if refresh_proxy_url not supplied",
"options" : [ ],
"kind" : "STRING",
"description" : "An optional OAuth Client ID used if the refresh proxy URL is not supplied.",
"hidden" : true,
"sensitive" : false,
"protected" : false
}, {
"name" : "oauth_credentials.client_secret",
"aliases" : [ ],
"label" : "Optional - OAuth Client Secret used if refresh_proxy_url not supplied",
"options" : [ ],
"kind" : "STRING",
"description" : "An optional OAuth Client Secret used if the refresh proxy URL is not supplied.",
"hidden" : true,
"sensitive" : false,
"protected" : false
}, {
"name" : "view_id",
"aliases" : [ ],
"label" : "View ID",
"options" : [ ],
"placeholder" : "Ex. 198343027",
"kind" : "STRING",
"description" : "The ID of the Google Analytics view to retrieve data from.",
"hidden" : false,
"sensitive" : false,
"required" : "true",
"protected" : false
}, {
"name" : "reports",
"aliases" : [ ],
"label" : "Reports",
"options" : [ ],
"placeholder" : "Ex. my_report_definition.json",
"kind" : "STRING",
"description" : "The specific reports to retrieve data from in the Google Analytics view.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "start_date",
"aliases" : [ ],
"label" : "Start date",
"options" : [ ],
"kind" : "DATE_ISO8601",
"description" : "The start date for the date range of data to retrieve.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "end_date",
"aliases" : [ ],
"label" : "End date",
"options" : [ ],
"kind" : "DATE_ISO8601",
"description" : "The end date for the date range of data to retrieve.",
"hidden" : false,
"sensitive" : false,
"protected" : false
} ],
"variants" : [ ],
"commands" : { },
"matatikaHidden" : false,
"requires" : [ ],
"fullDescription" : "Google Analytics is a web analytics service that provides insights into website traffic and user behavior.\n\nGoogle Analytics allows website owners to track and analyze various metrics related to their website's performance, such as the number of visitors, pageviews, bounce rate, and average session duration. It also provides information on the demographics and interests of website visitors, as well as the sources of traffic, including organic search, paid search, social media, and referrals. This data can be used to optimize website content and marketing strategies, as well as to measure the effectiveness of advertising campaigns. Additionally, Google Analytics offers advanced features such as goal tracking, e-commerce tracking, and custom reporting, making it a powerful tool for businesses of all sizes.\n### Prerequisites\nTo obtain the OAuth identity provider authorization endpoint used to create and refresh tokens, you need to create a project in the Google API Console and enable the Google Analytics API. Then, you can create OAuth 2.0 credentials and configure the authorized redirect URIs. The authorization endpoint will be provided in the credentials.\n\nThe OAuth scopes you need to request access to depend on the specific data you want to access in Google Analytics. For example, if you want to read data from a specific view, you will need to request the \"https://www.googleapis.com/auth/analytics.readonly\" scope. You can find a list of available scopes in the Google Analytics API documentation.\n\nTo obtain the Access Token and OAuth Refresh Token, you need to authenticate the user and obtain their consent to access their Google Analytics data. This can be done using the Google Sign-In API or the OAuth 2.0 authorization flow. Once the user has granted access, you will receive an Access Token and a Refresh Token that you can use to make API requests.\n\nTo obtain the View ID, you need to log in to your Google Analytics account and navigate to the Admin section. From there, you can select the account, property, and view that you want to access and find the View ID in the View Settings.\n\n## Settings\n\n\n### View ID\n\nThe ID of the Google Analytics view to retrieve data from.\n\n### Reports\n\nThe specific reports to retrieve data from in the Google Analytics view.\n\n### Start date\n\nThe start date for the date range of data to retrieve.\n\n### End date\n\nThe end date for the date range of data to retrieve.",
"_links" : {
"self" : {
"href" : "https://app.matatika.com/api/dataplugins/6b89da64-1538-4519-a188-5ff173208879"
},
"update dataplugin" : {
"href" : "https://app.matatika.com/api/workspaces/52652da2-a249-407e-a8c6-4e045838a591/dataplugins/6b89da64-1538-4519-a188-5ff173208879",
"type" : "PUT"
}
}
}
},
"managed" : false,
"draft" : true,
"_links" : {
"self" : {
"href" : "https://app.matatika.com/api/datacomponents/b6e29a0b-8836-4e3a-bbfd-f85163798009"
},
"update datacomponent" : {
"href" : "https://app.matatika.com/api/datacomponents/b6e29a0b-8836-4e3a-bbfd-f85163798009"
},
"delete datacomponent" : {
"href" : "https://app.matatika.com/api/datacomponents/b6e29a0b-8836-4e3a-bbfd-f85163798009"
}
}
}, {
"id" : "a604ff5e-5fae-4578-ae4a-650091f015f3",
"created" : "2025-06-19T10:19:42.42539",
"lastModified" : "2025-06-19T10:19:42.425391",
"name" : "Warehouse",
"properties" : {
"password" : "7PhOTBc5_2SMTn3on_ntw8iRA1",
"dbname" : "axvzzfd",
"default_target_schema" : "analytics",
"port" : "5432",
"host" : "sharp-banana2.postgres.database.azure.com",
"user" : "axvzzfd"
},
"commands" : { },
"dataPlugin" : "loaders/target-postgres--matatika",
"_embedded" : {
"dataplugin" : {
"id" : "82bf91fa-8adf-4421-863e-9a888ac3b620",
"pluginType" : "LOADER",
"name" : "target-postgres",
"namespace" : "postgres_transferwise",
"variant" : "matatika",
"label" : "Postgres Warehouse",
"description" : "Postgres Warehouse is a data warehousing solution built on top of the Postgres database management system.\n\nPostgres Warehouse is designed to handle large volumes of data and complex queries, making it an ideal solution for businesses that need to store and analyze large amounts of data. It provides a number of features that are specifically tailored to data warehousing, such as columnar storage, parallel processing, and support for advanced analytics. Additionally, Postgres Warehouse is highly scalable, allowing businesses to easily add more resources as their data needs grow. Overall, Postgres Warehouse is a powerful and flexible data warehousing solution that can help businesses make better decisions by providing them with the insights they need to succeed.\n### Prerequisites\nThe process of obtaining the required settings for connecting to a Postgres Warehouse may vary depending on the specific setup and configuration of the database. However, here are some general ways to obtain each of the required settings:\n\n- User: The user is typically created when the database is set up. You can ask the database administrator or check the database documentation to find out the username.\n- Password: The password is also typically created when the database is set up. You can ask the database administrator or check the database documentation to find out the password.\n- Host: The host is the server where the database is located. You can ask the database administrator or check the database documentation to find out the host name or IP address.\n- Port: The port is the number that the database listens on for incoming connections. The default port for Postgres is 5432, but it may be different depending on the configuration. You can ask the database administrator or check the database documentation to find out the port number.\n- Database Name: The database name is the name of the specific database you want to connect to. You can ask the database administrator or check the database documentation to find out the database name.\n- Default Target Schema: The default target schema is the schema that you want to use as the default when connecting to the database. This may be set up by the database administrator or you may need to create it yourself. You can ask the database administrator or check the database documentation to find out the default target schema.",
"logoUrl" : "/assets/logos/loaders/postgres.png",
"hidden" : false,
"docs" : "https://www.matatika.com/data-details/target-postgres/",
"pipUrl" : "git+https://github.com/Matatika/[email protected]",
"repo" : "git+https://github.com/Matatika/[email protected]",
"capabilities" : [ ],
"select" : [ ],
"update" : { },
"vars" : { },
"settings" : [ {
"name" : "user",
"aliases" : [ "username" ],
"label" : "User",
"options" : [ ],
"kind" : "STRING",
"description" : "The username used to connect to the Postgres Warehouse.",
"hidden" : false,
"sensitive" : false,
"required" : "true",
"protected" : false
}, {
"name" : "password",
"aliases" : [ ],
"label" : "Password",
"options" : [ ],
"kind" : "STRING",
"description" : "The password used to authenticate the user.",
"hidden" : false,
"sensitive" : true,
"required" : "true",
"protected" : false
}, {
"name" : "host",
"aliases" : [ "address" ],
"label" : "Host",
"options" : [ ],
"kind" : "STRING",
"description" : "The hostname or IP address of the Postgres Warehouse server.",
"hidden" : false,
"sensitive" : false,
"required" : "true",
"protected" : false
}, {
"name" : "port",
"aliases" : [ ],
"label" : "Port",
"value" : "5432",
"options" : [ ],
"kind" : "INTEGER",
"description" : "The port number used to connect to the Postgres Warehouse server.",
"hidden" : false,
"sensitive" : false,
"required" : "true",
"protected" : false
}, {
"name" : "dbname",
"aliases" : [ "database" ],
"label" : "Database Name",
"options" : [ ],
"kind" : "STRING",
"description" : "The name of the database to connect to.",
"hidden" : false,
"sensitive" : false,
"required" : "true",
"protected" : false
}, {
"name" : "default_target_schema",
"aliases" : [ ],
"label" : "Default Target Schema",
"value" : "analytics",
"options" : [ ],
"kind" : "STRING",
"description" : "The default schema to use when writing data to the Postgres Warehouse.",
"hidden" : false,
"sensitive" : false,
"required" : "true",
"protected" : false
}, {
"name" : "state_schema",
"aliases" : [ ],
"label" : "State Schema",
"value" : "public",
"options" : [ ],
"kind" : "STRING",
"description" : "The default schema to use when writing Meltano state.",
"hidden" : false,
"sensitive" : false,
"required" : "true",
"protected" : false
}, {
"name" : "ssl",
"aliases" : [ ],
"label" : "SSL",
"value" : "false",
"options" : [ ],
"kind" : "BOOLEAN",
"description" : "Whether or not to use SSL encryption when connecting to the Postgres Warehouse.",
"hidden" : true,
"sensitive" : false,
"protected" : false,
"value_post_processor" : "STRINGIFY"
}, {
"name" : "batch_size_rows",
"aliases" : [ ],
"label" : "Batch Size Rows",
"value" : "100000",
"options" : [ ],
"kind" : "INTEGER",
"description" : "The number of rows to write to the Postgres Warehouse in each batch.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "underscore_camel_case_fields",
"aliases" : [ ],
"label" : "Underscore Camel Case Fields",
"value" : "true",
"options" : [ ],
"kind" : "BOOLEAN",
"description" : "Whether or not to convert field names from camel case to underscore-separated format.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "flush_all_streams",
"aliases" : [ ],
"label" : "Flush All Streams",
"value" : "true",
"options" : [ ],
"kind" : "BOOLEAN",
"description" : "Whether or not to flush all streams to the Postgres Warehouse before closing the connection.",
"hidden" : true,
"sensitive" : false,
"protected" : false
}, {
"name" : "parallelism",
"aliases" : [ ],
"label" : "Parallelism",
"value" : "-1",
"options" : [ ],
"kind" : "INTEGER",
"description" : "The number of threads to use when writing data to the Postgres Warehouse.",
"hidden" : true,
"sensitive" : false,
"protected" : false
}, {
"name" : "parallelism_max",
"aliases" : [ ],
"label" : "Max Parallelism",
"value" : "16",
"options" : [ ],
"kind" : "INTEGER",
"description" : "The maximum number of threads to use when writing data to the Postgres Warehouse.",
"hidden" : true,
"sensitive" : false,
"protected" : false
}, {
"name" : "default_target_schema_select_permission",
"aliases" : [ ],
"label" : "Default Target Schema Select Permission",
"options" : [ ],
"kind" : "STRING",
"description" : "The permission level required to select data from the default target schema.",
"hidden" : true,
"sensitive" : false,
"protected" : false
}, {
"name" : "schema_mapping",
"aliases" : [ ],
"label" : "Schema Mapping",
"options" : [ ],
"kind" : "STRING",
"description" : "A mapping of source schema names to target schema names.",
"hidden" : true,
"sensitive" : false,
"protected" : false
}, {
"name" : "add_metadata_columns",
"aliases" : [ ],
"label" : "Add Metadata Columns",
"value" : "true",
"options" : [ ],
"kind" : "BOOLEAN",
"description" : "Whether or not to add metadata columns to the target table.",
"hidden" : true,
"sensitive" : false,
"protected" : false
}, {
"name" : "hard_delete",
"aliases" : [ ],
"label" : "Hard Delete",
"value" : "false",
"options" : [ ],
"kind" : "BOOLEAN",
"description" : "Whether or not to perform hard deletes when deleting data from the Postgres Warehouse.",
"hidden" : true,
"sensitive" : false,
"protected" : false
}, {
"name" : "data_flattening_max_level",
"aliases" : [ ],
"label" : "Data Flattening Max Level",
"value" : "10",
"options" : [ ],
"kind" : "INTEGER",
"description" : "The maximum level of nested data structures to flatten when writing data to the Postgres Warehouse.",
"hidden" : true,
"sensitive" : false,
"protected" : false
}, {
"name" : "primary_key_required",
"aliases" : [ ],
"label" : "Primary Key Required",
"value" : "false",
"options" : [ ],
"kind" : "BOOLEAN",
"description" : "Whether or not a primary key is required for the target table.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "validate_records",
"aliases" : [ ],
"label" : "Validate Records",
"value" : "false",
"options" : [ ],
"kind" : "BOOLEAN",
"description" : "Whether or not to validate records before writing them to the Postgres Warehouse.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "temp_dir",
"aliases" : [ ],
"label" : "Temporary Directory",
"options" : [ ],
"kind" : "STRING",
"description" : "The directory to use for temporary files when writing data to the Postgres Warehouse.",
"hidden" : true,
"sensitive" : false,
"protected" : false
}, {
"name" : "ssh_tunnel.host",
"aliases" : [ ],
"label" : "SSH Tunnel Host",
"options" : [ ],
"kind" : "STRING",
"description" : "Host of the bastion host, this is the host we'll connect to via ssh",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "ssh_tunnel.port",
"aliases" : [ ],
"label" : "SSH Tunnel Port",
"value" : "22",
"options" : [ ],
"kind" : "INTEGER",
"description" : "Port to connect to bastion host",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "ssh_tunnel.private_key",
"aliases" : [ ],
"label" : "SSH Tunnel Private Key",
"options" : [ ],
"kind" : "PASSWORD",
"description" : "A base64 encoded Private Key for authentication to the bastion host w/ key pair auth",
"hidden" : false,
"sensitive" : true,
"encoding" : "BASE64",
"protected" : false
}, {
"name" : "ssh_tunnel.private_key_password",
"aliases" : [ ],
"label" : "SSH Tunnel Private Key Password",
"options" : [ ],
"kind" : "PASSWORD",
"description" : "Private Key Password, leave None if no password is set",
"hidden" : false,
"sensitive" : true,
"protected" : false
}, {
"name" : "ssh_tunnel.username",
"aliases" : [ ],
"label" : "SSH Tunnel Username",
"options" : [ ],
"kind" : "STRING",
"description" : "Username to connect to bastion host",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "ssh_tunnel.password",
"aliases" : [ ],
"label" : "SSH Tunnel Password",
"options" : [ ],
"kind" : "STRING",
"description" : "Password to connect to bastion host w/ basic auth",
"hidden" : false,
"sensitive" : true,
"protected" : false
}, {
"name" : "ssh_tunnel.host_setting_name",
"aliases" : [ ],
"value" : "host",
"options" : [ ],
"kind" : "STRING",
"hidden" : true,
"sensitive" : false,
"protected" : false
}, {
"name" : "ssh_tunnel.port_setting_name",
"aliases" : [ ],
"value" : "port",
"options" : [ ],
"kind" : "STRING",
"hidden" : true,
"sensitive" : false,
"protected" : false
} ],
"variants" : [ ],
"commands" : { },
"matatikaHidden" : false,
"requires" : [ ],
"fullDescription" : "Postgres Warehouse is a data warehousing solution built on top of the Postgres database management system.\n\nPostgres Warehouse is designed to handle large volumes of data and complex queries, making it an ideal solution for businesses that need to store and analyze large amounts of data. It provides a number of features that are specifically tailored to data warehousing, such as columnar storage, parallel processing, and support for advanced analytics. Additionally, Postgres Warehouse is highly scalable, allowing businesses to easily add more resources as their data needs grow. Overall, Postgres Warehouse is a powerful and flexible data warehousing solution that can help businesses make better decisions by providing them with the insights they need to succeed.\n### Prerequisites\nThe process of obtaining the required settings for connecting to a Postgres Warehouse may vary depending on the specific setup and configuration of the database. However, here are some general ways to obtain each of the required settings:\n\n- User: The user is typically created when the database is set up. You can ask the database administrator or check the database documentation to find out the username.\n- Password: The password is also typically created when the database is set up. You can ask the database administrator or check the database documentation to find out the password.\n- Host: The host is the server where the database is located. You can ask the database administrator or check the database documentation to find out the host name or IP address.\n- Port: The port is the number that the database listens on for incoming connections. The default port for Postgres is 5432, but it may be different depending on the configuration. You can ask the database administrator or check the database documentation to find out the port number.\n- Database Name: The database name is the name of the specific database you want to connect to. You can ask the database administrator or check the database documentation to find out the database name.\n- Default Target Schema: The default target schema is the schema that you want to use as the default when connecting to the database. This may be set up by the database administrator or you may need to create it yourself. You can ask the database administrator or check the database documentation to find out the default target schema.\n\n## Settings\n\n\n### User\n\nThe username used to connect to the Postgres Warehouse.\n\n### Password\n\nThe password used to authenticate the user.\n\n### Host\n\nThe hostname or IP address of the Postgres Warehouse server.\n\n### Port\n\nThe port number used to connect to the Postgres Warehouse server.\n\n### Database Name\n\nThe name of the database to connect to.\n\n### Default Target Schema\n\nThe default schema to use when writing data to the Postgres Warehouse.\n\n### State Schema\n\nThe default schema to use when writing Meltano state.\n\n### Batch Size Rows\n\nThe number of rows to write to the Postgres Warehouse in each batch.\n\n### Underscore Camel Case Fields\n\nWhether or not to convert field names from camel case to underscore-separated format.\n\n### Primary Key Required\n\nWhether or not a primary key is required for the target table.\n\n### Validate Records\n\nWhether or not to validate records before writing them to the Postgres Warehouse.\n\n### SSH Tunnel Host\n\nHost of the bastion host, this is the host we'll connect to via ssh\n\n### SSH Tunnel Port\n\nPort to connect to bastion host\n\n### SSH Tunnel Private Key\n\nA base64 encoded Private Key for authentication to the bastion host w/ key pair auth\n\n### SSH Tunnel Private Key Password\n\nPrivate Key Password, leave None if no password is set\n\n### SSH Tunnel Username\n\nUsername to connect to bastion host\n\n### SSH Tunnel Password\n\nPassword to connect to bastion host w/ basic auth",
"_links" : {
"self" : {
"href" : "https://app.matatika.com/api/dataplugins/82bf91fa-8adf-4421-863e-9a888ac3b620"
},
"update dataplugin" : {
"href" : "https://app.matatika.com/api/workspaces/52652da2-a249-407e-a8c6-4e045838a591/dataplugins/82bf91fa-8adf-4421-863e-9a888ac3b620",
"type" : "PUT"
}
}
}
},
"managed" : true,
"draft" : false,
"_links" : {
"self" : {
"href" : "https://app.matatika.com/api/datacomponents/a604ff5e-5fae-4578-ae4a-650091f015f3"
},
"update datacomponent" : {
"href" : "https://app.matatika.com/api/datacomponents/a604ff5e-5fae-4578-ae4a-650091f015f3"
},
"delete datacomponent" : {
"href" : "https://app.matatika.com/api/datacomponents/a604ff5e-5fae-4578-ae4a-650091f015f3"
}
}
}, {
"id" : "a0195563-9e63-480a-93e9-731670490a91",
"created" : "2025-06-19T10:19:42.489029",
"lastModified" : "2025-06-19T10:19:42.489029",
"name" : "dbt",
"properties" : { },
"commands" : {
"compile" : {
"args" : "compile",
"description" : "Generates executable SQL from source model, test, and analysis files. Compiled SQL files are written to the target/ directory."
},
"seed" : {
"args" : "seed",
"description" : "Load data from csv files into your data warehouse."
},
"test" : {
"args" : "test",
"description" : "Runs tests on data in deployed models."
},
"docs-generate" : {
"args" : "docs generate",
"description" : "Generate documentation artifacts for your project."
},
"deps" : {
"args" : "deps",
"description" : "Pull the most recent version of the dependencies listed in packages.yml"
},
"run" : {
"args" : "run",
"description" : "Compile SQL and execute against the current target database."
},
"clean" : {
"args" : "clean",
"description" : "Delete all folders in the clean-targets list (usually the dbt_modules and target directories.)"
},
"snapshot" : {
"args" : "snapshot",
"description" : "Execute snapshots defined in your project."
}
},
"dataPlugin" : "transformers/dbt--dbt-labs",
"_embedded" : {
"dataplugin" : {
"id" : "c3dce0d8-4025-4484-9018-964d5ae3a740",
"pluginType" : "TRANSFORMER",
"name" : "dbt",
"namespace" : "dbt",
"variant" : "dbt-labs",
"label" : "dbt",
"description" : " Power your project transformations with dbt™, a SQL-first transformation tool that enables analytics engineers to develop transformations with code.\n\n***Version Control and CI/CD***\n\nUse Matatika to deploy and promote changes between dev, UAT, and production environments.\n\n***Test and Document***\n\nUse Matatika to develop and test every model prior to production release, and share dynamically generated documentation with all stakeholders.\n\n***Develop***\n\nWrite modular data transformations in .sql – Matatika together with dbt handles the chore of dependency management. ",
"logoUrl" : "/assets/images/transformer/dbt.png",
"hidden" : false,
"docs" : "https://www.matatika.com/data-details/dbt/",
"pipUrl" : "dbt-core~=1.7.0 dbt-postgres~=1.7.0 dbt-snowflake~=1.7.0 dbt-bigquery~=1.7.0 certifi==2025.1.31",
"repo" : "https://github.com/dbt-labs/dbt-core",
"capabilities" : [ ],
"select" : [ ],
"update" : { },
"vars" : { },
"settings" : [ {
"name" : "project_dir",
"aliases" : [ ],
"value" : "$MELTANO_PROJECT_ROOT/transform",
"options" : [ ],
"kind" : "STRING",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "profiles_dir",
"aliases" : [ ],
"value" : "$MELTANO_PROJECT_ROOT/transform/profile",
"options" : [ ],
"kind" : "STRING",
"hidden" : false,
"sensitive" : false,
"env" : "DBT_PROFILES_DIR",
"protected" : false
}, {
"name" : "target",
"aliases" : [ ],
"value" : "$MELTANO_LOAD__DIALECT",
"options" : [ ],
"kind" : "STRING",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "source_schema",
"aliases" : [ ],
"value" : "$MELTANO_LOAD__TARGET_SCHEMA",
"options" : [ ],
"kind" : "STRING",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "target_schema",
"aliases" : [ ],
"value" : "analytics",
"options" : [ ],
"kind" : "STRING",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "target_path",
"aliases" : [ ],
"value" : "$MELTANO_PROJECT_ROOT/.meltano/transformers/dbt/target",
"options" : [ ],
"kind" : "STRING",
"description" : "Output path for dbt generated artifacts",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "models",
"aliases" : [ ],
"value" : "$MELTANO_TRANSFORM__PACKAGE_NAME $MELTANO_EXTRACTOR_NAMESPACE my_meltano_project",
"options" : [ ],
"kind" : "STRING",
"hidden" : false,
"sensitive" : false,
"protected" : false
} ],
"variants" : [ ],
"commands" : {
"compile" : {
"args" : "compile",
"description" : "Generates executable SQL from source model, test, and analysis files. Compiled SQL files are written to the target/ directory."
},
"seed" : {
"args" : "seed",
"description" : "Load data from csv files into your data warehouse."
},
"test" : {
"args" : "test",
"description" : "Runs tests on data in deployed models."
},
"docs-generate" : {
"args" : "docs generate",
"description" : "Generate documentation artifacts for your project."
},
"deps" : {
"args" : "deps",
"description" : "Pull the most recent version of the dependencies listed in packages.yml"
},
"run" : {
"args" : "run",
"description" : "Compile SQL and execute against the current target database."
},
"clean" : {
"args" : "clean",
"description" : "Delete all folders in the clean-targets list (usually the dbt_modules and target directories.)"
},
"snapshot" : {
"args" : "snapshot",
"description" : "Execute snapshots defined in your project."
}
},
"matatikaHidden" : false,
"requires" : [ {
"id" : "e6c1ad3d-ebf5-4c4a-b129-f68156b47555",
"pluginType" : "FILE",
"name" : "files-dbt",
"namespace" : "dbt",
"variant" : "matatika",
"description" : " Files dbt is a file bundle that automatically configures your project to run transforms with dbt.\nThe bundle includes template project configuration:\n\n- transform/models (directory)\n- transform/profile/profiles.yml\n- transform/dbt_project.yml\n- transform/.gitignore\n- transform/macros/centralize_test_failures.sql\n",
"hidden" : false,
"pipUrl" : "git+https://github.com/Matatika/[email protected]",
"repo" : "https://github.com/Matatika/files-dbt",
"capabilities" : [ ],
"select" : [ ],
"update" : {
"transform/profile/profiles.yml" : "true"
},
"vars" : { },
"settings" : [ ],
"variants" : [ ],
"commands" : { },
"matatikaHidden" : false,
"requires" : [ ],
"fullDescription" : " Files dbt is a file bundle that automatically configures your project to run transforms with dbt.\nThe bundle includes template project configuration:\n\n- transform/models (directory)\n- transform/profile/profiles.yml\n- transform/dbt_project.yml\n- transform/.gitignore\n- transform/macros/centralize_test_failures.sql\n"
} ],
"fullDescription" : " Power your project transformations with dbt™, a SQL-first transformation tool that enables analytics engineers to develop transformations with code.\n\n***Version Control and CI/CD***\n\nUse Matatika to deploy and promote changes between dev, UAT, and production environments.\n\n***Test and Document***\n\nUse Matatika to develop and test every model prior to production release, and share dynamically generated documentation with all stakeholders.\n\n***Develop***\n\nWrite modular data transformations in .sql – Matatika together with dbt handles the chore of dependency management. \n\n## Settings\n\n\n### target_path\n\nOutput path for dbt generated artifacts",
"_links" : {
"self" : {
"href" : "https://app.matatika.com/api/dataplugins/c3dce0d8-4025-4484-9018-964d5ae3a740"
},
"update dataplugin" : {
"href" : "https://app.matatika.com/api/workspaces/52652da2-a249-407e-a8c6-4e045838a591/dataplugins/c3dce0d8-4025-4484-9018-964d5ae3a740",
"type" : "PUT"
}
}
}
},
"managed" : true,
"draft" : false,
"_links" : {
"self" : {
"href" : "https://app.matatika.com/api/datacomponents/a0195563-9e63-480a-93e9-731670490a91"
},
"update datacomponent" : {
"href" : "https://app.matatika.com/api/datacomponents/a0195563-9e63-480a-93e9-731670490a91"
},
"delete datacomponent" : {
"href" : "https://app.matatika.com/api/datacomponents/a0195563-9e63-480a-93e9-731670490a91"
}
}
} ]
},
"_links" : {
"update pipeline" : {
"href" : "https://app.matatika.com/api/workspaces/52652da2-a249-407e-a8c6-4e045838a591/pipelines/180a0df1-e833-4aae-b6ef-ee98f0f07b50",
"type" : "PUT"
},
"delete pipeline" : {
"href" : "https://app.matatika.com/api/pipelines/180a0df1-e833-4aae-b6ef-ee98f0f07b50",
"type" : "DELETE"
},
"draft pipeline" : {
"href" : "https://app.matatika.com/api/workspaces/52652da2-a249-407e-a8c6-4e045838a591/pipelines/180a0df1-e833-4aae-b6ef-ee98f0f07b50/draft",
"type" : "PUT"
},
"self" : {
"href" : "https://app.matatika.com/api/pipelines/180a0df1-e833-4aae-b6ef-ee98f0f07b50"
},
"environment" : {
"href" : "https://app.matatika.com/api/pipelines/180a0df1-e833-4aae-b6ef-ee98f0f07b50/environment"
},
"jobs" : {
"href" : "https://app.matatika.com/api/pipelines/180a0df1-e833-4aae-b6ef-ee98f0f07b50/jobs",
"type" : "GET"
},
"metrics" : {
"href" : "https://app.matatika.com/api/pipelines/180a0df1-e833-4aae-b6ef-ee98f0f07b50/metrics"
},
"add subscription" : {
"href" : "https://app.matatika.com/api/pipelines/180a0df1-e833-4aae-b6ef-ee98f0f07b50/subscriptions"
},
"verify pipeline" : {
"href" : "https://app.matatika.com/api/pipelines/180a0df1-e833-4aae-b6ef-ee98f0f07b50/verification",
"type" : "POST"
},
"create job" : {
"href" : "https://app.matatika.com/api/pipelines/180a0df1-e833-4aae-b6ef-ee98f0f07b50/jobs",
"type" : "POST"
}
}
}
Properties
For each setting s in the datacomponents’ dataplugin settings for each
| Path | Type | Description |
|---|---|---|
s.name |
s.kind |
Refer to s.description |
- Any required settings not satisfied by a datacomponent property must be provided as a pipeline property
- Any settings that are already satisfied by a datacomponent property can be overridden by a pipeline property
Formats
Pipeline Status
| Value | Description |
|---|---|
READY |
The pipeline completed processing resource changes |
PROVISIONING |
The pipeline is processing resource changes |
FAILED |
The pipeline failed to process resource changes |
Requests
- View all pipelines in a workspace
- View a pipeline
- Initialise a pipeline in a workspace
- Create or update a pipeline in a workspace
- Create or update a pipeline as a draft
- Validate a pipeline configuration in a workspace
- Verify a pipeline
- Delete a pipeline
- View pipeline metrics
See Also
- View all running or completed jobs for a pipeline
- Create a job from a pipeline
- Subscribe to a pipeline
View all pipelines in a workspace
GET
/api/workspaces/{workspace-id}/pipelines
Returns all configured pipelines in the workspace {workspace-id}.
Prerequisites
- Workspace
{workspace-id}must exist
Request
Example Snippets
cURL
curl -H "Authorization: Bearer $ACCESS_TOKEN" 'https://app.matatika.com:443/api/workspaces/52652da2-a249-407e-a8c6-4e045838a591/pipelines' -i -X GET \
-H 'Accept: application/json, application/javascript, text/javascript, text/json'
Python (requests)
import requests
url = "https://app.matatika.com:443/api/workspaces/52652da2-a249-407e-a8c6-4e045838a591/pipelines"
headers = {
'Authorization': ACCESS_TOKEN
}
response = requests.request("GET", url, headers=headers)
print(response.text.encode('utf8'))
Response
200 OK
Pipeline collection with HAL links.
{
"_embedded" : {
"pipelines" : [ {
"id" : "f42e86bc-e5f7-4965-9ee9-92bdf7a5ac36",
"status" : "DRAFT",
"name" : "SIT-generated pipeline [2025-06-19T11:19:47.868367]",
"schedule" : "0 0 0 25 12 ?",
"timeout" : 0,
"maxRetries" : 0,
"created" : "2025-06-19T10:19:48.096569",
"lastModified" : "2025-06-19T10:19:48.09657",
"properties" : { },
"dataComponents" : [ "tap-google-analytics", "Warehouse", "dbt" ],
"actions" : [ ],
"triggeredBy" : [ ],
"_embedded" : {
"dataComponents" : [ {
"id" : "b6e29a0b-8836-4e3a-bbfd-f85163798009",
"created" : "2025-06-19T10:19:47.983989",
"lastModified" : "2025-06-19T10:19:47.98399",
"name" : "tap-google-analytics",
"properties" : { },
"commands" : { },
"dataPlugin" : "extractors/tap-google-analytics--matatika",
"_embedded" : {
"dataplugin" : {
"id" : "6b89da64-1538-4519-a188-5ff173208879",
"pluginType" : "EXTRACTOR",
"name" : "tap-google-analytics",
"namespace" : "tap_google_analytics",
"variant" : "matatika",
"label" : "Google Analytics",
"description" : "Google Analytics is a web analytics service that provides insights into website traffic and user behavior.\n\nGoogle Analytics allows website owners to track and analyze various metrics related to their website's performance, such as the number of visitors, pageviews, bounce rate, and average session duration. It also provides information on the demographics and interests of website visitors, as well as the sources of traffic, including organic search, paid search, social media, and referrals. This data can be used to optimize website content and marketing strategies, as well as to measure the effectiveness of advertising campaigns. Additionally, Google Analytics offers advanced features such as goal tracking, e-commerce tracking, and custom reporting, making it a powerful tool for businesses of all sizes.\n### Prerequisites\nTo obtain the OAuth identity provider authorization endpoint used to create and refresh tokens, you need to create a project in the Google API Console and enable the Google Analytics API. Then, you can create OAuth 2.0 credentials and configure the authorized redirect URIs. The authorization endpoint will be provided in the credentials.\n\nThe OAuth scopes you need to request access to depend on the specific data you want to access in Google Analytics. For example, if you want to read data from a specific view, you will need to request the \"https://www.googleapis.com/auth/analytics.readonly\" scope. You can find a list of available scopes in the Google Analytics API documentation.\n\nTo obtain the Access Token and OAuth Refresh Token, you need to authenticate the user and obtain their consent to access their Google Analytics data. This can be done using the Google Sign-In API or the OAuth 2.0 authorization flow. Once the user has granted access, you will receive an Access Token and a Refresh Token that you can use to make API requests.\n\nTo obtain the View ID, you need to log in to your Google Analytics account and navigate to the Admin section. From there, you can select the account, property, and view that you want to access and find the View ID in the View Settings.",
"logoUrl" : "/assets/images/datasource/tap-google-analytics.svg",
"hidden" : false,
"docs" : "https://www.matatika.com/docs/instant-insights/tap-google-analytics/",
"pipUrl" : "git+https://github.com/Matatika/[email protected]",
"repo" : "https://github.com/Matatika/tap-google-analytics",
"capabilities" : [ "DISCOVER", "CATALOG", "STATE" ],
"select" : [ ],
"update" : { },
"vars" : { },
"settings" : [ {
"name" : "oauth_credentials.authorization_url",
"aliases" : [ ],
"label" : "OAuth identity provider authorization endpoint used create and refresh tokens",
"value" : "https://oauth2.googleapis.com/token",
"options" : [ ],
"kind" : "STRING",
"description" : "The endpoint used to create and refresh OAuth tokens.",
"hidden" : true,
"sensitive" : false,
"required" : "true",
"protected" : false
}, {
"name" : "oauth_credentials.scope",
"aliases" : [ ],
"label" : "OAuth scopes we need to request access to",
"value" : "profile email https://www.googleapis.com/auth/analytics.readonly",
"options" : [ ],
"kind" : "STRING",
"description" : "The specific scopes we need to request access to in order to connect to Google Analytics.",
"hidden" : true,
"sensitive" : false,
"required" : "true",
"protected" : false
}, {
"name" : "oauth_credentials.access_token",
"aliases" : [ ],
"label" : "Access Token",
"options" : [ ],
"kind" : "STRING",
"description" : "The token used to authenticate and authorize API requests.",
"hidden" : true,
"sensitive" : false,
"required" : "true",
"protected" : false
}, {
"name" : "oauth_credentials.refresh_token",
"aliases" : [ ],
"label" : "OAuth Refresh Token",
"options" : [ ],
"kind" : "STRING",
"description" : "The token used to refresh the access token when it expires.",
"hidden" : true,
"sensitive" : false,
"required" : "true",
"protected" : false
}, {
"name" : "oauth_credentials.refresh_proxy_url",
"aliases" : [ ],
"label" : "Optional - will be called with 'oauth_credentials.refresh_token' to refresh the access token",
"options" : [ ],
"kind" : "STRING",
"description" : "An optional function that will be called to refresh the access token using the refresh token.",
"hidden" : true,
"sensitive" : false,
"protected" : false
}, {
"name" : "oauth_credentials.refresh_proxy_url_auth",
"aliases" : [ ],
"label" : "Optional - Sets Authorization header on 'oauth_credentials.refresh_url' request",
"options" : [ ],
"kind" : "STRING",
"description" : "An optional setting that sets the Authorization header on the refresh URL request.",
"hidden" : true,
"sensitive" : false,
"protected" : false
}, {
"name" : "oauth_credentials.client_id",
"aliases" : [ ],
"label" : "Optional - OAuth Client ID used if refresh_proxy_url not supplied",
"options" : [ ],
"kind" : "STRING",
"description" : "An optional OAuth Client ID used if the refresh proxy URL is not supplied.",
"hidden" : true,
"sensitive" : false,
"protected" : false
}, {
"name" : "oauth_credentials.client_secret",
"aliases" : [ ],
"label" : "Optional - OAuth Client Secret used if refresh_proxy_url not supplied",
"options" : [ ],
"kind" : "STRING",
"description" : "An optional OAuth Client Secret used if the refresh proxy URL is not supplied.",
"hidden" : true,
"sensitive" : false,
"protected" : false
}, {
"name" : "view_id",
"aliases" : [ ],
"label" : "View ID",
"options" : [ ],
"placeholder" : "Ex. 198343027",
"kind" : "STRING",
"description" : "The ID of the Google Analytics view to retrieve data from.",
"hidden" : false,
"sensitive" : false,
"required" : "true",
"protected" : false
}, {
"name" : "reports",
"aliases" : [ ],
"label" : "Reports",
"options" : [ ],
"placeholder" : "Ex. my_report_definition.json",
"kind" : "STRING",
"description" : "The specific reports to retrieve data from in the Google Analytics view.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "start_date",
"aliases" : [ ],
"label" : "Start date",
"options" : [ ],
"kind" : "DATE_ISO8601",
"description" : "The start date for the date range of data to retrieve.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "end_date",
"aliases" : [ ],
"label" : "End date",
"options" : [ ],
"kind" : "DATE_ISO8601",
"description" : "The end date for the date range of data to retrieve.",
"hidden" : false,
"sensitive" : false,
"protected" : false
} ],
"variants" : [ ],
"commands" : { },
"matatikaHidden" : false,
"requires" : [ ],
"fullDescription" : "Google Analytics is a web analytics service that provides insights into website traffic and user behavior.\n\nGoogle Analytics allows website owners to track and analyze various metrics related to their website's performance, such as the number of visitors, pageviews, bounce rate, and average session duration. It also provides information on the demographics and interests of website visitors, as well as the sources of traffic, including organic search, paid search, social media, and referrals. This data can be used to optimize website content and marketing strategies, as well as to measure the effectiveness of advertising campaigns. Additionally, Google Analytics offers advanced features such as goal tracking, e-commerce tracking, and custom reporting, making it a powerful tool for businesses of all sizes.\n### Prerequisites\nTo obtain the OAuth identity provider authorization endpoint used to create and refresh tokens, you need to create a project in the Google API Console and enable the Google Analytics API. Then, you can create OAuth 2.0 credentials and configure the authorized redirect URIs. The authorization endpoint will be provided in the credentials.\n\nThe OAuth scopes you need to request access to depend on the specific data you want to access in Google Analytics. For example, if you want to read data from a specific view, you will need to request the \"https://www.googleapis.com/auth/analytics.readonly\" scope. You can find a list of available scopes in the Google Analytics API documentation.\n\nTo obtain the Access Token and OAuth Refresh Token, you need to authenticate the user and obtain their consent to access their Google Analytics data. This can be done using the Google Sign-In API or the OAuth 2.0 authorization flow. Once the user has granted access, you will receive an Access Token and a Refresh Token that you can use to make API requests.\n\nTo obtain the View ID, you need to log in to your Google Analytics account and navigate to the Admin section. From there, you can select the account, property, and view that you want to access and find the View ID in the View Settings.\n\n## Settings\n\n\n### View ID\n\nThe ID of the Google Analytics view to retrieve data from.\n\n### Reports\n\nThe specific reports to retrieve data from in the Google Analytics view.\n\n### Start date\n\nThe start date for the date range of data to retrieve.\n\n### End date\n\nThe end date for the date range of data to retrieve.",
"_links" : {
"self" : {
"href" : "https://app.matatika.com/api/dataplugins/6b89da64-1538-4519-a188-5ff173208879"
},
"update dataplugin" : {
"href" : "https://app.matatika.com/api/workspaces/52652da2-a249-407e-a8c6-4e045838a591/dataplugins/6b89da64-1538-4519-a188-5ff173208879",
"type" : "PUT"
}
}
}
},
"managed" : false,
"draft" : true,
"_links" : {
"self" : {
"href" : "https://app.matatika.com/api/datacomponents/b6e29a0b-8836-4e3a-bbfd-f85163798009"
},
"update datacomponent" : {
"href" : "https://app.matatika.com/api/datacomponents/b6e29a0b-8836-4e3a-bbfd-f85163798009"
},
"delete datacomponent" : {
"href" : "https://app.matatika.com/api/datacomponents/b6e29a0b-8836-4e3a-bbfd-f85163798009"
}
}
}, {
"id" : "a604ff5e-5fae-4578-ae4a-650091f015f3",
"created" : "2025-06-19T10:19:42.42539",
"lastModified" : "2025-06-19T10:19:42.425391",
"name" : "Warehouse",
"properties" : {
"password" : "7PhOTBc5_2SMTn3on_ntw8iRA1",
"dbname" : "axvzzfd",
"default_target_schema" : "analytics",
"port" : "5432",
"host" : "sharp-banana2.postgres.database.azure.com",
"user" : "axvzzfd"
},
"commands" : { },
"dataPlugin" : "loaders/target-postgres--matatika",
"_embedded" : {
"dataplugin" : {
"id" : "82bf91fa-8adf-4421-863e-9a888ac3b620",
"pluginType" : "LOADER",
"name" : "target-postgres",
"namespace" : "postgres_transferwise",
"variant" : "matatika",
"label" : "Postgres Warehouse",
"description" : "Postgres Warehouse is a data warehousing solution built on top of the Postgres database management system.\n\nPostgres Warehouse is designed to handle large volumes of data and complex queries, making it an ideal solution for businesses that need to store and analyze large amounts of data. It provides a number of features that are specifically tailored to data warehousing, such as columnar storage, parallel processing, and support for advanced analytics. Additionally, Postgres Warehouse is highly scalable, allowing businesses to easily add more resources as their data needs grow. Overall, Postgres Warehouse is a powerful and flexible data warehousing solution that can help businesses make better decisions by providing them with the insights they need to succeed.\n### Prerequisites\nThe process of obtaining the required settings for connecting to a Postgres Warehouse may vary depending on the specific setup and configuration of the database. However, here are some general ways to obtain each of the required settings:\n\n- User: The user is typically created when the database is set up. You can ask the database administrator or check the database documentation to find out the username.\n- Password: The password is also typically created when the database is set up. You can ask the database administrator or check the database documentation to find out the password.\n- Host: The host is the server where the database is located. You can ask the database administrator or check the database documentation to find out the host name or IP address.\n- Port: The port is the number that the database listens on for incoming connections. The default port for Postgres is 5432, but it may be different depending on the configuration. You can ask the database administrator or check the database documentation to find out the port number.\n- Database Name: The database name is the name of the specific database you want to connect to. You can ask the database administrator or check the database documentation to find out the database name.\n- Default Target Schema: The default target schema is the schema that you want to use as the default when connecting to the database. This may be set up by the database administrator or you may need to create it yourself. You can ask the database administrator or check the database documentation to find out the default target schema.",
"logoUrl" : "/assets/logos/loaders/postgres.png",
"hidden" : false,
"docs" : "https://www.matatika.com/data-details/target-postgres/",
"pipUrl" : "git+https://github.com/Matatika/[email protected]",
"repo" : "git+https://github.com/Matatika/[email protected]",
"capabilities" : [ ],
"select" : [ ],
"update" : { },
"vars" : { },
"settings" : [ {
"name" : "user",
"aliases" : [ "username" ],
"label" : "User",
"options" : [ ],
"kind" : "STRING",
"description" : "The username used to connect to the Postgres Warehouse.",
"hidden" : false,
"sensitive" : false,
"required" : "true",
"protected" : false
}, {
"name" : "password",
"aliases" : [ ],
"label" : "Password",
"options" : [ ],
"kind" : "STRING",
"description" : "The password used to authenticate the user.",
"hidden" : false,
"sensitive" : true,
"required" : "true",
"protected" : false
}, {
"name" : "host",
"aliases" : [ "address" ],
"label" : "Host",
"options" : [ ],
"kind" : "STRING",
"description" : "The hostname or IP address of the Postgres Warehouse server.",
"hidden" : false,
"sensitive" : false,
"required" : "true",
"protected" : false
}, {
"name" : "port",
"aliases" : [ ],
"label" : "Port",
"value" : "5432",
"options" : [ ],
"kind" : "INTEGER",
"description" : "The port number used to connect to the Postgres Warehouse server.",
"hidden" : false,
"sensitive" : false,
"required" : "true",
"protected" : false
}, {
"name" : "dbname",
"aliases" : [ "database" ],
"label" : "Database Name",
"options" : [ ],
"kind" : "STRING",
"description" : "The name of the database to connect to.",
"hidden" : false,
"sensitive" : false,
"required" : "true",
"protected" : false
}, {
"name" : "default_target_schema",
"aliases" : [ ],
"label" : "Default Target Schema",
"value" : "analytics",
"options" : [ ],
"kind" : "STRING",
"description" : "The default schema to use when writing data to the Postgres Warehouse.",
"hidden" : false,
"sensitive" : false,
"required" : "true",
"protected" : false
}, {
"name" : "state_schema",
"aliases" : [ ],
"label" : "State Schema",
"value" : "public",
"options" : [ ],
"kind" : "STRING",
"description" : "The default schema to use when writing Meltano state.",
"hidden" : false,
"sensitive" : false,
"required" : "true",
"protected" : false
}, {
"name" : "ssl",
"aliases" : [ ],
"label" : "SSL",
"value" : "false",
"options" : [ ],
"kind" : "BOOLEAN",
"description" : "Whether or not to use SSL encryption when connecting to the Postgres Warehouse.",
"hidden" : true,
"sensitive" : false,
"protected" : false,
"value_post_processor" : "STRINGIFY"
}, {
"name" : "batch_size_rows",
"aliases" : [ ],
"label" : "Batch Size Rows",
"value" : "100000",
"options" : [ ],
"kind" : "INTEGER",
"description" : "The number of rows to write to the Postgres Warehouse in each batch.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "underscore_camel_case_fields",
"aliases" : [ ],
"label" : "Underscore Camel Case Fields",
"value" : "true",
"options" : [ ],
"kind" : "BOOLEAN",
"description" : "Whether or not to convert field names from camel case to underscore-separated format.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "flush_all_streams",
"aliases" : [ ],
"label" : "Flush All Streams",
"value" : "true",
"options" : [ ],
"kind" : "BOOLEAN",
"description" : "Whether or not to flush all streams to the Postgres Warehouse before closing the connection.",
"hidden" : true,
"sensitive" : false,
"protected" : false
}, {
"name" : "parallelism",
"aliases" : [ ],
"label" : "Parallelism",
"value" : "-1",
"options" : [ ],
"kind" : "INTEGER",
"description" : "The number of threads to use when writing data to the Postgres Warehouse.",
"hidden" : true,
"sensitive" : false,
"protected" : false
}, {
"name" : "parallelism_max",
"aliases" : [ ],
"label" : "Max Parallelism",
"value" : "16",
"options" : [ ],
"kind" : "INTEGER",
"description" : "The maximum number of threads to use when writing data to the Postgres Warehouse.",
"hidden" : true,
"sensitive" : false,
"protected" : false
}, {
"name" : "default_target_schema_select_permission",
"aliases" : [ ],
"label" : "Default Target Schema Select Permission",
"options" : [ ],
"kind" : "STRING",
"description" : "The permission level required to select data from the default target schema.",
"hidden" : true,
"sensitive" : false,
"protected" : false
}, {
"name" : "schema_mapping",
"aliases" : [ ],
"label" : "Schema Mapping",
"options" : [ ],
"kind" : "STRING",
"description" : "A mapping of source schema names to target schema names.",
"hidden" : true,
"sensitive" : false,
"protected" : false
}, {
"name" : "add_metadata_columns",
"aliases" : [ ],
"label" : "Add Metadata Columns",
"value" : "true",
"options" : [ ],
"kind" : "BOOLEAN",
"description" : "Whether or not to add metadata columns to the target table.",
"hidden" : true,
"sensitive" : false,
"protected" : false
}, {
"name" : "hard_delete",
"aliases" : [ ],
"label" : "Hard Delete",
"value" : "false",
"options" : [ ],
"kind" : "BOOLEAN",
"description" : "Whether or not to perform hard deletes when deleting data from the Postgres Warehouse.",
"hidden" : true,
"sensitive" : false,
"protected" : false
}, {
"name" : "data_flattening_max_level",
"aliases" : [ ],
"label" : "Data Flattening Max Level",
"value" : "10",
"options" : [ ],
"kind" : "INTEGER",
"description" : "The maximum level of nested data structures to flatten when writing data to the Postgres Warehouse.",
"hidden" : true,
"sensitive" : false,
"protected" : false
}, {
"name" : "primary_key_required",
"aliases" : [ ],
"label" : "Primary Key Required",
"value" : "false",
"options" : [ ],
"kind" : "BOOLEAN",
"description" : "Whether or not a primary key is required for the target table.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "validate_records",
"aliases" : [ ],
"label" : "Validate Records",
"value" : "false",
"options" : [ ],
"kind" : "BOOLEAN",
"description" : "Whether or not to validate records before writing them to the Postgres Warehouse.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "temp_dir",
"aliases" : [ ],
"label" : "Temporary Directory",
"options" : [ ],
"kind" : "STRING",
"description" : "The directory to use for temporary files when writing data to the Postgres Warehouse.",
"hidden" : true,
"sensitive" : false,
"protected" : false
}, {
"name" : "ssh_tunnel.host",
"aliases" : [ ],
"label" : "SSH Tunnel Host",
"options" : [ ],
"kind" : "STRING",
"description" : "Host of the bastion host, this is the host we'll connect to via ssh",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "ssh_tunnel.port",
"aliases" : [ ],
"label" : "SSH Tunnel Port",
"value" : "22",
"options" : [ ],
"kind" : "INTEGER",
"description" : "Port to connect to bastion host",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "ssh_tunnel.private_key",
"aliases" : [ ],
"label" : "SSH Tunnel Private Key",
"options" : [ ],
"kind" : "PASSWORD",
"description" : "A base64 encoded Private Key for authentication to the bastion host w/ key pair auth",
"hidden" : false,
"sensitive" : true,
"encoding" : "BASE64",
"protected" : false
}, {
"name" : "ssh_tunnel.private_key_password",
"aliases" : [ ],
"label" : "SSH Tunnel Private Key Password",
"options" : [ ],
"kind" : "PASSWORD",
"description" : "Private Key Password, leave None if no password is set",
"hidden" : false,
"sensitive" : true,
"protected" : false
}, {
"name" : "ssh_tunnel.username",
"aliases" : [ ],
"label" : "SSH Tunnel Username",
"options" : [ ],
"kind" : "STRING",
"description" : "Username to connect to bastion host",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "ssh_tunnel.password",
"aliases" : [ ],
"label" : "SSH Tunnel Password",
"options" : [ ],
"kind" : "STRING",
"description" : "Password to connect to bastion host w/ basic auth",
"hidden" : false,
"sensitive" : true,
"protected" : false
}, {
"name" : "ssh_tunnel.host_setting_name",
"aliases" : [ ],
"value" : "host",
"options" : [ ],
"kind" : "STRING",
"hidden" : true,
"sensitive" : false,
"protected" : false
}, {
"name" : "ssh_tunnel.port_setting_name",
"aliases" : [ ],
"value" : "port",
"options" : [ ],
"kind" : "STRING",
"hidden" : true,
"sensitive" : false,
"protected" : false
} ],
"variants" : [ ],
"commands" : { },
"matatikaHidden" : false,
"requires" : [ ],
"fullDescription" : "Postgres Warehouse is a data warehousing solution built on top of the Postgres database management system.\n\nPostgres Warehouse is designed to handle large volumes of data and complex queries, making it an ideal solution for businesses that need to store and analyze large amounts of data. It provides a number of features that are specifically tailored to data warehousing, such as columnar storage, parallel processing, and support for advanced analytics. Additionally, Postgres Warehouse is highly scalable, allowing businesses to easily add more resources as their data needs grow. Overall, Postgres Warehouse is a powerful and flexible data warehousing solution that can help businesses make better decisions by providing them with the insights they need to succeed.\n### Prerequisites\nThe process of obtaining the required settings for connecting to a Postgres Warehouse may vary depending on the specific setup and configuration of the database. However, here are some general ways to obtain each of the required settings:\n\n- User: The user is typically created when the database is set up. You can ask the database administrator or check the database documentation to find out the username.\n- Password: The password is also typically created when the database is set up. You can ask the database administrator or check the database documentation to find out the password.\n- Host: The host is the server where the database is located. You can ask the database administrator or check the database documentation to find out the host name or IP address.\n- Port: The port is the number that the database listens on for incoming connections. The default port for Postgres is 5432, but it may be different depending on the configuration. You can ask the database administrator or check the database documentation to find out the port number.\n- Database Name: The database name is the name of the specific database you want to connect to. You can ask the database administrator or check the database documentation to find out the database name.\n- Default Target Schema: The default target schema is the schema that you want to use as the default when connecting to the database. This may be set up by the database administrator or you may need to create it yourself. You can ask the database administrator or check the database documentation to find out the default target schema.\n\n## Settings\n\n\n### User\n\nThe username used to connect to the Postgres Warehouse.\n\n### Password\n\nThe password used to authenticate the user.\n\n### Host\n\nThe hostname or IP address of the Postgres Warehouse server.\n\n### Port\n\nThe port number used to connect to the Postgres Warehouse server.\n\n### Database Name\n\nThe name of the database to connect to.\n\n### Default Target Schema\n\nThe default schema to use when writing data to the Postgres Warehouse.\n\n### State Schema\n\nThe default schema to use when writing Meltano state.\n\n### Batch Size Rows\n\nThe number of rows to write to the Postgres Warehouse in each batch.\n\n### Underscore Camel Case Fields\n\nWhether or not to convert field names from camel case to underscore-separated format.\n\n### Primary Key Required\n\nWhether or not a primary key is required for the target table.\n\n### Validate Records\n\nWhether or not to validate records before writing them to the Postgres Warehouse.\n\n### SSH Tunnel Host\n\nHost of the bastion host, this is the host we'll connect to via ssh\n\n### SSH Tunnel Port\n\nPort to connect to bastion host\n\n### SSH Tunnel Private Key\n\nA base64 encoded Private Key for authentication to the bastion host w/ key pair auth\n\n### SSH Tunnel Private Key Password\n\nPrivate Key Password, leave None if no password is set\n\n### SSH Tunnel Username\n\nUsername to connect to bastion host\n\n### SSH Tunnel Password\n\nPassword to connect to bastion host w/ basic auth",
"_links" : {
"self" : {
"href" : "https://app.matatika.com/api/dataplugins/82bf91fa-8adf-4421-863e-9a888ac3b620"
},
"update dataplugin" : {
"href" : "https://app.matatika.com/api/workspaces/52652da2-a249-407e-a8c6-4e045838a591/dataplugins/82bf91fa-8adf-4421-863e-9a888ac3b620",
"type" : "PUT"
}
}
}
},
"managed" : true,
"draft" : false,
"_links" : {
"self" : {
"href" : "https://app.matatika.com/api/datacomponents/a604ff5e-5fae-4578-ae4a-650091f015f3"
},
"update datacomponent" : {
"href" : "https://app.matatika.com/api/datacomponents/a604ff5e-5fae-4578-ae4a-650091f015f3"
},
"delete datacomponent" : {
"href" : "https://app.matatika.com/api/datacomponents/a604ff5e-5fae-4578-ae4a-650091f015f3"
}
}
}, {
"id" : "a0195563-9e63-480a-93e9-731670490a91",
"created" : "2025-06-19T10:19:42.489029",
"lastModified" : "2025-06-19T10:19:42.489029",
"name" : "dbt",
"properties" : { },
"commands" : {
"compile" : {
"args" : "compile",
"description" : "Generates executable SQL from source model, test, and analysis files. Compiled SQL files are written to the target/ directory."
},
"seed" : {
"args" : "seed",
"description" : "Load data from csv files into your data warehouse."
},
"test" : {
"args" : "test",
"description" : "Runs tests on data in deployed models."
},
"docs-generate" : {
"args" : "docs generate",
"description" : "Generate documentation artifacts for your project."
},
"deps" : {
"args" : "deps",
"description" : "Pull the most recent version of the dependencies listed in packages.yml"
},
"run" : {
"args" : "run",
"description" : "Compile SQL and execute against the current target database."
},
"clean" : {
"args" : "clean",
"description" : "Delete all folders in the clean-targets list (usually the dbt_modules and target directories.)"
},
"snapshot" : {
"args" : "snapshot",
"description" : "Execute snapshots defined in your project."
}
},
"dataPlugin" : "transformers/dbt--dbt-labs",
"_embedded" : {
"dataplugin" : {
"id" : "c3dce0d8-4025-4484-9018-964d5ae3a740",
"pluginType" : "TRANSFORMER",
"name" : "dbt",
"namespace" : "dbt",
"variant" : "dbt-labs",
"label" : "dbt",
"description" : " Power your project transformations with dbt™, a SQL-first transformation tool that enables analytics engineers to develop transformations with code.\n\n***Version Control and CI/CD***\n\nUse Matatika to deploy and promote changes between dev, UAT, and production environments.\n\n***Test and Document***\n\nUse Matatika to develop and test every model prior to production release, and share dynamically generated documentation with all stakeholders.\n\n***Develop***\n\nWrite modular data transformations in .sql – Matatika together with dbt handles the chore of dependency management. ",
"logoUrl" : "/assets/images/transformer/dbt.png",
"hidden" : false,
"docs" : "https://www.matatika.com/data-details/dbt/",
"pipUrl" : "dbt-core~=1.7.0 dbt-postgres~=1.7.0 dbt-snowflake~=1.7.0 dbt-bigquery~=1.7.0 certifi==2025.1.31",
"repo" : "https://github.com/dbt-labs/dbt-core",
"capabilities" : [ ],
"select" : [ ],
"update" : { },
"vars" : { },
"settings" : [ {
"name" : "project_dir",
"aliases" : [ ],
"value" : "$MELTANO_PROJECT_ROOT/transform",
"options" : [ ],
"kind" : "STRING",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "profiles_dir",
"aliases" : [ ],
"value" : "$MELTANO_PROJECT_ROOT/transform/profile",
"options" : [ ],
"kind" : "STRING",
"hidden" : false,
"sensitive" : false,
"env" : "DBT_PROFILES_DIR",
"protected" : false
}, {
"name" : "target",
"aliases" : [ ],
"value" : "$MELTANO_LOAD__DIALECT",
"options" : [ ],
"kind" : "STRING",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "source_schema",
"aliases" : [ ],
"value" : "$MELTANO_LOAD__TARGET_SCHEMA",
"options" : [ ],
"kind" : "STRING",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "target_schema",
"aliases" : [ ],
"value" : "analytics",
"options" : [ ],
"kind" : "STRING",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "target_path",
"aliases" : [ ],
"value" : "$MELTANO_PROJECT_ROOT/.meltano/transformers/dbt/target",
"options" : [ ],
"kind" : "STRING",
"description" : "Output path for dbt generated artifacts",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "models",
"aliases" : [ ],
"value" : "$MELTANO_TRANSFORM__PACKAGE_NAME $MELTANO_EXTRACTOR_NAMESPACE my_meltano_project",
"options" : [ ],
"kind" : "STRING",
"hidden" : false,
"sensitive" : false,
"protected" : false
} ],
"variants" : [ ],
"commands" : {
"compile" : {
"args" : "compile",
"description" : "Generates executable SQL from source model, test, and analysis files. Compiled SQL files are written to the target/ directory."
},
"seed" : {
"args" : "seed",
"description" : "Load data from csv files into your data warehouse."
},
"test" : {
"args" : "test",
"description" : "Runs tests on data in deployed models."
},
"docs-generate" : {
"args" : "docs generate",
"description" : "Generate documentation artifacts for your project."
},
"deps" : {
"args" : "deps",
"description" : "Pull the most recent version of the dependencies listed in packages.yml"
},
"run" : {
"args" : "run",
"description" : "Compile SQL and execute against the current target database."
},
"clean" : {
"args" : "clean",
"description" : "Delete all folders in the clean-targets list (usually the dbt_modules and target directories.)"
},
"snapshot" : {
"args" : "snapshot",
"description" : "Execute snapshots defined in your project."
}
},
"matatikaHidden" : false,
"requires" : [ {
"id" : "e6c1ad3d-ebf5-4c4a-b129-f68156b47555",
"pluginType" : "FILE",
"name" : "files-dbt",
"namespace" : "dbt",
"variant" : "matatika",
"description" : " Files dbt is a file bundle that automatically configures your project to run transforms with dbt.\nThe bundle includes template project configuration:\n\n- transform/models (directory)\n- transform/profile/profiles.yml\n- transform/dbt_project.yml\n- transform/.gitignore\n- transform/macros/centralize_test_failures.sql\n",
"hidden" : false,
"pipUrl" : "git+https://github.com/Matatika/[email protected]",
"repo" : "https://github.com/Matatika/files-dbt",
"capabilities" : [ ],
"select" : [ ],
"update" : {
"transform/profile/profiles.yml" : "true"
},
"vars" : { },
"settings" : [ ],
"variants" : [ ],
"commands" : { },
"matatikaHidden" : false,
"requires" : [ ],
"fullDescription" : " Files dbt is a file bundle that automatically configures your project to run transforms with dbt.\nThe bundle includes template project configuration:\n\n- transform/models (directory)\n- transform/profile/profiles.yml\n- transform/dbt_project.yml\n- transform/.gitignore\n- transform/macros/centralize_test_failures.sql\n"
} ],
"fullDescription" : " Power your project transformations with dbt™, a SQL-first transformation tool that enables analytics engineers to develop transformations with code.\n\n***Version Control and CI/CD***\n\nUse Matatika to deploy and promote changes between dev, UAT, and production environments.\n\n***Test and Document***\n\nUse Matatika to develop and test every model prior to production release, and share dynamically generated documentation with all stakeholders.\n\n***Develop***\n\nWrite modular data transformations in .sql – Matatika together with dbt handles the chore of dependency management. \n\n## Settings\n\n\n### target_path\n\nOutput path for dbt generated artifacts",
"_links" : {
"self" : {
"href" : "https://app.matatika.com/api/dataplugins/c3dce0d8-4025-4484-9018-964d5ae3a740"
},
"update dataplugin" : {
"href" : "https://app.matatika.com/api/workspaces/52652da2-a249-407e-a8c6-4e045838a591/dataplugins/c3dce0d8-4025-4484-9018-964d5ae3a740",
"type" : "PUT"
}
}
}
},
"managed" : true,
"draft" : false,
"_links" : {
"self" : {
"href" : "https://app.matatika.com/api/datacomponents/a0195563-9e63-480a-93e9-731670490a91"
},
"update datacomponent" : {
"href" : "https://app.matatika.com/api/datacomponents/a0195563-9e63-480a-93e9-731670490a91"
},
"delete datacomponent" : {
"href" : "https://app.matatika.com/api/datacomponents/a0195563-9e63-480a-93e9-731670490a91"
}
}
} ]
},
"_links" : {
"update pipeline" : {
"href" : "https://app.matatika.com/api/workspaces/52652da2-a249-407e-a8c6-4e045838a591/pipelines/f42e86bc-e5f7-4965-9ee9-92bdf7a5ac36",
"type" : "PUT"
},
"delete pipeline" : {
"href" : "https://app.matatika.com/api/pipelines/f42e86bc-e5f7-4965-9ee9-92bdf7a5ac36",
"type" : "DELETE"
},
"draft pipeline" : {
"href" : "https://app.matatika.com/api/workspaces/52652da2-a249-407e-a8c6-4e045838a591/pipelines/f42e86bc-e5f7-4965-9ee9-92bdf7a5ac36/draft",
"type" : "PUT"
},
"self" : {
"href" : "https://app.matatika.com/api/pipelines/f42e86bc-e5f7-4965-9ee9-92bdf7a5ac36"
},
"environment" : {
"href" : "https://app.matatika.com/api/pipelines/f42e86bc-e5f7-4965-9ee9-92bdf7a5ac36/environment"
},
"jobs" : {
"href" : "https://app.matatika.com/api/pipelines/f42e86bc-e5f7-4965-9ee9-92bdf7a5ac36/jobs",
"type" : "GET"
},
"metrics" : {
"href" : "https://app.matatika.com/api/pipelines/f42e86bc-e5f7-4965-9ee9-92bdf7a5ac36/metrics"
},
"add subscription" : {
"href" : "https://app.matatika.com/api/pipelines/f42e86bc-e5f7-4965-9ee9-92bdf7a5ac36/subscriptions"
},
"verify pipeline" : {
"href" : "https://app.matatika.com/api/pipelines/f42e86bc-e5f7-4965-9ee9-92bdf7a5ac36/verification",
"type" : "POST"
}
}
}, {
"id" : "180a0df1-e833-4aae-b6ef-ee98f0f07b50",
"status" : "READY",
"name" : "SIT-generated pipeline [2025-06-19T11:19:48.350863] (updated)",
"timeout" : 0,
"maxRetries" : 0,
"created" : "2025-06-19T10:19:48.519054",
"lastModified" : "2025-06-19T10:19:48.519056",
"properties" : { },
"dataComponents" : [ "tap-google-analytics", "Warehouse", "dbt" ],
"actions" : [ ],
"triggeredBy" : [ ],
"_embedded" : {
"dataComponents" : [ {
"id" : "b6e29a0b-8836-4e3a-bbfd-f85163798009",
"created" : "2025-06-19T10:19:47.983989",
"lastModified" : "2025-06-19T10:19:47.98399",
"name" : "tap-google-analytics",
"properties" : { },
"commands" : { },
"dataPlugin" : "extractors/tap-google-analytics--matatika",
"_embedded" : {
"dataplugin" : {
"id" : "6b89da64-1538-4519-a188-5ff173208879",
"pluginType" : "EXTRACTOR",
"name" : "tap-google-analytics",
"namespace" : "tap_google_analytics",
"variant" : "matatika",
"label" : "Google Analytics",
"description" : "Google Analytics is a web analytics service that provides insights into website traffic and user behavior.\n\nGoogle Analytics allows website owners to track and analyze various metrics related to their website's performance, such as the number of visitors, pageviews, bounce rate, and average session duration. It also provides information on the demographics and interests of website visitors, as well as the sources of traffic, including organic search, paid search, social media, and referrals. This data can be used to optimize website content and marketing strategies, as well as to measure the effectiveness of advertising campaigns. Additionally, Google Analytics offers advanced features such as goal tracking, e-commerce tracking, and custom reporting, making it a powerful tool for businesses of all sizes.\n### Prerequisites\nTo obtain the OAuth identity provider authorization endpoint used to create and refresh tokens, you need to create a project in the Google API Console and enable the Google Analytics API. Then, you can create OAuth 2.0 credentials and configure the authorized redirect URIs. The authorization endpoint will be provided in the credentials.\n\nThe OAuth scopes you need to request access to depend on the specific data you want to access in Google Analytics. For example, if you want to read data from a specific view, you will need to request the \"https://www.googleapis.com/auth/analytics.readonly\" scope. You can find a list of available scopes in the Google Analytics API documentation.\n\nTo obtain the Access Token and OAuth Refresh Token, you need to authenticate the user and obtain their consent to access their Google Analytics data. This can be done using the Google Sign-In API or the OAuth 2.0 authorization flow. Once the user has granted access, you will receive an Access Token and a Refresh Token that you can use to make API requests.\n\nTo obtain the View ID, you need to log in to your Google Analytics account and navigate to the Admin section. From there, you can select the account, property, and view that you want to access and find the View ID in the View Settings.",
"logoUrl" : "/assets/images/datasource/tap-google-analytics.svg",
"hidden" : false,
"docs" : "https://www.matatika.com/docs/instant-insights/tap-google-analytics/",
"pipUrl" : "git+https://github.com/Matatika/[email protected]",
"repo" : "https://github.com/Matatika/tap-google-analytics",
"capabilities" : [ "DISCOVER", "CATALOG", "STATE" ],
"select" : [ ],
"update" : { },
"vars" : { },
"settings" : [ {
"name" : "oauth_credentials.authorization_url",
"aliases" : [ ],
"label" : "OAuth identity provider authorization endpoint used create and refresh tokens",
"value" : "https://oauth2.googleapis.com/token",
"options" : [ ],
"kind" : "STRING",
"description" : "The endpoint used to create and refresh OAuth tokens.",
"hidden" : true,
"sensitive" : false,
"required" : "true",
"protected" : false
}, {
"name" : "oauth_credentials.scope",
"aliases" : [ ],
"label" : "OAuth scopes we need to request access to",
"value" : "profile email https://www.googleapis.com/auth/analytics.readonly",
"options" : [ ],
"kind" : "STRING",
"description" : "The specific scopes we need to request access to in order to connect to Google Analytics.",
"hidden" : true,
"sensitive" : false,
"required" : "true",
"protected" : false
}, {
"name" : "oauth_credentials.access_token",
"aliases" : [ ],
"label" : "Access Token",
"options" : [ ],
"kind" : "STRING",
"description" : "The token used to authenticate and authorize API requests.",
"hidden" : true,
"sensitive" : false,
"required" : "true",
"protected" : false
}, {
"name" : "oauth_credentials.refresh_token",
"aliases" : [ ],
"label" : "OAuth Refresh Token",
"options" : [ ],
"kind" : "STRING",
"description" : "The token used to refresh the access token when it expires.",
"hidden" : true,
"sensitive" : false,
"required" : "true",
"protected" : false
}, {
"name" : "oauth_credentials.refresh_proxy_url",
"aliases" : [ ],
"label" : "Optional - will be called with 'oauth_credentials.refresh_token' to refresh the access token",
"options" : [ ],
"kind" : "STRING",
"description" : "An optional function that will be called to refresh the access token using the refresh token.",
"hidden" : true,
"sensitive" : false,
"protected" : false
}, {
"name" : "oauth_credentials.refresh_proxy_url_auth",
"aliases" : [ ],
"label" : "Optional - Sets Authorization header on 'oauth_credentials.refresh_url' request",
"options" : [ ],
"kind" : "STRING",
"description" : "An optional setting that sets the Authorization header on the refresh URL request.",
"hidden" : true,
"sensitive" : false,
"protected" : false
}, {
"name" : "oauth_credentials.client_id",
"aliases" : [ ],
"label" : "Optional - OAuth Client ID used if refresh_proxy_url not supplied",
"options" : [ ],
"kind" : "STRING",
"description" : "An optional OAuth Client ID used if the refresh proxy URL is not supplied.",
"hidden" : true,
"sensitive" : false,
"protected" : false
}, {
"name" : "oauth_credentials.client_secret",
"aliases" : [ ],
"label" : "Optional - OAuth Client Secret used if refresh_proxy_url not supplied",
"options" : [ ],
"kind" : "STRING",
"description" : "An optional OAuth Client Secret used if the refresh proxy URL is not supplied.",
"hidden" : true,
"sensitive" : false,
"protected" : false
}, {
"name" : "view_id",
"aliases" : [ ],
"label" : "View ID",
"options" : [ ],
"placeholder" : "Ex. 198343027",
"kind" : "STRING",
"description" : "The ID of the Google Analytics view to retrieve data from.",
"hidden" : false,
"sensitive" : false,
"required" : "true",
"protected" : false
}, {
"name" : "reports",
"aliases" : [ ],
"label" : "Reports",
"options" : [ ],
"placeholder" : "Ex. my_report_definition.json",
"kind" : "STRING",
"description" : "The specific reports to retrieve data from in the Google Analytics view.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "start_date",
"aliases" : [ ],
"label" : "Start date",
"options" : [ ],
"kind" : "DATE_ISO8601",
"description" : "The start date for the date range of data to retrieve.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "end_date",
"aliases" : [ ],
"label" : "End date",
"options" : [ ],
"kind" : "DATE_ISO8601",
"description" : "The end date for the date range of data to retrieve.",
"hidden" : false,
"sensitive" : false,
"protected" : false
} ],
"variants" : [ ],
"commands" : { },
"matatikaHidden" : false,
"requires" : [ ],
"fullDescription" : "Google Analytics is a web analytics service that provides insights into website traffic and user behavior.\n\nGoogle Analytics allows website owners to track and analyze various metrics related to their website's performance, such as the number of visitors, pageviews, bounce rate, and average session duration. It also provides information on the demographics and interests of website visitors, as well as the sources of traffic, including organic search, paid search, social media, and referrals. This data can be used to optimize website content and marketing strategies, as well as to measure the effectiveness of advertising campaigns. Additionally, Google Analytics offers advanced features such as goal tracking, e-commerce tracking, and custom reporting, making it a powerful tool for businesses of all sizes.\n### Prerequisites\nTo obtain the OAuth identity provider authorization endpoint used to create and refresh tokens, you need to create a project in the Google API Console and enable the Google Analytics API. Then, you can create OAuth 2.0 credentials and configure the authorized redirect URIs. The authorization endpoint will be provided in the credentials.\n\nThe OAuth scopes you need to request access to depend on the specific data you want to access in Google Analytics. For example, if you want to read data from a specific view, you will need to request the \"https://www.googleapis.com/auth/analytics.readonly\" scope. You can find a list of available scopes in the Google Analytics API documentation.\n\nTo obtain the Access Token and OAuth Refresh Token, you need to authenticate the user and obtain their consent to access their Google Analytics data. This can be done using the Google Sign-In API or the OAuth 2.0 authorization flow. Once the user has granted access, you will receive an Access Token and a Refresh Token that you can use to make API requests.\n\nTo obtain the View ID, you need to log in to your Google Analytics account and navigate to the Admin section. From there, you can select the account, property, and view that you want to access and find the View ID in the View Settings.\n\n## Settings\n\n\n### View ID\n\nThe ID of the Google Analytics view to retrieve data from.\n\n### Reports\n\nThe specific reports to retrieve data from in the Google Analytics view.\n\n### Start date\n\nThe start date for the date range of data to retrieve.\n\n### End date\n\nThe end date for the date range of data to retrieve.",
"_links" : {
"self" : {
"href" : "https://app.matatika.com/api/dataplugins/6b89da64-1538-4519-a188-5ff173208879"
},
"update dataplugin" : {
"href" : "https://app.matatika.com/api/workspaces/52652da2-a249-407e-a8c6-4e045838a591/dataplugins/6b89da64-1538-4519-a188-5ff173208879",
"type" : "PUT"
}
}
}
},
"managed" : false,
"draft" : true,
"_links" : {
"self" : {
"href" : "https://app.matatika.com/api/datacomponents/b6e29a0b-8836-4e3a-bbfd-f85163798009"
},
"update datacomponent" : {
"href" : "https://app.matatika.com/api/datacomponents/b6e29a0b-8836-4e3a-bbfd-f85163798009"
},
"delete datacomponent" : {
"href" : "https://app.matatika.com/api/datacomponents/b6e29a0b-8836-4e3a-bbfd-f85163798009"
}
}
}, {
"id" : "a604ff5e-5fae-4578-ae4a-650091f015f3",
"created" : "2025-06-19T10:19:42.42539",
"lastModified" : "2025-06-19T10:19:42.425391",
"name" : "Warehouse",
"properties" : {
"password" : "7PhOTBc5_2SMTn3on_ntw8iRA1",
"dbname" : "axvzzfd",
"default_target_schema" : "analytics",
"port" : "5432",
"host" : "sharp-banana2.postgres.database.azure.com",
"user" : "axvzzfd"
},
"commands" : { },
"dataPlugin" : "loaders/target-postgres--matatika",
"_embedded" : {
"dataplugin" : {
"id" : "82bf91fa-8adf-4421-863e-9a888ac3b620",
"pluginType" : "LOADER",
"name" : "target-postgres",
"namespace" : "postgres_transferwise",
"variant" : "matatika",
"label" : "Postgres Warehouse",
"description" : "Postgres Warehouse is a data warehousing solution built on top of the Postgres database management system.\n\nPostgres Warehouse is designed to handle large volumes of data and complex queries, making it an ideal solution for businesses that need to store and analyze large amounts of data. It provides a number of features that are specifically tailored to data warehousing, such as columnar storage, parallel processing, and support for advanced analytics. Additionally, Postgres Warehouse is highly scalable, allowing businesses to easily add more resources as their data needs grow. Overall, Postgres Warehouse is a powerful and flexible data warehousing solution that can help businesses make better decisions by providing them with the insights they need to succeed.\n### Prerequisites\nThe process of obtaining the required settings for connecting to a Postgres Warehouse may vary depending on the specific setup and configuration of the database. However, here are some general ways to obtain each of the required settings:\n\n- User: The user is typically created when the database is set up. You can ask the database administrator or check the database documentation to find out the username.\n- Password: The password is also typically created when the database is set up. You can ask the database administrator or check the database documentation to find out the password.\n- Host: The host is the server where the database is located. You can ask the database administrator or check the database documentation to find out the host name or IP address.\n- Port: The port is the number that the database listens on for incoming connections. The default port for Postgres is 5432, but it may be different depending on the configuration. You can ask the database administrator or check the database documentation to find out the port number.\n- Database Name: The database name is the name of the specific database you want to connect to. You can ask the database administrator or check the database documentation to find out the database name.\n- Default Target Schema: The default target schema is the schema that you want to use as the default when connecting to the database. This may be set up by the database administrator or you may need to create it yourself. You can ask the database administrator or check the database documentation to find out the default target schema.",
"logoUrl" : "/assets/logos/loaders/postgres.png",
"hidden" : false,
"docs" : "https://www.matatika.com/data-details/target-postgres/",
"pipUrl" : "git+https://github.com/Matatika/[email protected]",
"repo" : "git+https://github.com/Matatika/[email protected]",
"capabilities" : [ ],
"select" : [ ],
"update" : { },
"vars" : { },
"settings" : [ {
"name" : "user",
"aliases" : [ "username" ],
"label" : "User",
"options" : [ ],
"kind" : "STRING",
"description" : "The username used to connect to the Postgres Warehouse.",
"hidden" : false,
"sensitive" : false,
"required" : "true",
"protected" : false
}, {
"name" : "password",
"aliases" : [ ],
"label" : "Password",
"options" : [ ],
"kind" : "STRING",
"description" : "The password used to authenticate the user.",
"hidden" : false,
"sensitive" : true,
"required" : "true",
"protected" : false
}, {
"name" : "host",
"aliases" : [ "address" ],
"label" : "Host",
"options" : [ ],
"kind" : "STRING",
"description" : "The hostname or IP address of the Postgres Warehouse server.",
"hidden" : false,
"sensitive" : false,
"required" : "true",
"protected" : false
}, {
"name" : "port",
"aliases" : [ ],
"label" : "Port",
"value" : "5432",
"options" : [ ],
"kind" : "INTEGER",
"description" : "The port number used to connect to the Postgres Warehouse server.",
"hidden" : false,
"sensitive" : false,
"required" : "true",
"protected" : false
}, {
"name" : "dbname",
"aliases" : [ "database" ],
"label" : "Database Name",
"options" : [ ],
"kind" : "STRING",
"description" : "The name of the database to connect to.",
"hidden" : false,
"sensitive" : false,
"required" : "true",
"protected" : false
}, {
"name" : "default_target_schema",
"aliases" : [ ],
"label" : "Default Target Schema",
"value" : "analytics",
"options" : [ ],
"kind" : "STRING",
"description" : "The default schema to use when writing data to the Postgres Warehouse.",
"hidden" : false,
"sensitive" : false,
"required" : "true",
"protected" : false
}, {
"name" : "state_schema",
"aliases" : [ ],
"label" : "State Schema",
"value" : "public",
"options" : [ ],
"kind" : "STRING",
"description" : "The default schema to use when writing Meltano state.",
"hidden" : false,
"sensitive" : false,
"required" : "true",
"protected" : false
}, {
"name" : "ssl",
"aliases" : [ ],
"label" : "SSL",
"value" : "false",
"options" : [ ],
"kind" : "BOOLEAN",
"description" : "Whether or not to use SSL encryption when connecting to the Postgres Warehouse.",
"hidden" : true,
"sensitive" : false,
"protected" : false,
"value_post_processor" : "STRINGIFY"
}, {
"name" : "batch_size_rows",
"aliases" : [ ],
"label" : "Batch Size Rows",
"value" : "100000",
"options" : [ ],
"kind" : "INTEGER",
"description" : "The number of rows to write to the Postgres Warehouse in each batch.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "underscore_camel_case_fields",
"aliases" : [ ],
"label" : "Underscore Camel Case Fields",
"value" : "true",
"options" : [ ],
"kind" : "BOOLEAN",
"description" : "Whether or not to convert field names from camel case to underscore-separated format.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "flush_all_streams",
"aliases" : [ ],
"label" : "Flush All Streams",
"value" : "true",
"options" : [ ],
"kind" : "BOOLEAN",
"description" : "Whether or not to flush all streams to the Postgres Warehouse before closing the connection.",
"hidden" : true,
"sensitive" : false,
"protected" : false
}, {
"name" : "parallelism",
"aliases" : [ ],
"label" : "Parallelism",
"value" : "-1",
"options" : [ ],
"kind" : "INTEGER",
"description" : "The number of threads to use when writing data to the Postgres Warehouse.",
"hidden" : true,
"sensitive" : false,
"protected" : false
}, {
"name" : "parallelism_max",
"aliases" : [ ],
"label" : "Max Parallelism",
"value" : "16",
"options" : [ ],
"kind" : "INTEGER",
"description" : "The maximum number of threads to use when writing data to the Postgres Warehouse.",
"hidden" : true,
"sensitive" : false,
"protected" : false
}, {
"name" : "default_target_schema_select_permission",
"aliases" : [ ],
"label" : "Default Target Schema Select Permission",
"options" : [ ],
"kind" : "STRING",
"description" : "The permission level required to select data from the default target schema.",
"hidden" : true,
"sensitive" : false,
"protected" : false
}, {
"name" : "schema_mapping",
"aliases" : [ ],
"label" : "Schema Mapping",
"options" : [ ],
"kind" : "STRING",
"description" : "A mapping of source schema names to target schema names.",
"hidden" : true,
"sensitive" : false,
"protected" : false
}, {
"name" : "add_metadata_columns",
"aliases" : [ ],
"label" : "Add Metadata Columns",
"value" : "true",
"options" : [ ],
"kind" : "BOOLEAN",
"description" : "Whether or not to add metadata columns to the target table.",
"hidden" : true,
"sensitive" : false,
"protected" : false
}, {
"name" : "hard_delete",
"aliases" : [ ],
"label" : "Hard Delete",
"value" : "false",
"options" : [ ],
"kind" : "BOOLEAN",
"description" : "Whether or not to perform hard deletes when deleting data from the Postgres Warehouse.",
"hidden" : true,
"sensitive" : false,
"protected" : false
}, {
"name" : "data_flattening_max_level",
"aliases" : [ ],
"label" : "Data Flattening Max Level",
"value" : "10",
"options" : [ ],
"kind" : "INTEGER",
"description" : "The maximum level of nested data structures to flatten when writing data to the Postgres Warehouse.",
"hidden" : true,
"sensitive" : false,
"protected" : false
}, {
"name" : "primary_key_required",
"aliases" : [ ],
"label" : "Primary Key Required",
"value" : "false",
"options" : [ ],
"kind" : "BOOLEAN",
"description" : "Whether or not a primary key is required for the target table.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "validate_records",
"aliases" : [ ],
"label" : "Validate Records",
"value" : "false",
"options" : [ ],
"kind" : "BOOLEAN",
"description" : "Whether or not to validate records before writing them to the Postgres Warehouse.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "temp_dir",
"aliases" : [ ],
"label" : "Temporary Directory",
"options" : [ ],
"kind" : "STRING",
"description" : "The directory to use for temporary files when writing data to the Postgres Warehouse.",
"hidden" : true,
"sensitive" : false,
"protected" : false
}, {
"name" : "ssh_tunnel.host",
"aliases" : [ ],
"label" : "SSH Tunnel Host",
"options" : [ ],
"kind" : "STRING",
"description" : "Host of the bastion host, this is the host we'll connect to via ssh",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "ssh_tunnel.port",
"aliases" : [ ],
"label" : "SSH Tunnel Port",
"value" : "22",
"options" : [ ],
"kind" : "INTEGER",
"description" : "Port to connect to bastion host",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "ssh_tunnel.private_key",
"aliases" : [ ],
"label" : "SSH Tunnel Private Key",
"options" : [ ],
"kind" : "PASSWORD",
"description" : "A base64 encoded Private Key for authentication to the bastion host w/ key pair auth",
"hidden" : false,
"sensitive" : true,
"encoding" : "BASE64",
"protected" : false
}, {
"name" : "ssh_tunnel.private_key_password",
"aliases" : [ ],
"label" : "SSH Tunnel Private Key Password",
"options" : [ ],
"kind" : "PASSWORD",
"description" : "Private Key Password, leave None if no password is set",
"hidden" : false,
"sensitive" : true,
"protected" : false
}, {
"name" : "ssh_tunnel.username",
"aliases" : [ ],
"label" : "SSH Tunnel Username",
"options" : [ ],
"kind" : "STRING",
"description" : "Username to connect to bastion host",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "ssh_tunnel.password",
"aliases" : [ ],
"label" : "SSH Tunnel Password",
"options" : [ ],
"kind" : "STRING",
"description" : "Password to connect to bastion host w/ basic auth",
"hidden" : false,
"sensitive" : true,
"protected" : false
}, {
"name" : "ssh_tunnel.host_setting_name",
"aliases" : [ ],
"value" : "host",
"options" : [ ],
"kind" : "STRING",
"hidden" : true,
"sensitive" : false,
"protected" : false
}, {
"name" : "ssh_tunnel.port_setting_name",
"aliases" : [ ],
"value" : "port",
"options" : [ ],
"kind" : "STRING",
"hidden" : true,
"sensitive" : false,
"protected" : false
} ],
"variants" : [ ],
"commands" : { },
"matatikaHidden" : false,
"requires" : [ ],
"fullDescription" : "Postgres Warehouse is a data warehousing solution built on top of the Postgres database management system.\n\nPostgres Warehouse is designed to handle large volumes of data and complex queries, making it an ideal solution for businesses that need to store and analyze large amounts of data. It provides a number of features that are specifically tailored to data warehousing, such as columnar storage, parallel processing, and support for advanced analytics. Additionally, Postgres Warehouse is highly scalable, allowing businesses to easily add more resources as their data needs grow. Overall, Postgres Warehouse is a powerful and flexible data warehousing solution that can help businesses make better decisions by providing them with the insights they need to succeed.\n### Prerequisites\nThe process of obtaining the required settings for connecting to a Postgres Warehouse may vary depending on the specific setup and configuration of the database. However, here are some general ways to obtain each of the required settings:\n\n- User: The user is typically created when the database is set up. You can ask the database administrator or check the database documentation to find out the username.\n- Password: The password is also typically created when the database is set up. You can ask the database administrator or check the database documentation to find out the password.\n- Host: The host is the server where the database is located. You can ask the database administrator or check the database documentation to find out the host name or IP address.\n- Port: The port is the number that the database listens on for incoming connections. The default port for Postgres is 5432, but it may be different depending on the configuration. You can ask the database administrator or check the database documentation to find out the port number.\n- Database Name: The database name is the name of the specific database you want to connect to. You can ask the database administrator or check the database documentation to find out the database name.\n- Default Target Schema: The default target schema is the schema that you want to use as the default when connecting to the database. This may be set up by the database administrator or you may need to create it yourself. You can ask the database administrator or check the database documentation to find out the default target schema.\n\n## Settings\n\n\n### User\n\nThe username used to connect to the Postgres Warehouse.\n\n### Password\n\nThe password used to authenticate the user.\n\n### Host\n\nThe hostname or IP address of the Postgres Warehouse server.\n\n### Port\n\nThe port number used to connect to the Postgres Warehouse server.\n\n### Database Name\n\nThe name of the database to connect to.\n\n### Default Target Schema\n\nThe default schema to use when writing data to the Postgres Warehouse.\n\n### State Schema\n\nThe default schema to use when writing Meltano state.\n\n### Batch Size Rows\n\nThe number of rows to write to the Postgres Warehouse in each batch.\n\n### Underscore Camel Case Fields\n\nWhether or not to convert field names from camel case to underscore-separated format.\n\n### Primary Key Required\n\nWhether or not a primary key is required for the target table.\n\n### Validate Records\n\nWhether or not to validate records before writing them to the Postgres Warehouse.\n\n### SSH Tunnel Host\n\nHost of the bastion host, this is the host we'll connect to via ssh\n\n### SSH Tunnel Port\n\nPort to connect to bastion host\n\n### SSH Tunnel Private Key\n\nA base64 encoded Private Key for authentication to the bastion host w/ key pair auth\n\n### SSH Tunnel Private Key Password\n\nPrivate Key Password, leave None if no password is set\n\n### SSH Tunnel Username\n\nUsername to connect to bastion host\n\n### SSH Tunnel Password\n\nPassword to connect to bastion host w/ basic auth",
"_links" : {
"self" : {
"href" : "https://app.matatika.com/api/dataplugins/82bf91fa-8adf-4421-863e-9a888ac3b620"
},
"update dataplugin" : {
"href" : "https://app.matatika.com/api/workspaces/52652da2-a249-407e-a8c6-4e045838a591/dataplugins/82bf91fa-8adf-4421-863e-9a888ac3b620",
"type" : "PUT"
}
}
}
},
"managed" : true,
"draft" : false,
"_links" : {
"self" : {
"href" : "https://app.matatika.com/api/datacomponents/a604ff5e-5fae-4578-ae4a-650091f015f3"
},
"update datacomponent" : {
"href" : "https://app.matatika.com/api/datacomponents/a604ff5e-5fae-4578-ae4a-650091f015f3"
},
"delete datacomponent" : {
"href" : "https://app.matatika.com/api/datacomponents/a604ff5e-5fae-4578-ae4a-650091f015f3"
}
}
}, {
"id" : "a0195563-9e63-480a-93e9-731670490a91",
"created" : "2025-06-19T10:19:42.489029",
"lastModified" : "2025-06-19T10:19:42.489029",
"name" : "dbt",
"properties" : { },
"commands" : {
"compile" : {
"args" : "compile",
"description" : "Generates executable SQL from source model, test, and analysis files. Compiled SQL files are written to the target/ directory."
},
"seed" : {
"args" : "seed",
"description" : "Load data from csv files into your data warehouse."
},
"test" : {
"args" : "test",
"description" : "Runs tests on data in deployed models."
},
"docs-generate" : {
"args" : "docs generate",
"description" : "Generate documentation artifacts for your project."
},
"deps" : {
"args" : "deps",
"description" : "Pull the most recent version of the dependencies listed in packages.yml"
},
"run" : {
"args" : "run",
"description" : "Compile SQL and execute against the current target database."
},
"clean" : {
"args" : "clean",
"description" : "Delete all folders in the clean-targets list (usually the dbt_modules and target directories.)"
},
"snapshot" : {
"args" : "snapshot",
"description" : "Execute snapshots defined in your project."
}
},
"dataPlugin" : "transformers/dbt--dbt-labs",
"_embedded" : {
"dataplugin" : {
"id" : "c3dce0d8-4025-4484-9018-964d5ae3a740",
"pluginType" : "TRANSFORMER",
"name" : "dbt",
"namespace" : "dbt",
"variant" : "dbt-labs",
"label" : "dbt",
"description" : " Power your project transformations with dbt™, a SQL-first transformation tool that enables analytics engineers to develop transformations with code.\n\n***Version Control and CI/CD***\n\nUse Matatika to deploy and promote changes between dev, UAT, and production environments.\n\n***Test and Document***\n\nUse Matatika to develop and test every model prior to production release, and share dynamically generated documentation with all stakeholders.\n\n***Develop***\n\nWrite modular data transformations in .sql – Matatika together with dbt handles the chore of dependency management. ",
"logoUrl" : "/assets/images/transformer/dbt.png",
"hidden" : false,
"docs" : "https://www.matatika.com/data-details/dbt/",
"pipUrl" : "dbt-core~=1.7.0 dbt-postgres~=1.7.0 dbt-snowflake~=1.7.0 dbt-bigquery~=1.7.0 certifi==2025.1.31",
"repo" : "https://github.com/dbt-labs/dbt-core",
"capabilities" : [ ],
"select" : [ ],
"update" : { },
"vars" : { },
"settings" : [ {
"name" : "project_dir",
"aliases" : [ ],
"value" : "$MELTANO_PROJECT_ROOT/transform",
"options" : [ ],
"kind" : "STRING",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "profiles_dir",
"aliases" : [ ],
"value" : "$MELTANO_PROJECT_ROOT/transform/profile",
"options" : [ ],
"kind" : "STRING",
"hidden" : false,
"sensitive" : false,
"env" : "DBT_PROFILES_DIR",
"protected" : false
}, {
"name" : "target",
"aliases" : [ ],
"value" : "$MELTANO_LOAD__DIALECT",
"options" : [ ],
"kind" : "STRING",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "source_schema",
"aliases" : [ ],
"value" : "$MELTANO_LOAD__TARGET_SCHEMA",
"options" : [ ],
"kind" : "STRING",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "target_schema",
"aliases" : [ ],
"value" : "analytics",
"options" : [ ],
"kind" : "STRING",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "target_path",
"aliases" : [ ],
"value" : "$MELTANO_PROJECT_ROOT/.meltano/transformers/dbt/target",
"options" : [ ],
"kind" : "STRING",
"description" : "Output path for dbt generated artifacts",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "models",
"aliases" : [ ],
"value" : "$MELTANO_TRANSFORM__PACKAGE_NAME $MELTANO_EXTRACTOR_NAMESPACE my_meltano_project",
"options" : [ ],
"kind" : "STRING",
"hidden" : false,
"sensitive" : false,
"protected" : false
} ],
"variants" : [ ],
"commands" : {
"compile" : {
"args" : "compile",
"description" : "Generates executable SQL from source model, test, and analysis files. Compiled SQL files are written to the target/ directory."
},
"seed" : {
"args" : "seed",
"description" : "Load data from csv files into your data warehouse."
},
"test" : {
"args" : "test",
"description" : "Runs tests on data in deployed models."
},
"docs-generate" : {
"args" : "docs generate",
"description" : "Generate documentation artifacts for your project."
},
"deps" : {
"args" : "deps",
"description" : "Pull the most recent version of the dependencies listed in packages.yml"
},
"run" : {
"args" : "run",
"description" : "Compile SQL and execute against the current target database."
},
"clean" : {
"args" : "clean",
"description" : "Delete all folders in the clean-targets list (usually the dbt_modules and target directories.)"
},
"snapshot" : {
"args" : "snapshot",
"description" : "Execute snapshots defined in your project."
}
},
"matatikaHidden" : false,
"requires" : [ {
"id" : "e6c1ad3d-ebf5-4c4a-b129-f68156b47555",
"pluginType" : "FILE",
"name" : "files-dbt",
"namespace" : "dbt",
"variant" : "matatika",
"description" : " Files dbt is a file bundle that automatically configures your project to run transforms with dbt.\nThe bundle includes template project configuration:\n\n- transform/models (directory)\n- transform/profile/profiles.yml\n- transform/dbt_project.yml\n- transform/.gitignore\n- transform/macros/centralize_test_failures.sql\n",
"hidden" : false,
"pipUrl" : "git+https://github.com/Matatika/[email protected]",
"repo" : "https://github.com/Matatika/files-dbt",
"capabilities" : [ ],
"select" : [ ],
"update" : {
"transform/profile/profiles.yml" : "true"
},
"vars" : { },
"settings" : [ ],
"variants" : [ ],
"commands" : { },
"matatikaHidden" : false,
"requires" : [ ],
"fullDescription" : " Files dbt is a file bundle that automatically configures your project to run transforms with dbt.\nThe bundle includes template project configuration:\n\n- transform/models (directory)\n- transform/profile/profiles.yml\n- transform/dbt_project.yml\n- transform/.gitignore\n- transform/macros/centralize_test_failures.sql\n"
} ],
"fullDescription" : " Power your project transformations with dbt™, a SQL-first transformation tool that enables analytics engineers to develop transformations with code.\n\n***Version Control and CI/CD***\n\nUse Matatika to deploy and promote changes between dev, UAT, and production environments.\n\n***Test and Document***\n\nUse Matatika to develop and test every model prior to production release, and share dynamically generated documentation with all stakeholders.\n\n***Develop***\n\nWrite modular data transformations in .sql – Matatika together with dbt handles the chore of dependency management. \n\n## Settings\n\n\n### target_path\n\nOutput path for dbt generated artifacts",
"_links" : {
"self" : {
"href" : "https://app.matatika.com/api/dataplugins/c3dce0d8-4025-4484-9018-964d5ae3a740"
},
"update dataplugin" : {
"href" : "https://app.matatika.com/api/workspaces/52652da2-a249-407e-a8c6-4e045838a591/dataplugins/c3dce0d8-4025-4484-9018-964d5ae3a740",
"type" : "PUT"
}
}
}
},
"managed" : true,
"draft" : false,
"_links" : {
"self" : {
"href" : "https://app.matatika.com/api/datacomponents/a0195563-9e63-480a-93e9-731670490a91"
},
"update datacomponent" : {
"href" : "https://app.matatika.com/api/datacomponents/a0195563-9e63-480a-93e9-731670490a91"
},
"delete datacomponent" : {
"href" : "https://app.matatika.com/api/datacomponents/a0195563-9e63-480a-93e9-731670490a91"
}
}
} ]
},
"_links" : {
"update pipeline" : {
"href" : "https://app.matatika.com/api/workspaces/52652da2-a249-407e-a8c6-4e045838a591/pipelines/180a0df1-e833-4aae-b6ef-ee98f0f07b50",
"type" : "PUT"
},
"delete pipeline" : {
"href" : "https://app.matatika.com/api/pipelines/180a0df1-e833-4aae-b6ef-ee98f0f07b50",
"type" : "DELETE"
},
"draft pipeline" : {
"href" : "https://app.matatika.com/api/workspaces/52652da2-a249-407e-a8c6-4e045838a591/pipelines/180a0df1-e833-4aae-b6ef-ee98f0f07b50/draft",
"type" : "PUT"
},
"self" : {
"href" : "https://app.matatika.com/api/pipelines/180a0df1-e833-4aae-b6ef-ee98f0f07b50"
},
"environment" : {
"href" : "https://app.matatika.com/api/pipelines/180a0df1-e833-4aae-b6ef-ee98f0f07b50/environment"
},
"jobs" : {
"href" : "https://app.matatika.com/api/pipelines/180a0df1-e833-4aae-b6ef-ee98f0f07b50/jobs",
"type" : "GET"
},
"metrics" : {
"href" : "https://app.matatika.com/api/pipelines/180a0df1-e833-4aae-b6ef-ee98f0f07b50/metrics"
},
"add subscription" : {
"href" : "https://app.matatika.com/api/pipelines/180a0df1-e833-4aae-b6ef-ee98f0f07b50/subscriptions"
},
"verify pipeline" : {
"href" : "https://app.matatika.com/api/pipelines/180a0df1-e833-4aae-b6ef-ee98f0f07b50/verification",
"type" : "POST"
},
"create job" : {
"href" : "https://app.matatika.com/api/pipelines/180a0df1-e833-4aae-b6ef-ee98f0f07b50/jobs",
"type" : "POST"
}
}
} ]
},
"_links" : {
"self" : {
"href" : "https://app.matatika.com/api/workspaces/52652da2-a249-407e-a8c6-4e045838a591/pipelines?page=0&size=20&sort=name,asc"
}
},
"page" : {
"size" : 20,
"totalElements" : 2,
"totalPages" : 1,
"number" : 0
}
}
View a pipeline
GET
/api/pipelines/{pipeline-id}
Returns the pipeline {pipeline-id}.
Prerequisites
- Pipeline
{pipeline-id}must exist
Request
Example Snippets
cURL
curl -H "Authorization: Bearer $ACCESS_TOKEN" 'https://app.matatika.com:443/api/pipelines/180a0df1-e833-4aae-b6ef-ee98f0f07b50' -i -X GET \
-H 'Accept: application/json, application/javascript, text/javascript, text/json'
Python (requests)
import requests
url = "https://app.matatika.com:443/api/pipelines/180a0df1-e833-4aae-b6ef-ee98f0f07b50"
headers = {
'Authorization': ACCESS_TOKEN
}
response = requests.request("GET", url, headers=headers)
print(response.text.encode('utf8'))
Response
200 OK
Pipeline with HAL links.
{
"id" : "180a0df1-e833-4aae-b6ef-ee98f0f07b50",
"status" : "READY",
"name" : "SIT-generated pipeline [2025-06-19T11:19:48.350863] (updated)",
"timeout" : 0,
"maxRetries" : 0,
"created" : "2025-06-19T10:19:48.519054",
"lastModified" : "2025-06-19T10:19:48.519056",
"properties" : {
"tap-google-analytics.oauth_credentials.client_secret" : "client_secret",
"tap-google-analytics.view_id" : "1234567890",
"tap-google-analytics.reports" : "reports",
"tap-google-analytics.oauth_credentials.refresh_token" : "refresh_token",
"tap-google-analytics.start_date" : "2025-05-19T11:19:47.874302+01:00",
"tap-google-analytics.end_date" : "2025-06-19T11:19:47.874360+01:00",
"tap-google-analytics.oauth_credentials.access_token" : "access_token",
"tap-google-analytics.oauth_credentials.client_id" : "client_id"
},
"dataComponents" : [ "tap-google-analytics", "Warehouse", "dbt" ],
"actions" : [ ],
"triggeredBy" : [ ],
"_embedded" : {
"dataComponents" : [ {
"id" : "b6e29a0b-8836-4e3a-bbfd-f85163798009",
"created" : "2025-06-19T10:19:47.983989",
"lastModified" : "2025-06-19T10:19:47.98399",
"name" : "tap-google-analytics",
"properties" : { },
"commands" : { },
"dataPlugin" : "extractors/tap-google-analytics--matatika",
"_embedded" : {
"dataplugin" : {
"id" : "6b89da64-1538-4519-a188-5ff173208879",
"pluginType" : "EXTRACTOR",
"name" : "tap-google-analytics",
"namespace" : "tap_google_analytics",
"variant" : "matatika",
"label" : "Google Analytics",
"description" : "Google Analytics is a web analytics service that provides insights into website traffic and user behavior.\n\nGoogle Analytics allows website owners to track and analyze various metrics related to their website's performance, such as the number of visitors, pageviews, bounce rate, and average session duration. It also provides information on the demographics and interests of website visitors, as well as the sources of traffic, including organic search, paid search, social media, and referrals. This data can be used to optimize website content and marketing strategies, as well as to measure the effectiveness of advertising campaigns. Additionally, Google Analytics offers advanced features such as goal tracking, e-commerce tracking, and custom reporting, making it a powerful tool for businesses of all sizes.\n### Prerequisites\nTo obtain the OAuth identity provider authorization endpoint used to create and refresh tokens, you need to create a project in the Google API Console and enable the Google Analytics API. Then, you can create OAuth 2.0 credentials and configure the authorized redirect URIs. The authorization endpoint will be provided in the credentials.\n\nThe OAuth scopes you need to request access to depend on the specific data you want to access in Google Analytics. For example, if you want to read data from a specific view, you will need to request the \"https://www.googleapis.com/auth/analytics.readonly\" scope. You can find a list of available scopes in the Google Analytics API documentation.\n\nTo obtain the Access Token and OAuth Refresh Token, you need to authenticate the user and obtain their consent to access their Google Analytics data. This can be done using the Google Sign-In API or the OAuth 2.0 authorization flow. Once the user has granted access, you will receive an Access Token and a Refresh Token that you can use to make API requests.\n\nTo obtain the View ID, you need to log in to your Google Analytics account and navigate to the Admin section. From there, you can select the account, property, and view that you want to access and find the View ID in the View Settings.",
"logoUrl" : "/assets/images/datasource/tap-google-analytics.svg",
"hidden" : false,
"docs" : "https://www.matatika.com/docs/instant-insights/tap-google-analytics/",
"pipUrl" : "git+https://github.com/Matatika/[email protected]",
"repo" : "https://github.com/Matatika/tap-google-analytics",
"capabilities" : [ "DISCOVER", "CATALOG", "STATE" ],
"select" : [ ],
"update" : { },
"vars" : { },
"settings" : [ {
"name" : "oauth_credentials.authorization_url",
"aliases" : [ ],
"label" : "OAuth identity provider authorization endpoint used create and refresh tokens",
"value" : "https://oauth2.googleapis.com/token",
"options" : [ ],
"kind" : "STRING",
"description" : "The endpoint used to create and refresh OAuth tokens.",
"hidden" : true,
"sensitive" : false,
"required" : "true",
"protected" : false
}, {
"name" : "oauth_credentials.scope",
"aliases" : [ ],
"label" : "OAuth scopes we need to request access to",
"value" : "profile email https://www.googleapis.com/auth/analytics.readonly",
"options" : [ ],
"kind" : "STRING",
"description" : "The specific scopes we need to request access to in order to connect to Google Analytics.",
"hidden" : true,
"sensitive" : false,
"required" : "true",
"protected" : false
}, {
"name" : "oauth_credentials.access_token",
"aliases" : [ ],
"label" : "Access Token",
"options" : [ ],
"kind" : "STRING",
"description" : "The token used to authenticate and authorize API requests.",
"hidden" : true,
"sensitive" : false,
"required" : "true",
"protected" : false
}, {
"name" : "oauth_credentials.refresh_token",
"aliases" : [ ],
"label" : "OAuth Refresh Token",
"options" : [ ],
"kind" : "STRING",
"description" : "The token used to refresh the access token when it expires.",
"hidden" : true,
"sensitive" : false,
"required" : "true",
"protected" : false
}, {
"name" : "oauth_credentials.refresh_proxy_url",
"aliases" : [ ],
"label" : "Optional - will be called with 'oauth_credentials.refresh_token' to refresh the access token",
"options" : [ ],
"kind" : "STRING",
"description" : "An optional function that will be called to refresh the access token using the refresh token.",
"hidden" : true,
"sensitive" : false,
"protected" : false
}, {
"name" : "oauth_credentials.refresh_proxy_url_auth",
"aliases" : [ ],
"label" : "Optional - Sets Authorization header on 'oauth_credentials.refresh_url' request",
"options" : [ ],
"kind" : "STRING",
"description" : "An optional setting that sets the Authorization header on the refresh URL request.",
"hidden" : true,
"sensitive" : false,
"protected" : false
}, {
"name" : "oauth_credentials.client_id",
"aliases" : [ ],
"label" : "Optional - OAuth Client ID used if refresh_proxy_url not supplied",
"options" : [ ],
"kind" : "STRING",
"description" : "An optional OAuth Client ID used if the refresh proxy URL is not supplied.",
"hidden" : true,
"sensitive" : false,
"protected" : false
}, {
"name" : "oauth_credentials.client_secret",
"aliases" : [ ],
"label" : "Optional - OAuth Client Secret used if refresh_proxy_url not supplied",
"options" : [ ],
"kind" : "STRING",
"description" : "An optional OAuth Client Secret used if the refresh proxy URL is not supplied.",
"hidden" : true,
"sensitive" : false,
"protected" : false
}, {
"name" : "view_id",
"aliases" : [ ],
"label" : "View ID",
"options" : [ ],
"placeholder" : "Ex. 198343027",
"kind" : "STRING",
"description" : "The ID of the Google Analytics view to retrieve data from.",
"hidden" : false,
"sensitive" : false,
"required" : "true",
"protected" : false
}, {
"name" : "reports",
"aliases" : [ ],
"label" : "Reports",
"options" : [ ],
"placeholder" : "Ex. my_report_definition.json",
"kind" : "STRING",
"description" : "The specific reports to retrieve data from in the Google Analytics view.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "start_date",
"aliases" : [ ],
"label" : "Start date",
"options" : [ ],
"kind" : "DATE_ISO8601",
"description" : "The start date for the date range of data to retrieve.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "end_date",
"aliases" : [ ],
"label" : "End date",
"options" : [ ],
"kind" : "DATE_ISO8601",
"description" : "The end date for the date range of data to retrieve.",
"hidden" : false,
"sensitive" : false,
"protected" : false
} ],
"variants" : [ ],
"commands" : { },
"matatikaHidden" : false,
"requires" : [ ],
"fullDescription" : "Google Analytics is a web analytics service that provides insights into website traffic and user behavior.\n\nGoogle Analytics allows website owners to track and analyze various metrics related to their website's performance, such as the number of visitors, pageviews, bounce rate, and average session duration. It also provides information on the demographics and interests of website visitors, as well as the sources of traffic, including organic search, paid search, social media, and referrals. This data can be used to optimize website content and marketing strategies, as well as to measure the effectiveness of advertising campaigns. Additionally, Google Analytics offers advanced features such as goal tracking, e-commerce tracking, and custom reporting, making it a powerful tool for businesses of all sizes.\n### Prerequisites\nTo obtain the OAuth identity provider authorization endpoint used to create and refresh tokens, you need to create a project in the Google API Console and enable the Google Analytics API. Then, you can create OAuth 2.0 credentials and configure the authorized redirect URIs. The authorization endpoint will be provided in the credentials.\n\nThe OAuth scopes you need to request access to depend on the specific data you want to access in Google Analytics. For example, if you want to read data from a specific view, you will need to request the \"https://www.googleapis.com/auth/analytics.readonly\" scope. You can find a list of available scopes in the Google Analytics API documentation.\n\nTo obtain the Access Token and OAuth Refresh Token, you need to authenticate the user and obtain their consent to access their Google Analytics data. This can be done using the Google Sign-In API or the OAuth 2.0 authorization flow. Once the user has granted access, you will receive an Access Token and a Refresh Token that you can use to make API requests.\n\nTo obtain the View ID, you need to log in to your Google Analytics account and navigate to the Admin section. From there, you can select the account, property, and view that you want to access and find the View ID in the View Settings.\n\n## Settings\n\n\n### View ID\n\nThe ID of the Google Analytics view to retrieve data from.\n\n### Reports\n\nThe specific reports to retrieve data from in the Google Analytics view.\n\n### Start date\n\nThe start date for the date range of data to retrieve.\n\n### End date\n\nThe end date for the date range of data to retrieve.",
"_links" : {
"self" : {
"href" : "https://app.matatika.com/api/dataplugins/6b89da64-1538-4519-a188-5ff173208879"
},
"update dataplugin" : {
"href" : "https://app.matatika.com/api/workspaces/52652da2-a249-407e-a8c6-4e045838a591/dataplugins/6b89da64-1538-4519-a188-5ff173208879",
"type" : "PUT"
}
}
}
},
"managed" : false,
"draft" : true,
"_links" : {
"self" : {
"href" : "https://app.matatika.com/api/datacomponents/b6e29a0b-8836-4e3a-bbfd-f85163798009"
},
"update datacomponent" : {
"href" : "https://app.matatika.com/api/datacomponents/b6e29a0b-8836-4e3a-bbfd-f85163798009"
},
"delete datacomponent" : {
"href" : "https://app.matatika.com/api/datacomponents/b6e29a0b-8836-4e3a-bbfd-f85163798009"
}
}
}, {
"id" : "a604ff5e-5fae-4578-ae4a-650091f015f3",
"created" : "2025-06-19T10:19:42.42539",
"lastModified" : "2025-06-19T10:19:42.425391",
"name" : "Warehouse",
"properties" : {
"password" : "7PhOTBc5_2SMTn3on_ntw8iRA1",
"dbname" : "axvzzfd",
"default_target_schema" : "analytics",
"port" : "5432",
"host" : "sharp-banana2.postgres.database.azure.com",
"user" : "axvzzfd"
},
"commands" : { },
"dataPlugin" : "loaders/target-postgres--matatika",
"_embedded" : {
"dataplugin" : {
"id" : "82bf91fa-8adf-4421-863e-9a888ac3b620",
"pluginType" : "LOADER",
"name" : "target-postgres",
"namespace" : "postgres_transferwise",
"variant" : "matatika",
"label" : "Postgres Warehouse",
"description" : "Postgres Warehouse is a data warehousing solution built on top of the Postgres database management system.\n\nPostgres Warehouse is designed to handle large volumes of data and complex queries, making it an ideal solution for businesses that need to store and analyze large amounts of data. It provides a number of features that are specifically tailored to data warehousing, such as columnar storage, parallel processing, and support for advanced analytics. Additionally, Postgres Warehouse is highly scalable, allowing businesses to easily add more resources as their data needs grow. Overall, Postgres Warehouse is a powerful and flexible data warehousing solution that can help businesses make better decisions by providing them with the insights they need to succeed.\n### Prerequisites\nThe process of obtaining the required settings for connecting to a Postgres Warehouse may vary depending on the specific setup and configuration of the database. However, here are some general ways to obtain each of the required settings:\n\n- User: The user is typically created when the database is set up. You can ask the database administrator or check the database documentation to find out the username.\n- Password: The password is also typically created when the database is set up. You can ask the database administrator or check the database documentation to find out the password.\n- Host: The host is the server where the database is located. You can ask the database administrator or check the database documentation to find out the host name or IP address.\n- Port: The port is the number that the database listens on for incoming connections. The default port for Postgres is 5432, but it may be different depending on the configuration. You can ask the database administrator or check the database documentation to find out the port number.\n- Database Name: The database name is the name of the specific database you want to connect to. You can ask the database administrator or check the database documentation to find out the database name.\n- Default Target Schema: The default target schema is the schema that you want to use as the default when connecting to the database. This may be set up by the database administrator or you may need to create it yourself. You can ask the database administrator or check the database documentation to find out the default target schema.",
"logoUrl" : "/assets/logos/loaders/postgres.png",
"hidden" : false,
"docs" : "https://www.matatika.com/data-details/target-postgres/",
"pipUrl" : "git+https://github.com/Matatika/[email protected]",
"repo" : "git+https://github.com/Matatika/[email protected]",
"capabilities" : [ ],
"select" : [ ],
"update" : { },
"vars" : { },
"settings" : [ {
"name" : "user",
"aliases" : [ "username" ],
"label" : "User",
"options" : [ ],
"kind" : "STRING",
"description" : "The username used to connect to the Postgres Warehouse.",
"hidden" : false,
"sensitive" : false,
"required" : "true",
"protected" : false
}, {
"name" : "password",
"aliases" : [ ],
"label" : "Password",
"options" : [ ],
"kind" : "STRING",
"description" : "The password used to authenticate the user.",
"hidden" : false,
"sensitive" : true,
"required" : "true",
"protected" : false
}, {
"name" : "host",
"aliases" : [ "address" ],
"label" : "Host",
"options" : [ ],
"kind" : "STRING",
"description" : "The hostname or IP address of the Postgres Warehouse server.",
"hidden" : false,
"sensitive" : false,
"required" : "true",
"protected" : false
}, {
"name" : "port",
"aliases" : [ ],
"label" : "Port",
"value" : "5432",
"options" : [ ],
"kind" : "INTEGER",
"description" : "The port number used to connect to the Postgres Warehouse server.",
"hidden" : false,
"sensitive" : false,
"required" : "true",
"protected" : false
}, {
"name" : "dbname",
"aliases" : [ "database" ],
"label" : "Database Name",
"options" : [ ],
"kind" : "STRING",
"description" : "The name of the database to connect to.",
"hidden" : false,
"sensitive" : false,
"required" : "true",
"protected" : false
}, {
"name" : "default_target_schema",
"aliases" : [ ],
"label" : "Default Target Schema",
"value" : "analytics",
"options" : [ ],
"kind" : "STRING",
"description" : "The default schema to use when writing data to the Postgres Warehouse.",
"hidden" : false,
"sensitive" : false,
"required" : "true",
"protected" : false
}, {
"name" : "state_schema",
"aliases" : [ ],
"label" : "State Schema",
"value" : "public",
"options" : [ ],
"kind" : "STRING",
"description" : "The default schema to use when writing Meltano state.",
"hidden" : false,
"sensitive" : false,
"required" : "true",
"protected" : false
}, {
"name" : "ssl",
"aliases" : [ ],
"label" : "SSL",
"value" : "false",
"options" : [ ],
"kind" : "BOOLEAN",
"description" : "Whether or not to use SSL encryption when connecting to the Postgres Warehouse.",
"hidden" : true,
"sensitive" : false,
"protected" : false,
"value_post_processor" : "STRINGIFY"
}, {
"name" : "batch_size_rows",
"aliases" : [ ],
"label" : "Batch Size Rows",
"value" : "100000",
"options" : [ ],
"kind" : "INTEGER",
"description" : "The number of rows to write to the Postgres Warehouse in each batch.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "underscore_camel_case_fields",
"aliases" : [ ],
"label" : "Underscore Camel Case Fields",
"value" : "true",
"options" : [ ],
"kind" : "BOOLEAN",
"description" : "Whether or not to convert field names from camel case to underscore-separated format.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "flush_all_streams",
"aliases" : [ ],
"label" : "Flush All Streams",
"value" : "true",
"options" : [ ],
"kind" : "BOOLEAN",
"description" : "Whether or not to flush all streams to the Postgres Warehouse before closing the connection.",
"hidden" : true,
"sensitive" : false,
"protected" : false
}, {
"name" : "parallelism",
"aliases" : [ ],
"label" : "Parallelism",
"value" : "-1",
"options" : [ ],
"kind" : "INTEGER",
"description" : "The number of threads to use when writing data to the Postgres Warehouse.",
"hidden" : true,
"sensitive" : false,
"protected" : false
}, {
"name" : "parallelism_max",
"aliases" : [ ],
"label" : "Max Parallelism",
"value" : "16",
"options" : [ ],
"kind" : "INTEGER",
"description" : "The maximum number of threads to use when writing data to the Postgres Warehouse.",
"hidden" : true,
"sensitive" : false,
"protected" : false
}, {
"name" : "default_target_schema_select_permission",
"aliases" : [ ],
"label" : "Default Target Schema Select Permission",
"options" : [ ],
"kind" : "STRING",
"description" : "The permission level required to select data from the default target schema.",
"hidden" : true,
"sensitive" : false,
"protected" : false
}, {
"name" : "schema_mapping",
"aliases" : [ ],
"label" : "Schema Mapping",
"options" : [ ],
"kind" : "STRING",
"description" : "A mapping of source schema names to target schema names.",
"hidden" : true,
"sensitive" : false,
"protected" : false
}, {
"name" : "add_metadata_columns",
"aliases" : [ ],
"label" : "Add Metadata Columns",
"value" : "true",
"options" : [ ],
"kind" : "BOOLEAN",
"description" : "Whether or not to add metadata columns to the target table.",
"hidden" : true,
"sensitive" : false,
"protected" : false
}, {
"name" : "hard_delete",
"aliases" : [ ],
"label" : "Hard Delete",
"value" : "false",
"options" : [ ],
"kind" : "BOOLEAN",
"description" : "Whether or not to perform hard deletes when deleting data from the Postgres Warehouse.",
"hidden" : true,
"sensitive" : false,
"protected" : false
}, {
"name" : "data_flattening_max_level",
"aliases" : [ ],
"label" : "Data Flattening Max Level",
"value" : "10",
"options" : [ ],
"kind" : "INTEGER",
"description" : "The maximum level of nested data structures to flatten when writing data to the Postgres Warehouse.",
"hidden" : true,
"sensitive" : false,
"protected" : false
}, {
"name" : "primary_key_required",
"aliases" : [ ],
"label" : "Primary Key Required",
"value" : "false",
"options" : [ ],
"kind" : "BOOLEAN",
"description" : "Whether or not a primary key is required for the target table.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "validate_records",
"aliases" : [ ],
"label" : "Validate Records",
"value" : "false",
"options" : [ ],
"kind" : "BOOLEAN",
"description" : "Whether or not to validate records before writing them to the Postgres Warehouse.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "temp_dir",
"aliases" : [ ],
"label" : "Temporary Directory",
"options" : [ ],
"kind" : "STRING",
"description" : "The directory to use for temporary files when writing data to the Postgres Warehouse.",
"hidden" : true,
"sensitive" : false,
"protected" : false
}, {
"name" : "ssh_tunnel.host",
"aliases" : [ ],
"label" : "SSH Tunnel Host",
"options" : [ ],
"kind" : "STRING",
"description" : "Host of the bastion host, this is the host we'll connect to via ssh",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "ssh_tunnel.port",
"aliases" : [ ],
"label" : "SSH Tunnel Port",
"value" : "22",
"options" : [ ],
"kind" : "INTEGER",
"description" : "Port to connect to bastion host",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "ssh_tunnel.private_key",
"aliases" : [ ],
"label" : "SSH Tunnel Private Key",
"options" : [ ],
"kind" : "PASSWORD",
"description" : "A base64 encoded Private Key for authentication to the bastion host w/ key pair auth",
"hidden" : false,
"sensitive" : true,
"encoding" : "BASE64",
"protected" : false
}, {
"name" : "ssh_tunnel.private_key_password",
"aliases" : [ ],
"label" : "SSH Tunnel Private Key Password",
"options" : [ ],
"kind" : "PASSWORD",
"description" : "Private Key Password, leave None if no password is set",
"hidden" : false,
"sensitive" : true,
"protected" : false
}, {
"name" : "ssh_tunnel.username",
"aliases" : [ ],
"label" : "SSH Tunnel Username",
"options" : [ ],
"kind" : "STRING",
"description" : "Username to connect to bastion host",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "ssh_tunnel.password",
"aliases" : [ ],
"label" : "SSH Tunnel Password",
"options" : [ ],
"kind" : "STRING",
"description" : "Password to connect to bastion host w/ basic auth",
"hidden" : false,
"sensitive" : true,
"protected" : false
}, {
"name" : "ssh_tunnel.host_setting_name",
"aliases" : [ ],
"value" : "host",
"options" : [ ],
"kind" : "STRING",
"hidden" : true,
"sensitive" : false,
"protected" : false
}, {
"name" : "ssh_tunnel.port_setting_name",
"aliases" : [ ],
"value" : "port",
"options" : [ ],
"kind" : "STRING",
"hidden" : true,
"sensitive" : false,
"protected" : false
} ],
"variants" : [ ],
"commands" : { },
"matatikaHidden" : false,
"requires" : [ ],
"fullDescription" : "Postgres Warehouse is a data warehousing solution built on top of the Postgres database management system.\n\nPostgres Warehouse is designed to handle large volumes of data and complex queries, making it an ideal solution for businesses that need to store and analyze large amounts of data. It provides a number of features that are specifically tailored to data warehousing, such as columnar storage, parallel processing, and support for advanced analytics. Additionally, Postgres Warehouse is highly scalable, allowing businesses to easily add more resources as their data needs grow. Overall, Postgres Warehouse is a powerful and flexible data warehousing solution that can help businesses make better decisions by providing them with the insights they need to succeed.\n### Prerequisites\nThe process of obtaining the required settings for connecting to a Postgres Warehouse may vary depending on the specific setup and configuration of the database. However, here are some general ways to obtain each of the required settings:\n\n- User: The user is typically created when the database is set up. You can ask the database administrator or check the database documentation to find out the username.\n- Password: The password is also typically created when the database is set up. You can ask the database administrator or check the database documentation to find out the password.\n- Host: The host is the server where the database is located. You can ask the database administrator or check the database documentation to find out the host name or IP address.\n- Port: The port is the number that the database listens on for incoming connections. The default port for Postgres is 5432, but it may be different depending on the configuration. You can ask the database administrator or check the database documentation to find out the port number.\n- Database Name: The database name is the name of the specific database you want to connect to. You can ask the database administrator or check the database documentation to find out the database name.\n- Default Target Schema: The default target schema is the schema that you want to use as the default when connecting to the database. This may be set up by the database administrator or you may need to create it yourself. You can ask the database administrator or check the database documentation to find out the default target schema.\n\n## Settings\n\n\n### User\n\nThe username used to connect to the Postgres Warehouse.\n\n### Password\n\nThe password used to authenticate the user.\n\n### Host\n\nThe hostname or IP address of the Postgres Warehouse server.\n\n### Port\n\nThe port number used to connect to the Postgres Warehouse server.\n\n### Database Name\n\nThe name of the database to connect to.\n\n### Default Target Schema\n\nThe default schema to use when writing data to the Postgres Warehouse.\n\n### State Schema\n\nThe default schema to use when writing Meltano state.\n\n### Batch Size Rows\n\nThe number of rows to write to the Postgres Warehouse in each batch.\n\n### Underscore Camel Case Fields\n\nWhether or not to convert field names from camel case to underscore-separated format.\n\n### Primary Key Required\n\nWhether or not a primary key is required for the target table.\n\n### Validate Records\n\nWhether or not to validate records before writing them to the Postgres Warehouse.\n\n### SSH Tunnel Host\n\nHost of the bastion host, this is the host we'll connect to via ssh\n\n### SSH Tunnel Port\n\nPort to connect to bastion host\n\n### SSH Tunnel Private Key\n\nA base64 encoded Private Key for authentication to the bastion host w/ key pair auth\n\n### SSH Tunnel Private Key Password\n\nPrivate Key Password, leave None if no password is set\n\n### SSH Tunnel Username\n\nUsername to connect to bastion host\n\n### SSH Tunnel Password\n\nPassword to connect to bastion host w/ basic auth",
"_links" : {
"self" : {
"href" : "https://app.matatika.com/api/dataplugins/82bf91fa-8adf-4421-863e-9a888ac3b620"
},
"update dataplugin" : {
"href" : "https://app.matatika.com/api/workspaces/52652da2-a249-407e-a8c6-4e045838a591/dataplugins/82bf91fa-8adf-4421-863e-9a888ac3b620",
"type" : "PUT"
}
}
}
},
"managed" : true,
"draft" : false,
"_links" : {
"self" : {
"href" : "https://app.matatika.com/api/datacomponents/a604ff5e-5fae-4578-ae4a-650091f015f3"
},
"update datacomponent" : {
"href" : "https://app.matatika.com/api/datacomponents/a604ff5e-5fae-4578-ae4a-650091f015f3"
},
"delete datacomponent" : {
"href" : "https://app.matatika.com/api/datacomponents/a604ff5e-5fae-4578-ae4a-650091f015f3"
}
}
}, {
"id" : "a0195563-9e63-480a-93e9-731670490a91",
"created" : "2025-06-19T10:19:42.489029",
"lastModified" : "2025-06-19T10:19:42.489029",
"name" : "dbt",
"properties" : { },
"commands" : {
"compile" : {
"args" : "compile",
"description" : "Generates executable SQL from source model, test, and analysis files. Compiled SQL files are written to the target/ directory."
},
"seed" : {
"args" : "seed",
"description" : "Load data from csv files into your data warehouse."
},
"test" : {
"args" : "test",
"description" : "Runs tests on data in deployed models."
},
"docs-generate" : {
"args" : "docs generate",
"description" : "Generate documentation artifacts for your project."
},
"deps" : {
"args" : "deps",
"description" : "Pull the most recent version of the dependencies listed in packages.yml"
},
"run" : {
"args" : "run",
"description" : "Compile SQL and execute against the current target database."
},
"clean" : {
"args" : "clean",
"description" : "Delete all folders in the clean-targets list (usually the dbt_modules and target directories.)"
},
"snapshot" : {
"args" : "snapshot",
"description" : "Execute snapshots defined in your project."
}
},
"dataPlugin" : "transformers/dbt--dbt-labs",
"_embedded" : {
"dataplugin" : {
"id" : "c3dce0d8-4025-4484-9018-964d5ae3a740",
"pluginType" : "TRANSFORMER",
"name" : "dbt",
"namespace" : "dbt",
"variant" : "dbt-labs",
"label" : "dbt",
"description" : " Power your project transformations with dbt™, a SQL-first transformation tool that enables analytics engineers to develop transformations with code.\n\n***Version Control and CI/CD***\n\nUse Matatika to deploy and promote changes between dev, UAT, and production environments.\n\n***Test and Document***\n\nUse Matatika to develop and test every model prior to production release, and share dynamically generated documentation with all stakeholders.\n\n***Develop***\n\nWrite modular data transformations in .sql – Matatika together with dbt handles the chore of dependency management. ",
"logoUrl" : "/assets/images/transformer/dbt.png",
"hidden" : false,
"docs" : "https://www.matatika.com/data-details/dbt/",
"pipUrl" : "dbt-core~=1.7.0 dbt-postgres~=1.7.0 dbt-snowflake~=1.7.0 dbt-bigquery~=1.7.0 certifi==2025.1.31",
"repo" : "https://github.com/dbt-labs/dbt-core",
"capabilities" : [ ],
"select" : [ ],
"update" : { },
"vars" : { },
"settings" : [ {
"name" : "project_dir",
"aliases" : [ ],
"value" : "$MELTANO_PROJECT_ROOT/transform",
"options" : [ ],
"kind" : "STRING",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "profiles_dir",
"aliases" : [ ],
"value" : "$MELTANO_PROJECT_ROOT/transform/profile",
"options" : [ ],
"kind" : "STRING",
"hidden" : false,
"sensitive" : false,
"env" : "DBT_PROFILES_DIR",
"protected" : false
}, {
"name" : "target",
"aliases" : [ ],
"value" : "$MELTANO_LOAD__DIALECT",
"options" : [ ],
"kind" : "STRING",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "source_schema",
"aliases" : [ ],
"value" : "$MELTANO_LOAD__TARGET_SCHEMA",
"options" : [ ],
"kind" : "STRING",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "target_schema",
"aliases" : [ ],
"value" : "analytics",
"options" : [ ],
"kind" : "STRING",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "target_path",
"aliases" : [ ],
"value" : "$MELTANO_PROJECT_ROOT/.meltano/transformers/dbt/target",
"options" : [ ],
"kind" : "STRING",
"description" : "Output path for dbt generated artifacts",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "models",
"aliases" : [ ],
"value" : "$MELTANO_TRANSFORM__PACKAGE_NAME $MELTANO_EXTRACTOR_NAMESPACE my_meltano_project",
"options" : [ ],
"kind" : "STRING",
"hidden" : false,
"sensitive" : false,
"protected" : false
} ],
"variants" : [ ],
"commands" : {
"compile" : {
"args" : "compile",
"description" : "Generates executable SQL from source model, test, and analysis files. Compiled SQL files are written to the target/ directory."
},
"seed" : {
"args" : "seed",
"description" : "Load data from csv files into your data warehouse."
},
"test" : {
"args" : "test",
"description" : "Runs tests on data in deployed models."
},
"docs-generate" : {
"args" : "docs generate",
"description" : "Generate documentation artifacts for your project."
},
"deps" : {
"args" : "deps",
"description" : "Pull the most recent version of the dependencies listed in packages.yml"
},
"run" : {
"args" : "run",
"description" : "Compile SQL and execute against the current target database."
},
"clean" : {
"args" : "clean",
"description" : "Delete all folders in the clean-targets list (usually the dbt_modules and target directories.)"
},
"snapshot" : {
"args" : "snapshot",
"description" : "Execute snapshots defined in your project."
}
},
"matatikaHidden" : false,
"requires" : [ {
"id" : "e6c1ad3d-ebf5-4c4a-b129-f68156b47555",
"pluginType" : "FILE",
"name" : "files-dbt",
"namespace" : "dbt",
"variant" : "matatika",
"description" : " Files dbt is a file bundle that automatically configures your project to run transforms with dbt.\nThe bundle includes template project configuration:\n\n- transform/models (directory)\n- transform/profile/profiles.yml\n- transform/dbt_project.yml\n- transform/.gitignore\n- transform/macros/centralize_test_failures.sql\n",
"hidden" : false,
"pipUrl" : "git+https://github.com/Matatika/[email protected]",
"repo" : "https://github.com/Matatika/files-dbt",
"capabilities" : [ ],
"select" : [ ],
"update" : {
"transform/profile/profiles.yml" : "true"
},
"vars" : { },
"settings" : [ ],
"variants" : [ ],
"commands" : { },
"matatikaHidden" : false,
"requires" : [ ],
"fullDescription" : " Files dbt is a file bundle that automatically configures your project to run transforms with dbt.\nThe bundle includes template project configuration:\n\n- transform/models (directory)\n- transform/profile/profiles.yml\n- transform/dbt_project.yml\n- transform/.gitignore\n- transform/macros/centralize_test_failures.sql\n"
} ],
"fullDescription" : " Power your project transformations with dbt™, a SQL-first transformation tool that enables analytics engineers to develop transformations with code.\n\n***Version Control and CI/CD***\n\nUse Matatika to deploy and promote changes between dev, UAT, and production environments.\n\n***Test and Document***\n\nUse Matatika to develop and test every model prior to production release, and share dynamically generated documentation with all stakeholders.\n\n***Develop***\n\nWrite modular data transformations in .sql – Matatika together with dbt handles the chore of dependency management. \n\n## Settings\n\n\n### target_path\n\nOutput path for dbt generated artifacts",
"_links" : {
"self" : {
"href" : "https://app.matatika.com/api/dataplugins/c3dce0d8-4025-4484-9018-964d5ae3a740"
},
"update dataplugin" : {
"href" : "https://app.matatika.com/api/workspaces/52652da2-a249-407e-a8c6-4e045838a591/dataplugins/c3dce0d8-4025-4484-9018-964d5ae3a740",
"type" : "PUT"
}
}
}
},
"managed" : true,
"draft" : false,
"_links" : {
"self" : {
"href" : "https://app.matatika.com/api/datacomponents/a0195563-9e63-480a-93e9-731670490a91"
},
"update datacomponent" : {
"href" : "https://app.matatika.com/api/datacomponents/a0195563-9e63-480a-93e9-731670490a91"
},
"delete datacomponent" : {
"href" : "https://app.matatika.com/api/datacomponents/a0195563-9e63-480a-93e9-731670490a91"
}
}
} ]
},
"_links" : {
"update pipeline" : {
"href" : "https://app.matatika.com/api/workspaces/52652da2-a249-407e-a8c6-4e045838a591/pipelines/180a0df1-e833-4aae-b6ef-ee98f0f07b50",
"type" : "PUT"
},
"delete pipeline" : {
"href" : "https://app.matatika.com/api/pipelines/180a0df1-e833-4aae-b6ef-ee98f0f07b50",
"type" : "DELETE"
},
"draft pipeline" : {
"href" : "https://app.matatika.com/api/workspaces/52652da2-a249-407e-a8c6-4e045838a591/pipelines/180a0df1-e833-4aae-b6ef-ee98f0f07b50/draft",
"type" : "PUT"
},
"self" : {
"href" : "https://app.matatika.com/api/pipelines/180a0df1-e833-4aae-b6ef-ee98f0f07b50"
},
"environment" : {
"href" : "https://app.matatika.com/api/pipelines/180a0df1-e833-4aae-b6ef-ee98f0f07b50/environment"
},
"jobs" : {
"href" : "https://app.matatika.com/api/pipelines/180a0df1-e833-4aae-b6ef-ee98f0f07b50/jobs",
"type" : "GET"
},
"metrics" : {
"href" : "https://app.matatika.com/api/pipelines/180a0df1-e833-4aae-b6ef-ee98f0f07b50/metrics"
},
"add subscription" : {
"href" : "https://app.matatika.com/api/pipelines/180a0df1-e833-4aae-b6ef-ee98f0f07b50/subscriptions"
},
"verify pipeline" : {
"href" : "https://app.matatika.com/api/pipelines/180a0df1-e833-4aae-b6ef-ee98f0f07b50/verification",
"type" : "POST"
},
"create job" : {
"href" : "https://app.matatika.com/api/pipelines/180a0df1-e833-4aae-b6ef-ee98f0f07b50/jobs",
"type" : "POST"
}
}
}
Initialise a pipeline in a workspace
POST
/api/workspaces/{workspace-id}/pipelines
Initialises a new pipeline in the workspace {workspace-id}.
Prerequisites
- Workspace
{workspace-id}must exist
Request
Example Snippets
cURL
curl -H "Authorization: Bearer $ACCESS_TOKEN" 'https://app.matatika.com:443/api/workspaces/52652da2-a249-407e-a8c6-4e045838a591/pipelines' -i -X POST \
-H 'Accept: application/json, application/javascript, text/javascript, text/json' \
-H 'Content-Type: application/json'
Python (requests)
import requests
url = "https://app.matatika.com:443/api/workspaces/52652da2-a249-407e-a8c6-4e045838a591/pipelines"
headers = {
'Authorization': ACCESS_TOKEN
}
response = requests.request("POST", url, headers=headers)
print(response.text.encode('utf8'))
Response
200 OK
Pipeline with HAL links.
{
"id" : "f42e86bc-e5f7-4965-9ee9-92bdf7a5ac36",
"status" : "PROVISIONING",
"timeout" : 0,
"maxRetries" : 0,
"created" : "2025-06-19T10:19:47.842329833",
"lastModified" : "2025-06-19T10:19:47.842332933",
"properties" : { },
"dataComponents" : [ ],
"actions" : [ ],
"triggeredBy" : [ ],
"_links" : {
"create pipeline" : {
"href" : "https://app.matatika.com/api/workspaces/52652da2-a249-407e-a8c6-4e045838a591/pipelines/f42e86bc-e5f7-4965-9ee9-92bdf7a5ac36",
"type" : "PUT"
},
"draft pipeline" : {
"href" : "https://app.matatika.com/api/workspaces/52652da2-a249-407e-a8c6-4e045838a591/pipelines/f42e86bc-e5f7-4965-9ee9-92bdf7a5ac36/draft",
"type" : "PUT"
},
"validate pipeline" : {
"href" : "https://app.matatika.com/api/workspaces/52652da2-a249-407e-a8c6-4e045838a591/pipelines/validation",
"type" : "POST"
}
}
}
Create or update a pipeline in a workspace
PUT
/api/workspaces/{workspace-id}/pipelines/{pipeline-id}
Creates or updates the pipeline {pipeline-id} in the workspace {workspace-id}.
Prerequisites
- Workspace
{workspace-id}must exist
Request
Body
Pipeline resource.
{
"name" : "SIT-generated pipeline [2025-06-19T11:19:47.868367]",
"dataComponents" : [ "extractors/tap-google-analytics", "Warehouse", "dbt" ],
"schedule" : "0 0 0 25 12 ?",
"properties" : {
"tap-google-analytics.view_id" : "1234567890",
"tap-google-analytics.oauth_credentials.client_secret" : "client_secret",
"tap-google-analytics.reports" : "reports",
"tap-google-analytics.oauth_credentials.refresh_token" : "refresh_token",
"tap-google-analytics.start_date" : "2025-05-19T11:19:47.874302+01:00",
"tap-google-analytics.end_date" : "2025-06-19T11:19:47.874360+01:00",
"tap-google-analytics.oauth_credentials.client_id" : "client_id",
"tap-google-analytics.oauth_credentials.access_token" : "access_token"
}
}
Example Snippets
cURL
curl -H "Authorization: Bearer $ACCESS_TOKEN" 'https://app.matatika.com:443/api/workspaces/52652da2-a249-407e-a8c6-4e045838a591/pipelines/f42e86bc-e5f7-4965-9ee9-92bdf7a5ac36' -i -X PUT \
-H 'Accept: application/json, application/javascript, text/javascript, text/json' \
-H 'Content-Type: application/json' \
-d '{
"name" : "SIT-generated pipeline [2025-06-19T11:19:47.868367]",
"dataComponents" : [ "extractors/tap-google-analytics", "Warehouse", "dbt" ],
"schedule" : "0 0 0 25 12 ?",
"properties" : {
"tap-google-analytics.view_id" : "1234567890",
"tap-google-analytics.oauth_credentials.client_secret" : "client_secret",
"tap-google-analytics.reports" : "reports",
"tap-google-analytics.oauth_credentials.refresh_token" : "refresh_token",
"tap-google-analytics.start_date" : "2025-05-19T11:19:47.874302+01:00",
"tap-google-analytics.end_date" : "2025-06-19T11:19:47.874360+01:00",
"tap-google-analytics.oauth_credentials.client_id" : "client_id",
"tap-google-analytics.oauth_credentials.access_token" : "access_token"
}
}'
Python (requests)
import requests
url = "https://app.matatika.com:443/api/workspaces/52652da2-a249-407e-a8c6-4e045838a591/pipelines/f42e86bc-e5f7-4965-9ee9-92bdf7a5ac36"
data = {
"name" : "SIT-generated pipeline [2025-06-19T11:19:47.868367]",
"dataComponents" : [ "extractors/tap-google-analytics", "Warehouse", "dbt" ],
"schedule" : "0 0 0 25 12 ?",
"properties" : {
"tap-google-analytics.view_id" : "1234567890",
"tap-google-analytics.oauth_credentials.client_secret" : "client_secret",
"tap-google-analytics.reports" : "reports",
"tap-google-analytics.oauth_credentials.refresh_token" : "refresh_token",
"tap-google-analytics.start_date" : "2025-05-19T11:19:47.874302+01:00",
"tap-google-analytics.end_date" : "2025-06-19T11:19:47.874360+01:00",
"tap-google-analytics.oauth_credentials.client_id" : "client_id",
"tap-google-analytics.oauth_credentials.access_token" : "access_token"
}
}
headers = {
'Authorization': ACCESS_TOKEN
}
response = requests.request("PUT", url, headers=headers, data=data)
print(response.text.encode('utf8'))
Response
200 OK / 201 Created
Pipeline with HAL links.
{
"id" : "f42e86bc-e5f7-4965-9ee9-92bdf7a5ac36",
"status" : "READY",
"name" : "SIT-generated pipeline [2025-06-19T11:19:47.868367]",
"schedule" : "0 0 0 25 12 ?",
"timeout" : 0,
"maxRetries" : 0,
"created" : "2025-06-19T10:19:48.096569",
"lastModified" : "2025-06-19T10:19:48.09657",
"properties" : {
"tap-google-analytics.oauth_credentials.client_secret" : "client_secret",
"tap-google-analytics.view_id" : "1234567890",
"tap-google-analytics.reports" : "reports",
"tap-google-analytics.oauth_credentials.refresh_token" : "refresh_token",
"tap-google-analytics.start_date" : "2025-05-19T11:19:47.874302+01:00",
"tap-google-analytics.end_date" : "2025-06-19T11:19:47.874360+01:00",
"tap-google-analytics.oauth_credentials.access_token" : "access_token",
"tap-google-analytics.oauth_credentials.client_id" : "client_id"
},
"dataComponents" : [ "tap-google-analytics", "Warehouse", "dbt" ],
"actions" : [ ],
"triggeredBy" : [ ],
"_embedded" : {
"dataComponents" : [ {
"id" : "b6e29a0b-8836-4e3a-bbfd-f85163798009",
"created" : "2025-06-19T10:19:47.983989",
"lastModified" : "2025-06-19T10:19:47.98399",
"name" : "tap-google-analytics",
"properties" : { },
"commands" : { },
"dataPlugin" : "extractors/tap-google-analytics--matatika",
"_embedded" : {
"dataplugin" : {
"id" : "6b89da64-1538-4519-a188-5ff173208879",
"pluginType" : "EXTRACTOR",
"name" : "tap-google-analytics",
"namespace" : "tap_google_analytics",
"variant" : "matatika",
"label" : "Google Analytics",
"description" : "Google Analytics is a web analytics service that provides insights into website traffic and user behavior.\n\nGoogle Analytics allows website owners to track and analyze various metrics related to their website's performance, such as the number of visitors, pageviews, bounce rate, and average session duration. It also provides information on the demographics and interests of website visitors, as well as the sources of traffic, including organic search, paid search, social media, and referrals. This data can be used to optimize website content and marketing strategies, as well as to measure the effectiveness of advertising campaigns. Additionally, Google Analytics offers advanced features such as goal tracking, e-commerce tracking, and custom reporting, making it a powerful tool for businesses of all sizes.\n### Prerequisites\nTo obtain the OAuth identity provider authorization endpoint used to create and refresh tokens, you need to create a project in the Google API Console and enable the Google Analytics API. Then, you can create OAuth 2.0 credentials and configure the authorized redirect URIs. The authorization endpoint will be provided in the credentials.\n\nThe OAuth scopes you need to request access to depend on the specific data you want to access in Google Analytics. For example, if you want to read data from a specific view, you will need to request the \"https://www.googleapis.com/auth/analytics.readonly\" scope. You can find a list of available scopes in the Google Analytics API documentation.\n\nTo obtain the Access Token and OAuth Refresh Token, you need to authenticate the user and obtain their consent to access their Google Analytics data. This can be done using the Google Sign-In API or the OAuth 2.0 authorization flow. Once the user has granted access, you will receive an Access Token and a Refresh Token that you can use to make API requests.\n\nTo obtain the View ID, you need to log in to your Google Analytics account and navigate to the Admin section. From there, you can select the account, property, and view that you want to access and find the View ID in the View Settings.",
"logoUrl" : "/assets/images/datasource/tap-google-analytics.svg",
"hidden" : false,
"docs" : "https://www.matatika.com/docs/instant-insights/tap-google-analytics/",
"pipUrl" : "git+https://github.com/Matatika/[email protected]",
"repo" : "https://github.com/Matatika/tap-google-analytics",
"capabilities" : [ "DISCOVER", "CATALOG", "STATE" ],
"select" : [ ],
"update" : { },
"vars" : { },
"settings" : [ {
"name" : "oauth_credentials.authorization_url",
"aliases" : [ ],
"label" : "OAuth identity provider authorization endpoint used create and refresh tokens",
"value" : "https://oauth2.googleapis.com/token",
"options" : [ ],
"kind" : "STRING",
"description" : "The endpoint used to create and refresh OAuth tokens.",
"hidden" : true,
"sensitive" : false,
"required" : "true",
"protected" : false
}, {
"name" : "oauth_credentials.scope",
"aliases" : [ ],
"label" : "OAuth scopes we need to request access to",
"value" : "profile email https://www.googleapis.com/auth/analytics.readonly",
"options" : [ ],
"kind" : "STRING",
"description" : "The specific scopes we need to request access to in order to connect to Google Analytics.",
"hidden" : true,
"sensitive" : false,
"required" : "true",
"protected" : false
}, {
"name" : "oauth_credentials.access_token",
"aliases" : [ ],
"label" : "Access Token",
"options" : [ ],
"kind" : "STRING",
"description" : "The token used to authenticate and authorize API requests.",
"hidden" : true,
"sensitive" : false,
"required" : "true",
"protected" : false
}, {
"name" : "oauth_credentials.refresh_token",
"aliases" : [ ],
"label" : "OAuth Refresh Token",
"options" : [ ],
"kind" : "STRING",
"description" : "The token used to refresh the access token when it expires.",
"hidden" : true,
"sensitive" : false,
"required" : "true",
"protected" : false
}, {
"name" : "oauth_credentials.refresh_proxy_url",
"aliases" : [ ],
"label" : "Optional - will be called with 'oauth_credentials.refresh_token' to refresh the access token",
"options" : [ ],
"kind" : "STRING",
"description" : "An optional function that will be called to refresh the access token using the refresh token.",
"hidden" : true,
"sensitive" : false,
"protected" : false
}, {
"name" : "oauth_credentials.refresh_proxy_url_auth",
"aliases" : [ ],
"label" : "Optional - Sets Authorization header on 'oauth_credentials.refresh_url' request",
"options" : [ ],
"kind" : "STRING",
"description" : "An optional setting that sets the Authorization header on the refresh URL request.",
"hidden" : true,
"sensitive" : false,
"protected" : false
}, {
"name" : "oauth_credentials.client_id",
"aliases" : [ ],
"label" : "Optional - OAuth Client ID used if refresh_proxy_url not supplied",
"options" : [ ],
"kind" : "STRING",
"description" : "An optional OAuth Client ID used if the refresh proxy URL is not supplied.",
"hidden" : true,
"sensitive" : false,
"protected" : false
}, {
"name" : "oauth_credentials.client_secret",
"aliases" : [ ],
"label" : "Optional - OAuth Client Secret used if refresh_proxy_url not supplied",
"options" : [ ],
"kind" : "STRING",
"description" : "An optional OAuth Client Secret used if the refresh proxy URL is not supplied.",
"hidden" : true,
"sensitive" : false,
"protected" : false
}, {
"name" : "view_id",
"aliases" : [ ],
"label" : "View ID",
"options" : [ ],
"placeholder" : "Ex. 198343027",
"kind" : "STRING",
"description" : "The ID of the Google Analytics view to retrieve data from.",
"hidden" : false,
"sensitive" : false,
"required" : "true",
"protected" : false
}, {
"name" : "reports",
"aliases" : [ ],
"label" : "Reports",
"options" : [ ],
"placeholder" : "Ex. my_report_definition.json",
"kind" : "STRING",
"description" : "The specific reports to retrieve data from in the Google Analytics view.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "start_date",
"aliases" : [ ],
"label" : "Start date",
"options" : [ ],
"kind" : "DATE_ISO8601",
"description" : "The start date for the date range of data to retrieve.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "end_date",
"aliases" : [ ],
"label" : "End date",
"options" : [ ],
"kind" : "DATE_ISO8601",
"description" : "The end date for the date range of data to retrieve.",
"hidden" : false,
"sensitive" : false,
"protected" : false
} ],
"variants" : [ ],
"commands" : { },
"matatikaHidden" : false,
"requires" : [ ],
"fullDescription" : "Google Analytics is a web analytics service that provides insights into website traffic and user behavior.\n\nGoogle Analytics allows website owners to track and analyze various metrics related to their website's performance, such as the number of visitors, pageviews, bounce rate, and average session duration. It also provides information on the demographics and interests of website visitors, as well as the sources of traffic, including organic search, paid search, social media, and referrals. This data can be used to optimize website content and marketing strategies, as well as to measure the effectiveness of advertising campaigns. Additionally, Google Analytics offers advanced features such as goal tracking, e-commerce tracking, and custom reporting, making it a powerful tool for businesses of all sizes.\n### Prerequisites\nTo obtain the OAuth identity provider authorization endpoint used to create and refresh tokens, you need to create a project in the Google API Console and enable the Google Analytics API. Then, you can create OAuth 2.0 credentials and configure the authorized redirect URIs. The authorization endpoint will be provided in the credentials.\n\nThe OAuth scopes you need to request access to depend on the specific data you want to access in Google Analytics. For example, if you want to read data from a specific view, you will need to request the \"https://www.googleapis.com/auth/analytics.readonly\" scope. You can find a list of available scopes in the Google Analytics API documentation.\n\nTo obtain the Access Token and OAuth Refresh Token, you need to authenticate the user and obtain their consent to access their Google Analytics data. This can be done using the Google Sign-In API or the OAuth 2.0 authorization flow. Once the user has granted access, you will receive an Access Token and a Refresh Token that you can use to make API requests.\n\nTo obtain the View ID, you need to log in to your Google Analytics account and navigate to the Admin section. From there, you can select the account, property, and view that you want to access and find the View ID in the View Settings.\n\n## Settings\n\n\n### View ID\n\nThe ID of the Google Analytics view to retrieve data from.\n\n### Reports\n\nThe specific reports to retrieve data from in the Google Analytics view.\n\n### Start date\n\nThe start date for the date range of data to retrieve.\n\n### End date\n\nThe end date for the date range of data to retrieve.",
"_links" : {
"self" : {
"href" : "https://app.matatika.com/api/dataplugins/6b89da64-1538-4519-a188-5ff173208879"
},
"update dataplugin" : {
"href" : "https://app.matatika.com/api/workspaces/52652da2-a249-407e-a8c6-4e045838a591/dataplugins/6b89da64-1538-4519-a188-5ff173208879",
"type" : "PUT"
}
}
}
},
"managed" : false,
"draft" : true,
"_links" : {
"self" : {
"href" : "https://app.matatika.com/api/datacomponents/b6e29a0b-8836-4e3a-bbfd-f85163798009"
},
"update datacomponent" : {
"href" : "https://app.matatika.com/api/datacomponents/b6e29a0b-8836-4e3a-bbfd-f85163798009"
},
"delete datacomponent" : {
"href" : "https://app.matatika.com/api/datacomponents/b6e29a0b-8836-4e3a-bbfd-f85163798009"
}
}
}, {
"id" : "a604ff5e-5fae-4578-ae4a-650091f015f3",
"created" : "2025-06-19T10:19:42.42539",
"lastModified" : "2025-06-19T10:19:42.425391",
"name" : "Warehouse",
"properties" : {
"password" : "7PhOTBc5_2SMTn3on_ntw8iRA1",
"dbname" : "axvzzfd",
"default_target_schema" : "analytics",
"port" : "5432",
"host" : "sharp-banana2.postgres.database.azure.com",
"user" : "axvzzfd"
},
"commands" : { },
"dataPlugin" : "loaders/target-postgres--matatika",
"_embedded" : {
"dataplugin" : {
"id" : "82bf91fa-8adf-4421-863e-9a888ac3b620",
"pluginType" : "LOADER",
"name" : "target-postgres",
"namespace" : "postgres_transferwise",
"variant" : "matatika",
"label" : "Postgres Warehouse",
"description" : "Postgres Warehouse is a data warehousing solution built on top of the Postgres database management system.\n\nPostgres Warehouse is designed to handle large volumes of data and complex queries, making it an ideal solution for businesses that need to store and analyze large amounts of data. It provides a number of features that are specifically tailored to data warehousing, such as columnar storage, parallel processing, and support for advanced analytics. Additionally, Postgres Warehouse is highly scalable, allowing businesses to easily add more resources as their data needs grow. Overall, Postgres Warehouse is a powerful and flexible data warehousing solution that can help businesses make better decisions by providing them with the insights they need to succeed.\n### Prerequisites\nThe process of obtaining the required settings for connecting to a Postgres Warehouse may vary depending on the specific setup and configuration of the database. However, here are some general ways to obtain each of the required settings:\n\n- User: The user is typically created when the database is set up. You can ask the database administrator or check the database documentation to find out the username.\n- Password: The password is also typically created when the database is set up. You can ask the database administrator or check the database documentation to find out the password.\n- Host: The host is the server where the database is located. You can ask the database administrator or check the database documentation to find out the host name or IP address.\n- Port: The port is the number that the database listens on for incoming connections. The default port for Postgres is 5432, but it may be different depending on the configuration. You can ask the database administrator or check the database documentation to find out the port number.\n- Database Name: The database name is the name of the specific database you want to connect to. You can ask the database administrator or check the database documentation to find out the database name.\n- Default Target Schema: The default target schema is the schema that you want to use as the default when connecting to the database. This may be set up by the database administrator or you may need to create it yourself. You can ask the database administrator or check the database documentation to find out the default target schema.",
"logoUrl" : "/assets/logos/loaders/postgres.png",
"hidden" : false,
"docs" : "https://www.matatika.com/data-details/target-postgres/",
"pipUrl" : "git+https://github.com/Matatika/[email protected]",
"repo" : "git+https://github.com/Matatika/[email protected]",
"capabilities" : [ ],
"select" : [ ],
"update" : { },
"vars" : { },
"settings" : [ {
"name" : "user",
"aliases" : [ "username" ],
"label" : "User",
"options" : [ ],
"kind" : "STRING",
"description" : "The username used to connect to the Postgres Warehouse.",
"hidden" : false,
"sensitive" : false,
"required" : "true",
"protected" : false
}, {
"name" : "password",
"aliases" : [ ],
"label" : "Password",
"options" : [ ],
"kind" : "STRING",
"description" : "The password used to authenticate the user.",
"hidden" : false,
"sensitive" : true,
"required" : "true",
"protected" : false
}, {
"name" : "host",
"aliases" : [ "address" ],
"label" : "Host",
"options" : [ ],
"kind" : "STRING",
"description" : "The hostname or IP address of the Postgres Warehouse server.",
"hidden" : false,
"sensitive" : false,
"required" : "true",
"protected" : false
}, {
"name" : "port",
"aliases" : [ ],
"label" : "Port",
"value" : "5432",
"options" : [ ],
"kind" : "INTEGER",
"description" : "The port number used to connect to the Postgres Warehouse server.",
"hidden" : false,
"sensitive" : false,
"required" : "true",
"protected" : false
}, {
"name" : "dbname",
"aliases" : [ "database" ],
"label" : "Database Name",
"options" : [ ],
"kind" : "STRING",
"description" : "The name of the database to connect to.",
"hidden" : false,
"sensitive" : false,
"required" : "true",
"protected" : false
}, {
"name" : "default_target_schema",
"aliases" : [ ],
"label" : "Default Target Schema",
"value" : "analytics",
"options" : [ ],
"kind" : "STRING",
"description" : "The default schema to use when writing data to the Postgres Warehouse.",
"hidden" : false,
"sensitive" : false,
"required" : "true",
"protected" : false
}, {
"name" : "state_schema",
"aliases" : [ ],
"label" : "State Schema",
"value" : "public",
"options" : [ ],
"kind" : "STRING",
"description" : "The default schema to use when writing Meltano state.",
"hidden" : false,
"sensitive" : false,
"required" : "true",
"protected" : false
}, {
"name" : "ssl",
"aliases" : [ ],
"label" : "SSL",
"value" : "false",
"options" : [ ],
"kind" : "BOOLEAN",
"description" : "Whether or not to use SSL encryption when connecting to the Postgres Warehouse.",
"hidden" : true,
"sensitive" : false,
"protected" : false,
"value_post_processor" : "STRINGIFY"
}, {
"name" : "batch_size_rows",
"aliases" : [ ],
"label" : "Batch Size Rows",
"value" : "100000",
"options" : [ ],
"kind" : "INTEGER",
"description" : "The number of rows to write to the Postgres Warehouse in each batch.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "underscore_camel_case_fields",
"aliases" : [ ],
"label" : "Underscore Camel Case Fields",
"value" : "true",
"options" : [ ],
"kind" : "BOOLEAN",
"description" : "Whether or not to convert field names from camel case to underscore-separated format.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "flush_all_streams",
"aliases" : [ ],
"label" : "Flush All Streams",
"value" : "true",
"options" : [ ],
"kind" : "BOOLEAN",
"description" : "Whether or not to flush all streams to the Postgres Warehouse before closing the connection.",
"hidden" : true,
"sensitive" : false,
"protected" : false
}, {
"name" : "parallelism",
"aliases" : [ ],
"label" : "Parallelism",
"value" : "-1",
"options" : [ ],
"kind" : "INTEGER",
"description" : "The number of threads to use when writing data to the Postgres Warehouse.",
"hidden" : true,
"sensitive" : false,
"protected" : false
}, {
"name" : "parallelism_max",
"aliases" : [ ],
"label" : "Max Parallelism",
"value" : "16",
"options" : [ ],
"kind" : "INTEGER",
"description" : "The maximum number of threads to use when writing data to the Postgres Warehouse.",
"hidden" : true,
"sensitive" : false,
"protected" : false
}, {
"name" : "default_target_schema_select_permission",
"aliases" : [ ],
"label" : "Default Target Schema Select Permission",
"options" : [ ],
"kind" : "STRING",
"description" : "The permission level required to select data from the default target schema.",
"hidden" : true,
"sensitive" : false,
"protected" : false
}, {
"name" : "schema_mapping",
"aliases" : [ ],
"label" : "Schema Mapping",
"options" : [ ],
"kind" : "STRING",
"description" : "A mapping of source schema names to target schema names.",
"hidden" : true,
"sensitive" : false,
"protected" : false
}, {
"name" : "add_metadata_columns",
"aliases" : [ ],
"label" : "Add Metadata Columns",
"value" : "true",
"options" : [ ],
"kind" : "BOOLEAN",
"description" : "Whether or not to add metadata columns to the target table.",
"hidden" : true,
"sensitive" : false,
"protected" : false
}, {
"name" : "hard_delete",
"aliases" : [ ],
"label" : "Hard Delete",
"value" : "false",
"options" : [ ],
"kind" : "BOOLEAN",
"description" : "Whether or not to perform hard deletes when deleting data from the Postgres Warehouse.",
"hidden" : true,
"sensitive" : false,
"protected" : false
}, {
"name" : "data_flattening_max_level",
"aliases" : [ ],
"label" : "Data Flattening Max Level",
"value" : "10",
"options" : [ ],
"kind" : "INTEGER",
"description" : "The maximum level of nested data structures to flatten when writing data to the Postgres Warehouse.",
"hidden" : true,
"sensitive" : false,
"protected" : false
}, {
"name" : "primary_key_required",
"aliases" : [ ],
"label" : "Primary Key Required",
"value" : "false",
"options" : [ ],
"kind" : "BOOLEAN",
"description" : "Whether or not a primary key is required for the target table.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "validate_records",
"aliases" : [ ],
"label" : "Validate Records",
"value" : "false",
"options" : [ ],
"kind" : "BOOLEAN",
"description" : "Whether or not to validate records before writing them to the Postgres Warehouse.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "temp_dir",
"aliases" : [ ],
"label" : "Temporary Directory",
"options" : [ ],
"kind" : "STRING",
"description" : "The directory to use for temporary files when writing data to the Postgres Warehouse.",
"hidden" : true,
"sensitive" : false,
"protected" : false
}, {
"name" : "ssh_tunnel.host",
"aliases" : [ ],
"label" : "SSH Tunnel Host",
"options" : [ ],
"kind" : "STRING",
"description" : "Host of the bastion host, this is the host we'll connect to via ssh",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "ssh_tunnel.port",
"aliases" : [ ],
"label" : "SSH Tunnel Port",
"value" : "22",
"options" : [ ],
"kind" : "INTEGER",
"description" : "Port to connect to bastion host",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "ssh_tunnel.private_key",
"aliases" : [ ],
"label" : "SSH Tunnel Private Key",
"options" : [ ],
"kind" : "PASSWORD",
"description" : "A base64 encoded Private Key for authentication to the bastion host w/ key pair auth",
"hidden" : false,
"sensitive" : true,
"encoding" : "BASE64",
"protected" : false
}, {
"name" : "ssh_tunnel.private_key_password",
"aliases" : [ ],
"label" : "SSH Tunnel Private Key Password",
"options" : [ ],
"kind" : "PASSWORD",
"description" : "Private Key Password, leave None if no password is set",
"hidden" : false,
"sensitive" : true,
"protected" : false
}, {
"name" : "ssh_tunnel.username",
"aliases" : [ ],
"label" : "SSH Tunnel Username",
"options" : [ ],
"kind" : "STRING",
"description" : "Username to connect to bastion host",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "ssh_tunnel.password",
"aliases" : [ ],
"label" : "SSH Tunnel Password",
"options" : [ ],
"kind" : "STRING",
"description" : "Password to connect to bastion host w/ basic auth",
"hidden" : false,
"sensitive" : true,
"protected" : false
}, {
"name" : "ssh_tunnel.host_setting_name",
"aliases" : [ ],
"value" : "host",
"options" : [ ],
"kind" : "STRING",
"hidden" : true,
"sensitive" : false,
"protected" : false
}, {
"name" : "ssh_tunnel.port_setting_name",
"aliases" : [ ],
"value" : "port",
"options" : [ ],
"kind" : "STRING",
"hidden" : true,
"sensitive" : false,
"protected" : false
} ],
"variants" : [ ],
"commands" : { },
"matatikaHidden" : false,
"requires" : [ ],
"fullDescription" : "Postgres Warehouse is a data warehousing solution built on top of the Postgres database management system.\n\nPostgres Warehouse is designed to handle large volumes of data and complex queries, making it an ideal solution for businesses that need to store and analyze large amounts of data. It provides a number of features that are specifically tailored to data warehousing, such as columnar storage, parallel processing, and support for advanced analytics. Additionally, Postgres Warehouse is highly scalable, allowing businesses to easily add more resources as their data needs grow. Overall, Postgres Warehouse is a powerful and flexible data warehousing solution that can help businesses make better decisions by providing them with the insights they need to succeed.\n### Prerequisites\nThe process of obtaining the required settings for connecting to a Postgres Warehouse may vary depending on the specific setup and configuration of the database. However, here are some general ways to obtain each of the required settings:\n\n- User: The user is typically created when the database is set up. You can ask the database administrator or check the database documentation to find out the username.\n- Password: The password is also typically created when the database is set up. You can ask the database administrator or check the database documentation to find out the password.\n- Host: The host is the server where the database is located. You can ask the database administrator or check the database documentation to find out the host name or IP address.\n- Port: The port is the number that the database listens on for incoming connections. The default port for Postgres is 5432, but it may be different depending on the configuration. You can ask the database administrator or check the database documentation to find out the port number.\n- Database Name: The database name is the name of the specific database you want to connect to. You can ask the database administrator or check the database documentation to find out the database name.\n- Default Target Schema: The default target schema is the schema that you want to use as the default when connecting to the database. This may be set up by the database administrator or you may need to create it yourself. You can ask the database administrator or check the database documentation to find out the default target schema.\n\n## Settings\n\n\n### User\n\nThe username used to connect to the Postgres Warehouse.\n\n### Password\n\nThe password used to authenticate the user.\n\n### Host\n\nThe hostname or IP address of the Postgres Warehouse server.\n\n### Port\n\nThe port number used to connect to the Postgres Warehouse server.\n\n### Database Name\n\nThe name of the database to connect to.\n\n### Default Target Schema\n\nThe default schema to use when writing data to the Postgres Warehouse.\n\n### State Schema\n\nThe default schema to use when writing Meltano state.\n\n### Batch Size Rows\n\nThe number of rows to write to the Postgres Warehouse in each batch.\n\n### Underscore Camel Case Fields\n\nWhether or not to convert field names from camel case to underscore-separated format.\n\n### Primary Key Required\n\nWhether or not a primary key is required for the target table.\n\n### Validate Records\n\nWhether or not to validate records before writing them to the Postgres Warehouse.\n\n### SSH Tunnel Host\n\nHost of the bastion host, this is the host we'll connect to via ssh\n\n### SSH Tunnel Port\n\nPort to connect to bastion host\n\n### SSH Tunnel Private Key\n\nA base64 encoded Private Key for authentication to the bastion host w/ key pair auth\n\n### SSH Tunnel Private Key Password\n\nPrivate Key Password, leave None if no password is set\n\n### SSH Tunnel Username\n\nUsername to connect to bastion host\n\n### SSH Tunnel Password\n\nPassword to connect to bastion host w/ basic auth",
"_links" : {
"self" : {
"href" : "https://app.matatika.com/api/dataplugins/82bf91fa-8adf-4421-863e-9a888ac3b620"
},
"update dataplugin" : {
"href" : "https://app.matatika.com/api/workspaces/52652da2-a249-407e-a8c6-4e045838a591/dataplugins/82bf91fa-8adf-4421-863e-9a888ac3b620",
"type" : "PUT"
}
}
}
},
"managed" : true,
"draft" : false,
"_links" : {
"self" : {
"href" : "https://app.matatika.com/api/datacomponents/a604ff5e-5fae-4578-ae4a-650091f015f3"
},
"update datacomponent" : {
"href" : "https://app.matatika.com/api/datacomponents/a604ff5e-5fae-4578-ae4a-650091f015f3"
},
"delete datacomponent" : {
"href" : "https://app.matatika.com/api/datacomponents/a604ff5e-5fae-4578-ae4a-650091f015f3"
}
}
}, {
"id" : "a0195563-9e63-480a-93e9-731670490a91",
"created" : "2025-06-19T10:19:42.489029",
"lastModified" : "2025-06-19T10:19:42.489029",
"name" : "dbt",
"properties" : { },
"commands" : {
"compile" : {
"args" : "compile",
"description" : "Generates executable SQL from source model, test, and analysis files. Compiled SQL files are written to the target/ directory."
},
"seed" : {
"args" : "seed",
"description" : "Load data from csv files into your data warehouse."
},
"test" : {
"args" : "test",
"description" : "Runs tests on data in deployed models."
},
"docs-generate" : {
"args" : "docs generate",
"description" : "Generate documentation artifacts for your project."
},
"deps" : {
"args" : "deps",
"description" : "Pull the most recent version of the dependencies listed in packages.yml"
},
"run" : {
"args" : "run",
"description" : "Compile SQL and execute against the current target database."
},
"clean" : {
"args" : "clean",
"description" : "Delete all folders in the clean-targets list (usually the dbt_modules and target directories.)"
},
"snapshot" : {
"args" : "snapshot",
"description" : "Execute snapshots defined in your project."
}
},
"dataPlugin" : "transformers/dbt--dbt-labs",
"_embedded" : {
"dataplugin" : {
"id" : "c3dce0d8-4025-4484-9018-964d5ae3a740",
"pluginType" : "TRANSFORMER",
"name" : "dbt",
"namespace" : "dbt",
"variant" : "dbt-labs",
"label" : "dbt",
"description" : " Power your project transformations with dbt™, a SQL-first transformation tool that enables analytics engineers to develop transformations with code.\n\n***Version Control and CI/CD***\n\nUse Matatika to deploy and promote changes between dev, UAT, and production environments.\n\n***Test and Document***\n\nUse Matatika to develop and test every model prior to production release, and share dynamically generated documentation with all stakeholders.\n\n***Develop***\n\nWrite modular data transformations in .sql – Matatika together with dbt handles the chore of dependency management. ",
"logoUrl" : "/assets/images/transformer/dbt.png",
"hidden" : false,
"docs" : "https://www.matatika.com/data-details/dbt/",
"pipUrl" : "dbt-core~=1.7.0 dbt-postgres~=1.7.0 dbt-snowflake~=1.7.0 dbt-bigquery~=1.7.0 certifi==2025.1.31",
"repo" : "https://github.com/dbt-labs/dbt-core",
"capabilities" : [ ],
"select" : [ ],
"update" : { },
"vars" : { },
"settings" : [ {
"name" : "project_dir",
"aliases" : [ ],
"value" : "$MELTANO_PROJECT_ROOT/transform",
"options" : [ ],
"kind" : "STRING",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "profiles_dir",
"aliases" : [ ],
"value" : "$MELTANO_PROJECT_ROOT/transform/profile",
"options" : [ ],
"kind" : "STRING",
"hidden" : false,
"sensitive" : false,
"env" : "DBT_PROFILES_DIR",
"protected" : false
}, {
"name" : "target",
"aliases" : [ ],
"value" : "$MELTANO_LOAD__DIALECT",
"options" : [ ],
"kind" : "STRING",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "source_schema",
"aliases" : [ ],
"value" : "$MELTANO_LOAD__TARGET_SCHEMA",
"options" : [ ],
"kind" : "STRING",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "target_schema",
"aliases" : [ ],
"value" : "analytics",
"options" : [ ],
"kind" : "STRING",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "target_path",
"aliases" : [ ],
"value" : "$MELTANO_PROJECT_ROOT/.meltano/transformers/dbt/target",
"options" : [ ],
"kind" : "STRING",
"description" : "Output path for dbt generated artifacts",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "models",
"aliases" : [ ],
"value" : "$MELTANO_TRANSFORM__PACKAGE_NAME $MELTANO_EXTRACTOR_NAMESPACE my_meltano_project",
"options" : [ ],
"kind" : "STRING",
"hidden" : false,
"sensitive" : false,
"protected" : false
} ],
"variants" : [ ],
"commands" : {
"compile" : {
"args" : "compile",
"description" : "Generates executable SQL from source model, test, and analysis files. Compiled SQL files are written to the target/ directory."
},
"seed" : {
"args" : "seed",
"description" : "Load data from csv files into your data warehouse."
},
"test" : {
"args" : "test",
"description" : "Runs tests on data in deployed models."
},
"docs-generate" : {
"args" : "docs generate",
"description" : "Generate documentation artifacts for your project."
},
"deps" : {
"args" : "deps",
"description" : "Pull the most recent version of the dependencies listed in packages.yml"
},
"run" : {
"args" : "run",
"description" : "Compile SQL and execute against the current target database."
},
"clean" : {
"args" : "clean",
"description" : "Delete all folders in the clean-targets list (usually the dbt_modules and target directories.)"
},
"snapshot" : {
"args" : "snapshot",
"description" : "Execute snapshots defined in your project."
}
},
"matatikaHidden" : false,
"requires" : [ {
"id" : "e6c1ad3d-ebf5-4c4a-b129-f68156b47555",
"pluginType" : "FILE",
"name" : "files-dbt",
"namespace" : "dbt",
"variant" : "matatika",
"description" : " Files dbt is a file bundle that automatically configures your project to run transforms with dbt.\nThe bundle includes template project configuration:\n\n- transform/models (directory)\n- transform/profile/profiles.yml\n- transform/dbt_project.yml\n- transform/.gitignore\n- transform/macros/centralize_test_failures.sql\n",
"hidden" : false,
"pipUrl" : "git+https://github.com/Matatika/[email protected]",
"repo" : "https://github.com/Matatika/files-dbt",
"capabilities" : [ ],
"select" : [ ],
"update" : {
"transform/profile/profiles.yml" : "true"
},
"vars" : { },
"settings" : [ ],
"variants" : [ ],
"commands" : { },
"matatikaHidden" : false,
"requires" : [ ],
"fullDescription" : " Files dbt is a file bundle that automatically configures your project to run transforms with dbt.\nThe bundle includes template project configuration:\n\n- transform/models (directory)\n- transform/profile/profiles.yml\n- transform/dbt_project.yml\n- transform/.gitignore\n- transform/macros/centralize_test_failures.sql\n"
} ],
"fullDescription" : " Power your project transformations with dbt™, a SQL-first transformation tool that enables analytics engineers to develop transformations with code.\n\n***Version Control and CI/CD***\n\nUse Matatika to deploy and promote changes between dev, UAT, and production environments.\n\n***Test and Document***\n\nUse Matatika to develop and test every model prior to production release, and share dynamically generated documentation with all stakeholders.\n\n***Develop***\n\nWrite modular data transformations in .sql – Matatika together with dbt handles the chore of dependency management. \n\n## Settings\n\n\n### target_path\n\nOutput path for dbt generated artifacts",
"_links" : {
"self" : {
"href" : "https://app.matatika.com/api/dataplugins/c3dce0d8-4025-4484-9018-964d5ae3a740"
},
"update dataplugin" : {
"href" : "https://app.matatika.com/api/workspaces/52652da2-a249-407e-a8c6-4e045838a591/dataplugins/c3dce0d8-4025-4484-9018-964d5ae3a740",
"type" : "PUT"
}
}
}
},
"managed" : true,
"draft" : false,
"_links" : {
"self" : {
"href" : "https://app.matatika.com/api/datacomponents/a0195563-9e63-480a-93e9-731670490a91"
},
"update datacomponent" : {
"href" : "https://app.matatika.com/api/datacomponents/a0195563-9e63-480a-93e9-731670490a91"
},
"delete datacomponent" : {
"href" : "https://app.matatika.com/api/datacomponents/a0195563-9e63-480a-93e9-731670490a91"
}
}
} ]
},
"_links" : {
"update pipeline" : {
"href" : "https://app.matatika.com/api/workspaces/52652da2-a249-407e-a8c6-4e045838a591/pipelines/f42e86bc-e5f7-4965-9ee9-92bdf7a5ac36",
"type" : "PUT"
},
"delete pipeline" : {
"href" : "https://app.matatika.com/api/pipelines/f42e86bc-e5f7-4965-9ee9-92bdf7a5ac36",
"type" : "DELETE"
},
"draft pipeline" : {
"href" : "https://app.matatika.com/api/workspaces/52652da2-a249-407e-a8c6-4e045838a591/pipelines/f42e86bc-e5f7-4965-9ee9-92bdf7a5ac36/draft",
"type" : "PUT"
},
"self" : {
"href" : "https://app.matatika.com/api/pipelines/f42e86bc-e5f7-4965-9ee9-92bdf7a5ac36"
},
"environment" : {
"href" : "https://app.matatika.com/api/pipelines/f42e86bc-e5f7-4965-9ee9-92bdf7a5ac36/environment"
},
"jobs" : {
"href" : "https://app.matatika.com/api/pipelines/f42e86bc-e5f7-4965-9ee9-92bdf7a5ac36/jobs",
"type" : "GET"
},
"metrics" : {
"href" : "https://app.matatika.com/api/pipelines/f42e86bc-e5f7-4965-9ee9-92bdf7a5ac36/metrics"
},
"add subscription" : {
"href" : "https://app.matatika.com/api/pipelines/f42e86bc-e5f7-4965-9ee9-92bdf7a5ac36/subscriptions"
},
"verify pipeline" : {
"href" : "https://app.matatika.com/api/pipelines/f42e86bc-e5f7-4965-9ee9-92bdf7a5ac36/verification",
"type" : "POST"
},
"create job" : {
"href" : "https://app.matatika.com/api/pipelines/f42e86bc-e5f7-4965-9ee9-92bdf7a5ac36/jobs",
"type" : "POST"
}
}
}
Create or update a pipeline as a draft
PUT
/api/workspaces/{workspace-id}/pipelines/{pipeline-id}/draft
Creates or updates the pipeline {pipeline-id} in the workspace {workspace-id} as a draft.
Prerequisites
- Workspace
{workspace-id}must exist
Request
Body
Pipeline resource.
{
"name" : "SIT-generated pipeline [2025-06-19T11:19:48.350863]",
"dataComponents" : [ "tap-google-analytics", "Warehouse", "dbt" ]
}
Example Snippets
cURL
curl -H "Authorization: Bearer $ACCESS_TOKEN" 'https://app.matatika.com:443/api/workspaces/52652da2-a249-407e-a8c6-4e045838a591/pipelines/180a0df1-e833-4aae-b6ef-ee98f0f07b50/draft' -i -X PUT \
-H 'Accept: application/json, application/javascript, text/javascript, text/json' \
-H 'Content-Type: application/json' \
-d '{
"name" : "SIT-generated pipeline [2025-06-19T11:19:48.350863]",
"dataComponents" : [ "tap-google-analytics", "Warehouse", "dbt" ]
}'
Python (requests)
import requests
url = "https://app.matatika.com:443/api/workspaces/52652da2-a249-407e-a8c6-4e045838a591/pipelines/180a0df1-e833-4aae-b6ef-ee98f0f07b50/draft"
data = {
"name" : "SIT-generated pipeline [2025-06-19T11:19:48.350863]",
"dataComponents" : [ "tap-google-analytics", "Warehouse", "dbt" ]
}
headers = {
'Authorization': ACCESS_TOKEN
}
response = requests.request("PUT", url, headers=headers, data=data)
print(response.text.encode('utf8'))
Response
200 OK / 201 Created
Pipeline with HAL links.
{
"id" : "180a0df1-e833-4aae-b6ef-ee98f0f07b50",
"status" : "DRAFT",
"name" : "SIT-generated pipeline [2025-06-19T11:19:48.350863]",
"timeout" : 0,
"maxRetries" : 0,
"created" : "2025-06-19T10:19:48.519054",
"lastModified" : "2025-06-19T10:19:48.519056",
"properties" : { },
"dataComponents" : [ "tap-google-analytics", "Warehouse", "dbt" ],
"actions" : [ ],
"triggeredBy" : [ ],
"_embedded" : {
"dataComponents" : [ {
"id" : "b6e29a0b-8836-4e3a-bbfd-f85163798009",
"created" : "2025-06-19T10:19:47.983989",
"lastModified" : "2025-06-19T10:19:47.98399",
"name" : "tap-google-analytics",
"properties" : { },
"commands" : { },
"dataPlugin" : "extractors/tap-google-analytics--matatika",
"_embedded" : {
"dataplugin" : {
"id" : "6b89da64-1538-4519-a188-5ff173208879",
"pluginType" : "EXTRACTOR",
"name" : "tap-google-analytics",
"namespace" : "tap_google_analytics",
"variant" : "matatika",
"label" : "Google Analytics",
"description" : "Google Analytics is a web analytics service that provides insights into website traffic and user behavior.\n\nGoogle Analytics allows website owners to track and analyze various metrics related to their website's performance, such as the number of visitors, pageviews, bounce rate, and average session duration. It also provides information on the demographics and interests of website visitors, as well as the sources of traffic, including organic search, paid search, social media, and referrals. This data can be used to optimize website content and marketing strategies, as well as to measure the effectiveness of advertising campaigns. Additionally, Google Analytics offers advanced features such as goal tracking, e-commerce tracking, and custom reporting, making it a powerful tool for businesses of all sizes.\n### Prerequisites\nTo obtain the OAuth identity provider authorization endpoint used to create and refresh tokens, you need to create a project in the Google API Console and enable the Google Analytics API. Then, you can create OAuth 2.0 credentials and configure the authorized redirect URIs. The authorization endpoint will be provided in the credentials.\n\nThe OAuth scopes you need to request access to depend on the specific data you want to access in Google Analytics. For example, if you want to read data from a specific view, you will need to request the \"https://www.googleapis.com/auth/analytics.readonly\" scope. You can find a list of available scopes in the Google Analytics API documentation.\n\nTo obtain the Access Token and OAuth Refresh Token, you need to authenticate the user and obtain their consent to access their Google Analytics data. This can be done using the Google Sign-In API or the OAuth 2.0 authorization flow. Once the user has granted access, you will receive an Access Token and a Refresh Token that you can use to make API requests.\n\nTo obtain the View ID, you need to log in to your Google Analytics account and navigate to the Admin section. From there, you can select the account, property, and view that you want to access and find the View ID in the View Settings.",
"logoUrl" : "/assets/images/datasource/tap-google-analytics.svg",
"hidden" : false,
"docs" : "https://www.matatika.com/docs/instant-insights/tap-google-analytics/",
"pipUrl" : "git+https://github.com/Matatika/[email protected]",
"repo" : "https://github.com/Matatika/tap-google-analytics",
"capabilities" : [ "DISCOVER", "CATALOG", "STATE" ],
"select" : [ ],
"update" : { },
"vars" : { },
"settings" : [ {
"name" : "oauth_credentials.authorization_url",
"aliases" : [ ],
"label" : "OAuth identity provider authorization endpoint used create and refresh tokens",
"value" : "https://oauth2.googleapis.com/token",
"options" : [ ],
"kind" : "STRING",
"description" : "The endpoint used to create and refresh OAuth tokens.",
"hidden" : true,
"sensitive" : false,
"required" : "true",
"protected" : false
}, {
"name" : "oauth_credentials.scope",
"aliases" : [ ],
"label" : "OAuth scopes we need to request access to",
"value" : "profile email https://www.googleapis.com/auth/analytics.readonly",
"options" : [ ],
"kind" : "STRING",
"description" : "The specific scopes we need to request access to in order to connect to Google Analytics.",
"hidden" : true,
"sensitive" : false,
"required" : "true",
"protected" : false
}, {
"name" : "oauth_credentials.access_token",
"aliases" : [ ],
"label" : "Access Token",
"options" : [ ],
"kind" : "STRING",
"description" : "The token used to authenticate and authorize API requests.",
"hidden" : true,
"sensitive" : false,
"required" : "true",
"protected" : false
}, {
"name" : "oauth_credentials.refresh_token",
"aliases" : [ ],
"label" : "OAuth Refresh Token",
"options" : [ ],
"kind" : "STRING",
"description" : "The token used to refresh the access token when it expires.",
"hidden" : true,
"sensitive" : false,
"required" : "true",
"protected" : false
}, {
"name" : "oauth_credentials.refresh_proxy_url",
"aliases" : [ ],
"label" : "Optional - will be called with 'oauth_credentials.refresh_token' to refresh the access token",
"options" : [ ],
"kind" : "STRING",
"description" : "An optional function that will be called to refresh the access token using the refresh token.",
"hidden" : true,
"sensitive" : false,
"protected" : false
}, {
"name" : "oauth_credentials.refresh_proxy_url_auth",
"aliases" : [ ],
"label" : "Optional - Sets Authorization header on 'oauth_credentials.refresh_url' request",
"options" : [ ],
"kind" : "STRING",
"description" : "An optional setting that sets the Authorization header on the refresh URL request.",
"hidden" : true,
"sensitive" : false,
"protected" : false
}, {
"name" : "oauth_credentials.client_id",
"aliases" : [ ],
"label" : "Optional - OAuth Client ID used if refresh_proxy_url not supplied",
"options" : [ ],
"kind" : "STRING",
"description" : "An optional OAuth Client ID used if the refresh proxy URL is not supplied.",
"hidden" : true,
"sensitive" : false,
"protected" : false
}, {
"name" : "oauth_credentials.client_secret",
"aliases" : [ ],
"label" : "Optional - OAuth Client Secret used if refresh_proxy_url not supplied",
"options" : [ ],
"kind" : "STRING",
"description" : "An optional OAuth Client Secret used if the refresh proxy URL is not supplied.",
"hidden" : true,
"sensitive" : false,
"protected" : false
}, {
"name" : "view_id",
"aliases" : [ ],
"label" : "View ID",
"options" : [ ],
"placeholder" : "Ex. 198343027",
"kind" : "STRING",
"description" : "The ID of the Google Analytics view to retrieve data from.",
"hidden" : false,
"sensitive" : false,
"required" : "true",
"protected" : false
}, {
"name" : "reports",
"aliases" : [ ],
"label" : "Reports",
"options" : [ ],
"placeholder" : "Ex. my_report_definition.json",
"kind" : "STRING",
"description" : "The specific reports to retrieve data from in the Google Analytics view.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "start_date",
"aliases" : [ ],
"label" : "Start date",
"options" : [ ],
"kind" : "DATE_ISO8601",
"description" : "The start date for the date range of data to retrieve.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "end_date",
"aliases" : [ ],
"label" : "End date",
"options" : [ ],
"kind" : "DATE_ISO8601",
"description" : "The end date for the date range of data to retrieve.",
"hidden" : false,
"sensitive" : false,
"protected" : false
} ],
"variants" : [ ],
"commands" : { },
"matatikaHidden" : false,
"requires" : [ ],
"fullDescription" : "Google Analytics is a web analytics service that provides insights into website traffic and user behavior.\n\nGoogle Analytics allows website owners to track and analyze various metrics related to their website's performance, such as the number of visitors, pageviews, bounce rate, and average session duration. It also provides information on the demographics and interests of website visitors, as well as the sources of traffic, including organic search, paid search, social media, and referrals. This data can be used to optimize website content and marketing strategies, as well as to measure the effectiveness of advertising campaigns. Additionally, Google Analytics offers advanced features such as goal tracking, e-commerce tracking, and custom reporting, making it a powerful tool for businesses of all sizes.\n### Prerequisites\nTo obtain the OAuth identity provider authorization endpoint used to create and refresh tokens, you need to create a project in the Google API Console and enable the Google Analytics API. Then, you can create OAuth 2.0 credentials and configure the authorized redirect URIs. The authorization endpoint will be provided in the credentials.\n\nThe OAuth scopes you need to request access to depend on the specific data you want to access in Google Analytics. For example, if you want to read data from a specific view, you will need to request the \"https://www.googleapis.com/auth/analytics.readonly\" scope. You can find a list of available scopes in the Google Analytics API documentation.\n\nTo obtain the Access Token and OAuth Refresh Token, you need to authenticate the user and obtain their consent to access their Google Analytics data. This can be done using the Google Sign-In API or the OAuth 2.0 authorization flow. Once the user has granted access, you will receive an Access Token and a Refresh Token that you can use to make API requests.\n\nTo obtain the View ID, you need to log in to your Google Analytics account and navigate to the Admin section. From there, you can select the account, property, and view that you want to access and find the View ID in the View Settings.\n\n## Settings\n\n\n### View ID\n\nThe ID of the Google Analytics view to retrieve data from.\n\n### Reports\n\nThe specific reports to retrieve data from in the Google Analytics view.\n\n### Start date\n\nThe start date for the date range of data to retrieve.\n\n### End date\n\nThe end date for the date range of data to retrieve.",
"_links" : {
"self" : {
"href" : "https://app.matatika.com/api/dataplugins/6b89da64-1538-4519-a188-5ff173208879"
},
"update dataplugin" : {
"href" : "https://app.matatika.com/api/workspaces/52652da2-a249-407e-a8c6-4e045838a591/dataplugins/6b89da64-1538-4519-a188-5ff173208879",
"type" : "PUT"
}
}
}
},
"managed" : false,
"draft" : true,
"_links" : {
"self" : {
"href" : "https://app.matatika.com/api/datacomponents/b6e29a0b-8836-4e3a-bbfd-f85163798009"
},
"update datacomponent" : {
"href" : "https://app.matatika.com/api/datacomponents/b6e29a0b-8836-4e3a-bbfd-f85163798009"
},
"delete datacomponent" : {
"href" : "https://app.matatika.com/api/datacomponents/b6e29a0b-8836-4e3a-bbfd-f85163798009"
}
}
}, {
"id" : "a604ff5e-5fae-4578-ae4a-650091f015f3",
"created" : "2025-06-19T10:19:42.42539",
"lastModified" : "2025-06-19T10:19:42.425391",
"name" : "Warehouse",
"properties" : {
"password" : "7PhOTBc5_2SMTn3on_ntw8iRA1",
"dbname" : "axvzzfd",
"default_target_schema" : "analytics",
"port" : "5432",
"host" : "sharp-banana2.postgres.database.azure.com",
"user" : "axvzzfd"
},
"commands" : { },
"dataPlugin" : "loaders/target-postgres--matatika",
"_embedded" : {
"dataplugin" : {
"id" : "82bf91fa-8adf-4421-863e-9a888ac3b620",
"pluginType" : "LOADER",
"name" : "target-postgres",
"namespace" : "postgres_transferwise",
"variant" : "matatika",
"label" : "Postgres Warehouse",
"description" : "Postgres Warehouse is a data warehousing solution built on top of the Postgres database management system.\n\nPostgres Warehouse is designed to handle large volumes of data and complex queries, making it an ideal solution for businesses that need to store and analyze large amounts of data. It provides a number of features that are specifically tailored to data warehousing, such as columnar storage, parallel processing, and support for advanced analytics. Additionally, Postgres Warehouse is highly scalable, allowing businesses to easily add more resources as their data needs grow. Overall, Postgres Warehouse is a powerful and flexible data warehousing solution that can help businesses make better decisions by providing them with the insights they need to succeed.\n### Prerequisites\nThe process of obtaining the required settings for connecting to a Postgres Warehouse may vary depending on the specific setup and configuration of the database. However, here are some general ways to obtain each of the required settings:\n\n- User: The user is typically created when the database is set up. You can ask the database administrator or check the database documentation to find out the username.\n- Password: The password is also typically created when the database is set up. You can ask the database administrator or check the database documentation to find out the password.\n- Host: The host is the server where the database is located. You can ask the database administrator or check the database documentation to find out the host name or IP address.\n- Port: The port is the number that the database listens on for incoming connections. The default port for Postgres is 5432, but it may be different depending on the configuration. You can ask the database administrator or check the database documentation to find out the port number.\n- Database Name: The database name is the name of the specific database you want to connect to. You can ask the database administrator or check the database documentation to find out the database name.\n- Default Target Schema: The default target schema is the schema that you want to use as the default when connecting to the database. This may be set up by the database administrator or you may need to create it yourself. You can ask the database administrator or check the database documentation to find out the default target schema.",
"logoUrl" : "/assets/logos/loaders/postgres.png",
"hidden" : false,
"docs" : "https://www.matatika.com/data-details/target-postgres/",
"pipUrl" : "git+https://github.com/Matatika/[email protected]",
"repo" : "git+https://github.com/Matatika/[email protected]",
"capabilities" : [ ],
"select" : [ ],
"update" : { },
"vars" : { },
"settings" : [ {
"name" : "user",
"aliases" : [ "username" ],
"label" : "User",
"options" : [ ],
"kind" : "STRING",
"description" : "The username used to connect to the Postgres Warehouse.",
"hidden" : false,
"sensitive" : false,
"required" : "true",
"protected" : false
}, {
"name" : "password",
"aliases" : [ ],
"label" : "Password",
"options" : [ ],
"kind" : "STRING",
"description" : "The password used to authenticate the user.",
"hidden" : false,
"sensitive" : true,
"required" : "true",
"protected" : false
}, {
"name" : "host",
"aliases" : [ "address" ],
"label" : "Host",
"options" : [ ],
"kind" : "STRING",
"description" : "The hostname or IP address of the Postgres Warehouse server.",
"hidden" : false,
"sensitive" : false,
"required" : "true",
"protected" : false
}, {
"name" : "port",
"aliases" : [ ],
"label" : "Port",
"value" : "5432",
"options" : [ ],
"kind" : "INTEGER",
"description" : "The port number used to connect to the Postgres Warehouse server.",
"hidden" : false,
"sensitive" : false,
"required" : "true",
"protected" : false
}, {
"name" : "dbname",
"aliases" : [ "database" ],
"label" : "Database Name",
"options" : [ ],
"kind" : "STRING",
"description" : "The name of the database to connect to.",
"hidden" : false,
"sensitive" : false,
"required" : "true",
"protected" : false
}, {
"name" : "default_target_schema",
"aliases" : [ ],
"label" : "Default Target Schema",
"value" : "analytics",
"options" : [ ],
"kind" : "STRING",
"description" : "The default schema to use when writing data to the Postgres Warehouse.",
"hidden" : false,
"sensitive" : false,
"required" : "true",
"protected" : false
}, {
"name" : "state_schema",
"aliases" : [ ],
"label" : "State Schema",
"value" : "public",
"options" : [ ],
"kind" : "STRING",
"description" : "The default schema to use when writing Meltano state.",
"hidden" : false,
"sensitive" : false,
"required" : "true",
"protected" : false
}, {
"name" : "ssl",
"aliases" : [ ],
"label" : "SSL",
"value" : "false",
"options" : [ ],
"kind" : "BOOLEAN",
"description" : "Whether or not to use SSL encryption when connecting to the Postgres Warehouse.",
"hidden" : true,
"sensitive" : false,
"protected" : false,
"value_post_processor" : "STRINGIFY"
}, {
"name" : "batch_size_rows",
"aliases" : [ ],
"label" : "Batch Size Rows",
"value" : "100000",
"options" : [ ],
"kind" : "INTEGER",
"description" : "The number of rows to write to the Postgres Warehouse in each batch.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "underscore_camel_case_fields",
"aliases" : [ ],
"label" : "Underscore Camel Case Fields",
"value" : "true",
"options" : [ ],
"kind" : "BOOLEAN",
"description" : "Whether or not to convert field names from camel case to underscore-separated format.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "flush_all_streams",
"aliases" : [ ],
"label" : "Flush All Streams",
"value" : "true",
"options" : [ ],
"kind" : "BOOLEAN",
"description" : "Whether or not to flush all streams to the Postgres Warehouse before closing the connection.",
"hidden" : true,
"sensitive" : false,
"protected" : false
}, {
"name" : "parallelism",
"aliases" : [ ],
"label" : "Parallelism",
"value" : "-1",
"options" : [ ],
"kind" : "INTEGER",
"description" : "The number of threads to use when writing data to the Postgres Warehouse.",
"hidden" : true,
"sensitive" : false,
"protected" : false
}, {
"name" : "parallelism_max",
"aliases" : [ ],
"label" : "Max Parallelism",
"value" : "16",
"options" : [ ],
"kind" : "INTEGER",
"description" : "The maximum number of threads to use when writing data to the Postgres Warehouse.",
"hidden" : true,
"sensitive" : false,
"protected" : false
}, {
"name" : "default_target_schema_select_permission",
"aliases" : [ ],
"label" : "Default Target Schema Select Permission",
"options" : [ ],
"kind" : "STRING",
"description" : "The permission level required to select data from the default target schema.",
"hidden" : true,
"sensitive" : false,
"protected" : false
}, {
"name" : "schema_mapping",
"aliases" : [ ],
"label" : "Schema Mapping",
"options" : [ ],
"kind" : "STRING",
"description" : "A mapping of source schema names to target schema names.",
"hidden" : true,
"sensitive" : false,
"protected" : false
}, {
"name" : "add_metadata_columns",
"aliases" : [ ],
"label" : "Add Metadata Columns",
"value" : "true",
"options" : [ ],
"kind" : "BOOLEAN",
"description" : "Whether or not to add metadata columns to the target table.",
"hidden" : true,
"sensitive" : false,
"protected" : false
}, {
"name" : "hard_delete",
"aliases" : [ ],
"label" : "Hard Delete",
"value" : "false",
"options" : [ ],
"kind" : "BOOLEAN",
"description" : "Whether or not to perform hard deletes when deleting data from the Postgres Warehouse.",
"hidden" : true,
"sensitive" : false,
"protected" : false
}, {
"name" : "data_flattening_max_level",
"aliases" : [ ],
"label" : "Data Flattening Max Level",
"value" : "10",
"options" : [ ],
"kind" : "INTEGER",
"description" : "The maximum level of nested data structures to flatten when writing data to the Postgres Warehouse.",
"hidden" : true,
"sensitive" : false,
"protected" : false
}, {
"name" : "primary_key_required",
"aliases" : [ ],
"label" : "Primary Key Required",
"value" : "false",
"options" : [ ],
"kind" : "BOOLEAN",
"description" : "Whether or not a primary key is required for the target table.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "validate_records",
"aliases" : [ ],
"label" : "Validate Records",
"value" : "false",
"options" : [ ],
"kind" : "BOOLEAN",
"description" : "Whether or not to validate records before writing them to the Postgres Warehouse.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "temp_dir",
"aliases" : [ ],
"label" : "Temporary Directory",
"options" : [ ],
"kind" : "STRING",
"description" : "The directory to use for temporary files when writing data to the Postgres Warehouse.",
"hidden" : true,
"sensitive" : false,
"protected" : false
}, {
"name" : "ssh_tunnel.host",
"aliases" : [ ],
"label" : "SSH Tunnel Host",
"options" : [ ],
"kind" : "STRING",
"description" : "Host of the bastion host, this is the host we'll connect to via ssh",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "ssh_tunnel.port",
"aliases" : [ ],
"label" : "SSH Tunnel Port",
"value" : "22",
"options" : [ ],
"kind" : "INTEGER",
"description" : "Port to connect to bastion host",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "ssh_tunnel.private_key",
"aliases" : [ ],
"label" : "SSH Tunnel Private Key",
"options" : [ ],
"kind" : "PASSWORD",
"description" : "A base64 encoded Private Key for authentication to the bastion host w/ key pair auth",
"hidden" : false,
"sensitive" : true,
"encoding" : "BASE64",
"protected" : false
}, {
"name" : "ssh_tunnel.private_key_password",
"aliases" : [ ],
"label" : "SSH Tunnel Private Key Password",
"options" : [ ],
"kind" : "PASSWORD",
"description" : "Private Key Password, leave None if no password is set",
"hidden" : false,
"sensitive" : true,
"protected" : false
}, {
"name" : "ssh_tunnel.username",
"aliases" : [ ],
"label" : "SSH Tunnel Username",
"options" : [ ],
"kind" : "STRING",
"description" : "Username to connect to bastion host",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "ssh_tunnel.password",
"aliases" : [ ],
"label" : "SSH Tunnel Password",
"options" : [ ],
"kind" : "STRING",
"description" : "Password to connect to bastion host w/ basic auth",
"hidden" : false,
"sensitive" : true,
"protected" : false
}, {
"name" : "ssh_tunnel.host_setting_name",
"aliases" : [ ],
"value" : "host",
"options" : [ ],
"kind" : "STRING",
"hidden" : true,
"sensitive" : false,
"protected" : false
}, {
"name" : "ssh_tunnel.port_setting_name",
"aliases" : [ ],
"value" : "port",
"options" : [ ],
"kind" : "STRING",
"hidden" : true,
"sensitive" : false,
"protected" : false
} ],
"variants" : [ ],
"commands" : { },
"matatikaHidden" : false,
"requires" : [ ],
"fullDescription" : "Postgres Warehouse is a data warehousing solution built on top of the Postgres database management system.\n\nPostgres Warehouse is designed to handle large volumes of data and complex queries, making it an ideal solution for businesses that need to store and analyze large amounts of data. It provides a number of features that are specifically tailored to data warehousing, such as columnar storage, parallel processing, and support for advanced analytics. Additionally, Postgres Warehouse is highly scalable, allowing businesses to easily add more resources as their data needs grow. Overall, Postgres Warehouse is a powerful and flexible data warehousing solution that can help businesses make better decisions by providing them with the insights they need to succeed.\n### Prerequisites\nThe process of obtaining the required settings for connecting to a Postgres Warehouse may vary depending on the specific setup and configuration of the database. However, here are some general ways to obtain each of the required settings:\n\n- User: The user is typically created when the database is set up. You can ask the database administrator or check the database documentation to find out the username.\n- Password: The password is also typically created when the database is set up. You can ask the database administrator or check the database documentation to find out the password.\n- Host: The host is the server where the database is located. You can ask the database administrator or check the database documentation to find out the host name or IP address.\n- Port: The port is the number that the database listens on for incoming connections. The default port for Postgres is 5432, but it may be different depending on the configuration. You can ask the database administrator or check the database documentation to find out the port number.\n- Database Name: The database name is the name of the specific database you want to connect to. You can ask the database administrator or check the database documentation to find out the database name.\n- Default Target Schema: The default target schema is the schema that you want to use as the default when connecting to the database. This may be set up by the database administrator or you may need to create it yourself. You can ask the database administrator or check the database documentation to find out the default target schema.\n\n## Settings\n\n\n### User\n\nThe username used to connect to the Postgres Warehouse.\n\n### Password\n\nThe password used to authenticate the user.\n\n### Host\n\nThe hostname or IP address of the Postgres Warehouse server.\n\n### Port\n\nThe port number used to connect to the Postgres Warehouse server.\n\n### Database Name\n\nThe name of the database to connect to.\n\n### Default Target Schema\n\nThe default schema to use when writing data to the Postgres Warehouse.\n\n### State Schema\n\nThe default schema to use when writing Meltano state.\n\n### Batch Size Rows\n\nThe number of rows to write to the Postgres Warehouse in each batch.\n\n### Underscore Camel Case Fields\n\nWhether or not to convert field names from camel case to underscore-separated format.\n\n### Primary Key Required\n\nWhether or not a primary key is required for the target table.\n\n### Validate Records\n\nWhether or not to validate records before writing them to the Postgres Warehouse.\n\n### SSH Tunnel Host\n\nHost of the bastion host, this is the host we'll connect to via ssh\n\n### SSH Tunnel Port\n\nPort to connect to bastion host\n\n### SSH Tunnel Private Key\n\nA base64 encoded Private Key for authentication to the bastion host w/ key pair auth\n\n### SSH Tunnel Private Key Password\n\nPrivate Key Password, leave None if no password is set\n\n### SSH Tunnel Username\n\nUsername to connect to bastion host\n\n### SSH Tunnel Password\n\nPassword to connect to bastion host w/ basic auth",
"_links" : {
"self" : {
"href" : "https://app.matatika.com/api/dataplugins/82bf91fa-8adf-4421-863e-9a888ac3b620"
},
"update dataplugin" : {
"href" : "https://app.matatika.com/api/workspaces/52652da2-a249-407e-a8c6-4e045838a591/dataplugins/82bf91fa-8adf-4421-863e-9a888ac3b620",
"type" : "PUT"
}
}
}
},
"managed" : true,
"draft" : false,
"_links" : {
"self" : {
"href" : "https://app.matatika.com/api/datacomponents/a604ff5e-5fae-4578-ae4a-650091f015f3"
},
"update datacomponent" : {
"href" : "https://app.matatika.com/api/datacomponents/a604ff5e-5fae-4578-ae4a-650091f015f3"
},
"delete datacomponent" : {
"href" : "https://app.matatika.com/api/datacomponents/a604ff5e-5fae-4578-ae4a-650091f015f3"
}
}
}, {
"id" : "a0195563-9e63-480a-93e9-731670490a91",
"created" : "2025-06-19T10:19:42.489029",
"lastModified" : "2025-06-19T10:19:42.489029",
"name" : "dbt",
"properties" : { },
"commands" : {
"compile" : {
"args" : "compile",
"description" : "Generates executable SQL from source model, test, and analysis files. Compiled SQL files are written to the target/ directory."
},
"seed" : {
"args" : "seed",
"description" : "Load data from csv files into your data warehouse."
},
"test" : {
"args" : "test",
"description" : "Runs tests on data in deployed models."
},
"docs-generate" : {
"args" : "docs generate",
"description" : "Generate documentation artifacts for your project."
},
"deps" : {
"args" : "deps",
"description" : "Pull the most recent version of the dependencies listed in packages.yml"
},
"run" : {
"args" : "run",
"description" : "Compile SQL and execute against the current target database."
},
"clean" : {
"args" : "clean",
"description" : "Delete all folders in the clean-targets list (usually the dbt_modules and target directories.)"
},
"snapshot" : {
"args" : "snapshot",
"description" : "Execute snapshots defined in your project."
}
},
"dataPlugin" : "transformers/dbt--dbt-labs",
"_embedded" : {
"dataplugin" : {
"id" : "c3dce0d8-4025-4484-9018-964d5ae3a740",
"pluginType" : "TRANSFORMER",
"name" : "dbt",
"namespace" : "dbt",
"variant" : "dbt-labs",
"label" : "dbt",
"description" : " Power your project transformations with dbt™, a SQL-first transformation tool that enables analytics engineers to develop transformations with code.\n\n***Version Control and CI/CD***\n\nUse Matatika to deploy and promote changes between dev, UAT, and production environments.\n\n***Test and Document***\n\nUse Matatika to develop and test every model prior to production release, and share dynamically generated documentation with all stakeholders.\n\n***Develop***\n\nWrite modular data transformations in .sql – Matatika together with dbt handles the chore of dependency management. ",
"logoUrl" : "/assets/images/transformer/dbt.png",
"hidden" : false,
"docs" : "https://www.matatika.com/data-details/dbt/",
"pipUrl" : "dbt-core~=1.7.0 dbt-postgres~=1.7.0 dbt-snowflake~=1.7.0 dbt-bigquery~=1.7.0 certifi==2025.1.31",
"repo" : "https://github.com/dbt-labs/dbt-core",
"capabilities" : [ ],
"select" : [ ],
"update" : { },
"vars" : { },
"settings" : [ {
"name" : "project_dir",
"aliases" : [ ],
"value" : "$MELTANO_PROJECT_ROOT/transform",
"options" : [ ],
"kind" : "STRING",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "profiles_dir",
"aliases" : [ ],
"value" : "$MELTANO_PROJECT_ROOT/transform/profile",
"options" : [ ],
"kind" : "STRING",
"hidden" : false,
"sensitive" : false,
"env" : "DBT_PROFILES_DIR",
"protected" : false
}, {
"name" : "target",
"aliases" : [ ],
"value" : "$MELTANO_LOAD__DIALECT",
"options" : [ ],
"kind" : "STRING",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "source_schema",
"aliases" : [ ],
"value" : "$MELTANO_LOAD__TARGET_SCHEMA",
"options" : [ ],
"kind" : "STRING",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "target_schema",
"aliases" : [ ],
"value" : "analytics",
"options" : [ ],
"kind" : "STRING",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "target_path",
"aliases" : [ ],
"value" : "$MELTANO_PROJECT_ROOT/.meltano/transformers/dbt/target",
"options" : [ ],
"kind" : "STRING",
"description" : "Output path for dbt generated artifacts",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "models",
"aliases" : [ ],
"value" : "$MELTANO_TRANSFORM__PACKAGE_NAME $MELTANO_EXTRACTOR_NAMESPACE my_meltano_project",
"options" : [ ],
"kind" : "STRING",
"hidden" : false,
"sensitive" : false,
"protected" : false
} ],
"variants" : [ ],
"commands" : {
"compile" : {
"args" : "compile",
"description" : "Generates executable SQL from source model, test, and analysis files. Compiled SQL files are written to the target/ directory."
},
"seed" : {
"args" : "seed",
"description" : "Load data from csv files into your data warehouse."
},
"test" : {
"args" : "test",
"description" : "Runs tests on data in deployed models."
},
"docs-generate" : {
"args" : "docs generate",
"description" : "Generate documentation artifacts for your project."
},
"deps" : {
"args" : "deps",
"description" : "Pull the most recent version of the dependencies listed in packages.yml"
},
"run" : {
"args" : "run",
"description" : "Compile SQL and execute against the current target database."
},
"clean" : {
"args" : "clean",
"description" : "Delete all folders in the clean-targets list (usually the dbt_modules and target directories.)"
},
"snapshot" : {
"args" : "snapshot",
"description" : "Execute snapshots defined in your project."
}
},
"matatikaHidden" : false,
"requires" : [ {
"id" : "e6c1ad3d-ebf5-4c4a-b129-f68156b47555",
"pluginType" : "FILE",
"name" : "files-dbt",
"namespace" : "dbt",
"variant" : "matatika",
"description" : " Files dbt is a file bundle that automatically configures your project to run transforms with dbt.\nThe bundle includes template project configuration:\n\n- transform/models (directory)\n- transform/profile/profiles.yml\n- transform/dbt_project.yml\n- transform/.gitignore\n- transform/macros/centralize_test_failures.sql\n",
"hidden" : false,
"pipUrl" : "git+https://github.com/Matatika/[email protected]",
"repo" : "https://github.com/Matatika/files-dbt",
"capabilities" : [ ],
"select" : [ ],
"update" : {
"transform/profile/profiles.yml" : "true"
},
"vars" : { },
"settings" : [ ],
"variants" : [ ],
"commands" : { },
"matatikaHidden" : false,
"requires" : [ ],
"fullDescription" : " Files dbt is a file bundle that automatically configures your project to run transforms with dbt.\nThe bundle includes template project configuration:\n\n- transform/models (directory)\n- transform/profile/profiles.yml\n- transform/dbt_project.yml\n- transform/.gitignore\n- transform/macros/centralize_test_failures.sql\n"
} ],
"fullDescription" : " Power your project transformations with dbt™, a SQL-first transformation tool that enables analytics engineers to develop transformations with code.\n\n***Version Control and CI/CD***\n\nUse Matatika to deploy and promote changes between dev, UAT, and production environments.\n\n***Test and Document***\n\nUse Matatika to develop and test every model prior to production release, and share dynamically generated documentation with all stakeholders.\n\n***Develop***\n\nWrite modular data transformations in .sql – Matatika together with dbt handles the chore of dependency management. \n\n## Settings\n\n\n### target_path\n\nOutput path for dbt generated artifacts",
"_links" : {
"self" : {
"href" : "https://app.matatika.com/api/dataplugins/c3dce0d8-4025-4484-9018-964d5ae3a740"
},
"update dataplugin" : {
"href" : "https://app.matatika.com/api/workspaces/52652da2-a249-407e-a8c6-4e045838a591/dataplugins/c3dce0d8-4025-4484-9018-964d5ae3a740",
"type" : "PUT"
}
}
}
},
"managed" : true,
"draft" : false,
"_links" : {
"self" : {
"href" : "https://app.matatika.com/api/datacomponents/a0195563-9e63-480a-93e9-731670490a91"
},
"update datacomponent" : {
"href" : "https://app.matatika.com/api/datacomponents/a0195563-9e63-480a-93e9-731670490a91"
},
"delete datacomponent" : {
"href" : "https://app.matatika.com/api/datacomponents/a0195563-9e63-480a-93e9-731670490a91"
}
}
} ]
},
"_links" : {
"update pipeline" : {
"href" : "https://app.matatika.com/api/workspaces/52652da2-a249-407e-a8c6-4e045838a591/pipelines/180a0df1-e833-4aae-b6ef-ee98f0f07b50",
"type" : "PUT"
},
"delete pipeline" : {
"href" : "https://app.matatika.com/api/pipelines/180a0df1-e833-4aae-b6ef-ee98f0f07b50",
"type" : "DELETE"
},
"draft pipeline" : {
"href" : "https://app.matatika.com/api/workspaces/52652da2-a249-407e-a8c6-4e045838a591/pipelines/180a0df1-e833-4aae-b6ef-ee98f0f07b50/draft",
"type" : "PUT"
},
"self" : {
"href" : "https://app.matatika.com/api/pipelines/180a0df1-e833-4aae-b6ef-ee98f0f07b50"
},
"environment" : {
"href" : "https://app.matatika.com/api/pipelines/180a0df1-e833-4aae-b6ef-ee98f0f07b50/environment"
},
"jobs" : {
"href" : "https://app.matatika.com/api/pipelines/180a0df1-e833-4aae-b6ef-ee98f0f07b50/jobs",
"type" : "GET"
},
"metrics" : {
"href" : "https://app.matatika.com/api/pipelines/180a0df1-e833-4aae-b6ef-ee98f0f07b50/metrics"
},
"add subscription" : {
"href" : "https://app.matatika.com/api/pipelines/180a0df1-e833-4aae-b6ef-ee98f0f07b50/subscriptions"
},
"verify pipeline" : {
"href" : "https://app.matatika.com/api/pipelines/180a0df1-e833-4aae-b6ef-ee98f0f07b50/verification",
"type" : "POST"
}
}
}
Validate a pipeline configuration in a workspace
POST
/api/workspaces/{workspace-id}/pipelines/validation
Validates a pipeline configuration in the workspace {workspace-id}.
Prerequisites
- Workspace
{workspace-id}must exist
Request
Body
Pipeline resource.
{
"name" : "SIT-generated pipeline [2025-06-19T11:19:48.350863] (updated)",
"dataComponents" : [ "tap-google-analytics", "Warehouse", "dbt" ]
}
Example Snippets
cURL
curl -H "Authorization: Bearer $ACCESS_TOKEN" 'https://app.matatika.com:443/api/workspaces/52652da2-a249-407e-a8c6-4e045838a591/pipelines/validation' -i -X POST \
-H 'Accept: application/json, application/javascript, text/javascript, text/json' \
-H 'Content-Type: application/json' \
-d '{
"name" : "SIT-generated pipeline [2025-06-19T11:19:48.350863] (updated)",
"dataComponents" : [ "tap-google-analytics", "Warehouse", "dbt" ]
}'
Python (requests)
import requests
url = "https://app.matatika.com:443/api/workspaces/52652da2-a249-407e-a8c6-4e045838a591/pipelines/validation"
data = {
"name" : "SIT-generated pipeline [2025-06-19T11:19:48.350863] (updated)",
"dataComponents" : [ "tap-google-analytics", "Warehouse", "dbt" ]
}
headers = {
'Authorization': ACCESS_TOKEN
}
response = requests.request("POST", url, headers=headers, data=data)
print(response.text.encode('utf8'))
Response
200 OK
No response body provided.
400 Bad Request
Pipeline property validation errors.
{
"timestamp" : "2025-06-19T10:19:50.290016003",
"status" : 400,
"error" : "Bad Request",
"message" : "3 validation errors from 'resource'",
"errors" : [ {
"codes" : [ "NotBlank.oauth_credentials.access_token", "NotBlank" ],
"defaultMessage" : "No value given for setting",
"objectName" : "resource",
"field" : "properties.tap-google-analytics.oauth_credentials.access_token",
"bindingFailure" : true,
"code" : "NotBlank"
}, {
"codes" : [ "NotBlank.oauth_credentials.refresh_token", "NotBlank" ],
"defaultMessage" : "No value given for setting",
"objectName" : "resource",
"field" : "properties.tap-google-analytics.oauth_credentials.refresh_token",
"bindingFailure" : true,
"code" : "NotBlank"
}, {
"codes" : [ "NotBlank.view_id", "NotBlank" ],
"defaultMessage" : "No value given for setting",
"objectName" : "resource",
"field" : "properties.tap-google-analytics.view_id",
"bindingFailure" : true,
"code" : "NotBlank"
} ],
"path" : "/api/workspaces/52652da2-a249-407e-a8c6-4e045838a591/pipelines/validation"
}
Verify a pipeline
POST
/api/pipelines/{pipeline-id}/verification
Verifies the configuration of the pipeline {pipeline-id}.
Prerequisites
- Pipeline
{pipeline-id}must exist
Request
Example Snippets
cURL
curl -H "Authorization: Bearer $ACCESS_TOKEN" 'https://app.matatika.com:443/api/pipelines/115c2d52-1515-4e95-8148-69eb6052fcf1/verification' -i -X POST \
-H 'Accept: application/json, application/javascript, text/javascript, text/json' \
-H 'Content-Type: application/json'
Python (requests)
import requests
url = "https://app.matatika.com:443/api/pipelines/115c2d52-1515-4e95-8148-69eb6052fcf1/verification"
headers = {
'Authorization': ACCESS_TOKEN
}
response = requests.request("POST", url, headers=headers)
print(response.text.encode('utf8'))
Response
200 OK
Job with HAL links.
{
"id" : "5910300a-7b07-4689-8f16-6b0996507a0e",
"created" : "2025-06-19T10:10:42.590399",
"type" : "PIPELINE_VERIFY",
"maxAttempts" : 0,
"attempt" : 0,
"status" : "QUEUED",
"_embedded" : {
"pipeline" : {
"id" : "115c2d52-1515-4e95-8148-69eb6052fcf1",
"status" : "READY",
"name" : "SIT-Generated Pipeline [2025-06-19T11:09:38.151404]",
"timeout" : 0,
"maxRetries" : 0,
"created" : "2025-06-19T10:09:38.331496",
"lastModified" : "2025-06-19T10:09:38.331497",
"properties" : { },
"dataComponents" : [ "tap-matatika-sit", "Warehouse", "dbt" ],
"actions" : [ ],
"triggeredBy" : [ ],
"repositoryPath" : "pipelines/SIT-Generated Pipeline [2025-06-19T11:09:38.151404].yml"
},
"profile" : {
"id" : "auth0|5eb0327cbfd7490bff55feeb",
"name" : "[email protected]",
"handle" : "@sit+prod",
"email" : "[email protected]"
}
},
"_links" : {
"self" : {
"href" : "https://app.matatika.com/api/jobs/5910300a-7b07-4689-8f16-6b0996507a0e"
},
"delete job" : {
"href" : "https://app.matatika.com/api/jobs/5910300a-7b07-4689-8f16-6b0996507a0e",
"type" : "DELETE"
},
"logs" : {
"href" : "https://app.matatika.com/api/jobs/5910300a-7b07-4689-8f16-6b0996507a0e/logs?sequence=0",
"type" : "GET"
},
"withdraw job" : {
"href" : "https://app.matatika.com/api/jobs/5910300a-7b07-4689-8f16-6b0996507a0e/stopped",
"type" : "PUT"
}
}
}
Delete a pipeline
DELETE
/api/pipelines/{pipeline-id}
Deletes the pipeline {pipeline-id}.
Prerequisites
- Pipeline
{pipeline-id}must exist
Request
Example Snippets
cURL
curl -H "Authorization: Bearer $ACCESS_TOKEN" 'https://app.matatika.com:443/api/pipelines/180a0df1-e833-4aae-b6ef-ee98f0f07b50' -i -X DELETE \
-H 'Accept: application/json, application/javascript, text/javascript, text/json'
Python (requests)
import requests
url = "https://app.matatika.com:443/api/pipelines/180a0df1-e833-4aae-b6ef-ee98f0f07b50"
headers = {
'Authorization': ACCESS_TOKEN
}
response = requests.request("DELETE", url, headers=headers)
print(response.text.encode('utf8'))
Response
204 No Content
No response body provided.
View pipeline metrics
GET
/api/pipelines/{pipeline-id}/metrics
Returns the pipeline metrics for each job of {pipeline-id}.
Prerequisites
- Pipeline
{pipeline-id}must exist
Request
Example Snippets
cURL
curl -H "Authorization: Bearer $ACCESS_TOKEN" 'https://app.matatika.com:443/api/pipelines/115c2d52-1515-4e95-8148-69eb6052fcf1/metrics' -i -X GET \
-H 'Accept: application/json, application/javascript, text/javascript, text/json'
Python (requests)
import requests
url = "https://app.matatika.com:443/api/pipelines/115c2d52-1515-4e95-8148-69eb6052fcf1/metrics"
headers = {
'Authorization': ACCESS_TOKEN
}
response = requests.request("GET", url, headers=headers)
print(response.text.encode('utf8'))
Response
200: The dataset data (defaults to JSON format).
[ {
"metrics.job-created" : "2025-06-19 10:10:42",
"metrics.value" : 0.0
}, {
"metrics.job-created" : "2025-06-19 10:11:24",
"metrics.value" : 6.0
} ]
204: No response body, metrics not enabled.