Dataplugins
Dataplugins simply define a source of data from a given repository. Matatika provides a number of pre-configured platform-wide dataplugins out-the-box, as well as the ability to create custom dataplugins through the API. From these, pipeline jobs can be run to inject data into a workspace.
Objects
Dataplugin
| Path | Type | Format | Description |
|---|---|---|---|
id |
String |
Version 4 UUID | The dataplugin ID |
name |
String |
The dataplugin name | |
description |
String |
A description of the dataplugin | |
repositoryUrl |
String |
URL | The dataplugin repository URL |
settings |
Array of Setting |
The dataplugin settings |
{
"id" : "115f4bd2-6897-479b-816d-c2f9d18fb7cd",
"pluginType" : "LOADER",
"name" : "target-postgres",
"namespace" : "postgres_transferwise",
"variant" : "transferwise",
"label" : "Postgres Warehouse",
"description" : "Loader for Postgres Data Warehouse\n\nThe World's Most Advanced Open Source Relational Database\n\n## Settings\n\n### User\nPostgres user\n\n### Password\nPostgres password\n\n### Host\nPostgres host\nExample: `test-host.db.elephantsql.com`\n\n### Port\nPostgres port\nExample: `5432`\n\n### Database Name\nPostgres database name\n\n### Default Target Schema\nName of the schema where tables will be created (no database prefix)",
"logoUrl" : "/assets/logos/loaders/postgres.png",
"hidden" : false,
"pipUrl" : "git+https://github.com/Matatika/[email protected]",
"repo" : "git+https://github.com/Matatika/[email protected]",
"capabilities" : [ ],
"select" : [ ],
"update" : { },
"vars" : { },
"settings" : [ {
"name" : "user",
"aliases" : [ "username" ],
"label" : "User",
"kind" : "STRING",
"description" : "PostgreSQL user",
"required" : "true",
"protected" : false
}, {
"name" : "password",
"aliases" : [ ],
"label" : "Password",
"kind" : "PASSWORD",
"description" : "PostgreSQL password",
"required" : "true",
"protected" : false
}, {
"name" : "host",
"aliases" : [ "address" ],
"label" : "Host",
"kind" : "STRING",
"description" : "PostgreSQL host",
"required" : "true",
"protected" : false
}, {
"name" : "port",
"aliases" : [ ],
"label" : "Port",
"value" : "5432",
"kind" : "INTEGER",
"description" : "PostgreSQL port",
"required" : "true",
"protected" : false
}, {
"name" : "dbname",
"aliases" : [ "database" ],
"label" : "Database Name",
"kind" : "STRING",
"description" : "PostgreSQL database name",
"required" : "true",
"protected" : false
}, {
"name" : "default_target_schema",
"aliases" : [ ],
"label" : "Default Target Schema",
"value" : "analytics",
"kind" : "STRING",
"description" : "Name of the schema where the tables will be created.",
"required" : "true",
"protected" : false
}, {
"name" : "ssl",
"aliases" : [ ],
"label" : "SSL",
"value" : "false",
"kind" : "HIDDEN",
"protected" : false,
"value_post_processor" : "STRINGIFY"
}, {
"name" : "batch_size_rows",
"aliases" : [ ],
"label" : "Batch Size Rows",
"value" : "100000",
"kind" : "INTEGER",
"description" : "Maximum number of rows in each batch. At the end of each batch, the rows in the batch are loaded into Postgres.",
"protected" : false
}, {
"name" : "underscore_camel_case_fields",
"aliases" : [ ],
"label" : "Underscore Camel Case Fields",
"value" : "true",
"kind" : "HIDDEN",
"description" : "Enable underscoring camel case field names, make the resulting postgres column names the same as from the Meltano variant of target-postgres.",
"protected" : false
}, {
"name" : "flush_all_streams",
"aliases" : [ ],
"label" : "Flush All Streams",
"value" : "false",
"kind" : "HIDDEN",
"description" : "Flush and load every stream into Postgres when one batch is full. Warning: This may trigger the COPY command to use files with low number of records.",
"protected" : false
}, {
"name" : "parallelism",
"aliases" : [ ],
"label" : "Parallelism",
"value" : "0",
"kind" : "HIDDEN",
"description" : "The number of threads used to flush tables. 0 will create a thread for each stream, up to parallelism_max. -1 will create a thread for each CPU core. Any other positive number will create that number of threads, up to parallelism_max.",
"protected" : false
}, {
"name" : "parallelism_max",
"aliases" : [ ],
"label" : "Max Parallelism",
"value" : "16",
"kind" : "HIDDEN",
"description" : "Max number of parallel threads to use when flushing tables.",
"protected" : false
}, {
"name" : "default_target_schema_select_permission",
"aliases" : [ ],
"label" : "Default Target Schema Select Permission",
"kind" : "HIDDEN",
"description" : "Grant USAGE privilege on newly created schemas and grant SELECT privilege on newly created tables to a specific role or a list of roles. If `schema_mapping` is not defined then every stream sent by the tap is granted accordingly.",
"protected" : false
}, {
"name" : "schema_mapping",
"aliases" : [ ],
"label" : "Schema Mapping",
"kind" : "HIDDEN",
"description" : "Useful if you want to load multiple streams from one tap to multiple Postgres schemas.\n\nIf the tap sends the `stream_id` in `<schema_name>-<table_name>` format then this option overwrites the `default_target_schema` value.\nNote, that using `schema_mapping` you can overwrite the `default_target_schema_select_permission` value to grant SELECT permissions to different groups per schemas or optionally you can create indices automatically for the replicated tables.\n",
"protected" : false
}, {
"name" : "add_metadata_columns",
"aliases" : [ ],
"label" : "Add Metadata Columns",
"value" : "true",
"kind" : "HIDDEN",
"description" : "Metadata columns add extra row level information about data ingestions, (i.e. when was the row read in source, when was inserted or deleted in postgres etc.) Metadata columns are creating automatically by adding extra columns to the tables with a column prefix `_SDC_`. The column names are following the stitch naming conventions documented at https://www.stitchdata.com/docs/data-structure/integration-schemas#sdc-columns. Enabling metadata columns will flag the deleted rows by setting the `_SDC_DELETED_AT` metadata column. Without the `add_metadata_columns` option the deleted rows from singer taps will not be recongisable in Postgres.",
"protected" : false
}, {
"name" : "hard_delete",
"aliases" : [ ],
"label" : "Hard Delete",
"value" : "false",
"kind" : "HIDDEN",
"description" : "When `hard_delete` option is true then DELETE SQL commands will be performed in Postgres to delete rows in tables. It's achieved by continuously checking the `_SDC_DELETED_AT` metadata column sent by the singer tap. Due to deleting rows requires metadata columns, `hard_delete` option automatically enables the `add_metadata_columns` option as well.",
"protected" : false
}, {
"name" : "data_flattening_max_level",
"aliases" : [ ],
"label" : "Data Flattening Max Level",
"value" : "10",
"kind" : "HIDDEN",
"description" : "Object type RECORD items from taps can be transformed to flattened columns by creating columns automatically. When value is 0 (default) then flattening functionality is turned off.",
"protected" : false
}, {
"name" : "primary_key_required",
"aliases" : [ ],
"label" : "Primary Key Required",
"value" : "false",
"kind" : "BOOLEAN",
"description" : "Log based and Incremental replications on tables with no Primary Key cause duplicates when merging UPDATE events. When set to true, stop loading data if no Primary Key is defined.",
"protected" : false
}, {
"name" : "validate_records",
"aliases" : [ ],
"label" : "Validate Records",
"value" : "false",
"kind" : "BOOLEAN",
"description" : "Validate every single record message to the corresponding JSON schema. This option is disabled by default and invalid RECORD messages will fail only at load time by Postgres. Enabling this option will detect invalid records earlier but could cause performance degradation.",
"protected" : false
}, {
"name" : "temp_dir",
"aliases" : [ ],
"label" : "Temporary Directory",
"kind" : "HIDDEN",
"description" : "(Default: platform-dependent) Directory of temporary CSV files with RECORD messages.",
"protected" : false
} ],
"variants" : [ ],
"commands" : { },
"matatikaHidden" : false,
"_links" : {
"self" : {
"href" : "https://catalog.matatika.com/api/dataplugins/115f4bd2-6897-479b-816d-c2f9d18fb7cd"
},
"update dataplugin" : {
"href" : "https://catalog.matatika.com/api/workspaces/c8bd869f-87e5-4545-a75d-f29b178079ea/dataplugins/115f4bd2-6897-479b-816d-c2f9d18fb7cd",
"type" : "PUT"
}
}
}
Setting
| Path | Type | Format | Description |
|---|---|---|---|
name |
String |
The setting name | |
value |
String |
The setting default value | |
label |
String |
The setting label | |
protected |
Boolean |
The setting protection status | |
kind |
String |
Setting Kind | The setting kind |
description |
String |
A description of the setting | |
placeholder |
String |
The setting placeholder text | |
envAliases |
Array of String |
Environment variable aliases for the setting | |
documentation |
String |
URL | The setting documentation URL |
oauth |
OAuth |
The setting OAuth configuration | |
env |
String |
OAuth
| Path | Type | Format | Description |
|---|---|---|---|
provider |
String |
The OAuth provider |
Formats
Setting Kind
String
| Value | Description |
|---|---|
STRING |
String setting |
INTEGER |
Integer setting |
PASSWORD |
Password setting |
HIDDEN |
Hidden setting |
BOOLEAN |
Boolean setting |
DATE_ISO8601 |
ISO 8601 date setting |
EMAIL |
Email setting |
OAUTH |
OAuth setting |
FILE |
File setting |
ARRAY |
Array setting |
Requests
- View all supported dataplugins
- View the Matatika
discovery.yml - View all workspace dataplugins
- View a workspace
discovery.yml - View a dataplugin
- Initialise a new dataplugin
- Publish dataplugins from a
discovery.yml - Create a dataplugin
- Update a dataplugin
- Delete a dataplugin
View all supported dataplugins
GET
/api/dataplugins
Returns all dataplugins supported by Matatika.
Request
Example Snippets
- cURL
curl -H "Authorization: Bearer $ACCESS_TOKEN" 'https://catalog.matatika.com:443/api/dataplugins' -i -X GET \
-H 'Accept: application/json, application/javascript, text/javascript, text/json' \
-H 'Content-Type: application/json'
- Python (requests)
import requests
url = "https://catalog.matatika.com:443/api/dataplugins"
headers = {
'Authorization': ACCESS_TOKEN
}
response = requests.request("GET", url, headers=headers)
print(response.text.encode('utf8'))
Response
200 OK
Dataplugin collection with HAL links.
{
"_embedded" : {
"dataplugins" : [ {
"id" : "33444aa0-a5e9-4edb-927a-d0c15707baa0",
"pluginType" : "EXTRACTOR",
"name" : "tap-solarvista",
"namespace" : "tap_solarvista",
"variant" : "matatika",
"label" : "Solarvista Live",
"description" : "Solarvista Live is a Field Service Management platform to manage, plan, and mobilise your engineers.\n\n## Prerequisites\n\nBefore importing data from Solarvista Live you will need to have performed the initial setup in Solarvista.\n\n[Solarvista Live Prerequisites](https://www.matatika.com/docs/data-sources/tap-solarvista/prerequisites)\n\n\n## Data source settings\n\n### Account\n\nSolarvista API `account`\n\nTo get your `account` id you will need to follow the steps that are linked in our [Prerequisites](https://www.matatika.com/docs/data-sources/tap-solarvista/prerequisites) section.\n\n### ClientId\n\nSolarvista API `clientId`\n\nTo get your `clientId` you will need to follow the steps that are linked in our [Prerequisites](https://www.matatika.com/docs/data-sources/tap-solarvista/prerequisites) section.\n\n### Code\n\nSolarvista API `code`\n\nTo get your `code` you will need to follow the steps that are linked in our [Prerequisites](https://www.matatika.com/docs/data-sources/tap-solarvista/prerequisites) section.\n\n### Datasources\n\nLeave this field blank to accept the default Solarvista datasources or customise the type of data to sync, we support:\n- `customer`\n- `site`\n- `equipment`\n- `work-item`\n- `work-item-history`\n- `activity`\n- `users`\n- `territory`\n- `project`\n- `appointment`\n- `skill`\n\n\n### Start Date\n\nProviding a start date allows you to set how much historical data you wish to import into your workspace. \nNote: this setting will only work on your first pipeline run, as from then on we will use your pipeline state to sync only new data for you.\n\nThis date has to be in the ISO-8601 date format, see the example below.\n\nFormat: `YYYY-MM-DDTHH:MM:SSZ`\nExample: `2020-10-01T13:12:11Z`\n\n### Force Start Date\n\nProviding a force start date will make your pipeline sync all data from a specific date, ignoring any previous pipeline state.\n\nThis date has to be in the ISO-8601 date format, see the example below.\n\nFormat: `YYYY-MM-DDTHH:MM:SSZ`\nExample: `2020-10-01T13:12:11Z`\n\n\n## Learn more\n\n[Matatika Solavista Live Data Source Documentation](https://www.matatika.com/docs/data-sources/tap-solarvista)\n\n[Meltano Hub Solarvista Live Documentation](https://hub.meltano.com/taps/solarvista)",
"logoUrl" : "/assets/images/datasource/tap-solarvista.png",
"hidden" : false,
"docs" : "https://www.matatika.com/docs/data-sources/tap-solarvista/",
"pipUrl" : "git+https://github.com/Matatika/[email protected]",
"repo" : "https://github.com/Matatika/tap-solarvista",
"capabilities" : [ "state" ],
"select" : [ ],
"update" : { },
"vars" : { },
"settings" : [ {
"name" : "datasources",
"aliases" : [ ],
"label" : "Datasources",
"kind" : "STRING",
"required" : "true",
"protected" : false
}, {
"name" : "account",
"aliases" : [ ],
"label" : "Account",
"kind" : "STRING",
"required" : "true",
"protected" : false
}, {
"name" : "clientId",
"aliases" : [ ],
"label" : "Client ID",
"kind" : "STRING",
"required" : "true",
"protected" : false
}, {
"name" : "code",
"aliases" : [ ],
"label" : "Code",
"kind" : "PASSWORD",
"required" : "true",
"protected" : false
}, {
"name" : "start_date",
"aliases" : [ ],
"label" : "Start Date",
"kind" : "DATE_ISO8601",
"protected" : false
}, {
"name" : "force_start_date",
"aliases" : [ ],
"label" : "Force Start Date",
"kind" : "DATE_ISO8601",
"protected" : false
} ],
"variants" : [ ],
"commands" : { },
"matatikaHidden" : false,
"_links" : {
"self" : {
"href" : "https://catalog.matatika.com/api/dataplugins/33444aa0-a5e9-4edb-927a-d0c15707baa0"
}
}
}, {
"id" : "1149bda6-c93f-4db6-a22c-f95afd60d575",
"pluginType" : "FILE",
"name" : "analyze-sit",
"namespace" : "tap_matatika_sit",
"variant" : "matatika",
"hidden" : false,
"pipUrl" : "git+https://github.com/Matatika/analyze-sit.git",
"repo" : "https://github.com/Matatika/analyze-sit",
"capabilities" : [ ],
"select" : [ ],
"update" : {
"analyze/datasets/tap-matatika-sit/user-ages.yml" : "true",
"analyze/datasets/tap-matatika-sit/user-genders.yml" : "true"
},
"vars" : { },
"settings" : [ ],
"variants" : [ ],
"commands" : { },
"matatikaHidden" : false,
"_links" : {
"self" : {
"href" : "https://catalog.matatika.com/api/dataplugins/1149bda6-c93f-4db6-a22c-f95afd60d575"
}
}
}, {
"id" : "6c5a07d0-8580-4bf3-a56e-fb87f7c24c09",
"pluginType" : "EXTRACTOR",
"name" : "tap-auth0",
"namespace" : "tap_auth0",
"variant" : "matatika",
"label" : "Auth0",
"description" : "Auth0 is an identity and access management service, that allows you to manage the access of your users.\n\n## Prerequisites\n\nTo find out how to get the required credentials for this data source please go to our documentation: [Auth0 Prerequisites](https://www.matatika.com/docs/data-sources/tap-auth0/prerequisites)\n\n## Data source settings\n\n### Client ID\n\nYour Auth0 Machine-to-Machine application client id.\n\n### Client Secret\n\nYour Auth0 Machine-to-Machine application client secret.\n\n### Domain\n\nYour Auth0 tenant domain in the format `<TENANT_NAME>.<REGION_IDENTIFIER>.auth0.com``\n\n## Learn more\n\n[Matatika Auth0 Data Source Documentation](https://www.matatika.com/docs/data-sources/tap-auth0)\n\n[Meltano Hub Auth0 Documentation](https://hub.meltano.com/taps/auth0)",
"logoUrl" : "/assets/images/datasource/tap-auth0.png",
"hidden" : false,
"docs" : "https://www.matatika.com/docs/data-sources/tap-auth0/",
"pipUrl" : "git+https://github.com/Matatika/[email protected]",
"repo" : "https://github.com/Matatika/tap-auth0",
"capabilities" : [ "catalog", "state", "discover" ],
"select" : [ ],
"update" : { },
"vars" : { },
"settings" : [ {
"name" : "client_id",
"aliases" : [ ],
"label" : "Client ID",
"kind" : "PASSWORD",
"required" : "true",
"protected" : false
}, {
"name" : "client_secret",
"aliases" : [ ],
"label" : "Client Secret",
"kind" : "PASSWORD",
"required" : "true",
"protected" : false
}, {
"name" : "domain",
"aliases" : [ ],
"label" : "Domain",
"kind" : "STRING",
"required" : "true",
"protected" : false
}, {
"name" : "job_poll_interval_ms",
"aliases" : [ ],
"value" : "2000",
"kind" : "HIDDEN",
"protected" : false
}, {
"name" : "job_poll_max_count",
"aliases" : [ ],
"value" : "10",
"kind" : "HIDDEN",
"protected" : false
} ],
"variants" : [ ],
"commands" : { },
"matatikaHidden" : false,
"_links" : {
"self" : {
"href" : "https://catalog.matatika.com/api/dataplugins/6c5a07d0-8580-4bf3-a56e-fb87f7c24c09"
}
}
}, {
"id" : "6472b907-3f72-4456-9ce3-dd97236ba84f",
"pluginType" : "FILE",
"name" : "analyze-google-analytics",
"namespace" : "tap_google_analytics",
"variant" : "matatika",
"label" : "Google Analytics Insights",
"description" : "Instant insights on users, locations, sources, and sessions from Google Analytics.",
"hidden" : false,
"pipUrl" : "git+https://github.com/Matatika/[email protected]",
"repo" : "https://github.com/Matatika/analyze-google-analytics",
"capabilities" : [ ],
"select" : [ ],
"update" : {
"*.yml" : "true"
},
"vars" : { },
"settings" : [ ],
"variants" : [ ],
"commands" : { },
"matatikaHidden" : false,
"_links" : {
"self" : {
"href" : "https://catalog.matatika.com/api/dataplugins/6472b907-3f72-4456-9ce3-dd97236ba84f"
}
}
}, {
"id" : "4f3acdb4-898b-4ddf-a70f-1141f7b73129",
"pluginType" : "TRANSFORM",
"name" : "dbt-solarvista",
"namespace" : "tap_solarvista",
"variant" : "matatika",
"hidden" : false,
"pipUrl" : "https://github.com/Matatika/[email protected]",
"repo" : "https://github.com/Matatika/dbt-tap-solarvista",
"capabilities" : [ ],
"select" : [ ],
"update" : { },
"vars" : {
"schema" : ""
},
"settings" : [ ],
"variants" : [ ],
"commands" : { },
"matatikaHidden" : false,
"_links" : {
"self" : {
"href" : "https://catalog.matatika.com/api/dataplugins/4f3acdb4-898b-4ddf-a70f-1141f7b73129"
}
}
}, {
"id" : "14518e68-ecda-48c9-9c93-155453d89ef2",
"pluginType" : "FILE",
"name" : "analyze-auth0",
"namespace" : "tap_auth0",
"variant" : "matatika",
"label" : "Auth0 Insights",
"description" : "Instant insights on users, logins and quotas from Auth0.",
"hidden" : false,
"pipUrl" : "git+https://github.com/Matatika/[email protected]",
"repo" : "https://github.com/Matatika/analyze-auth0",
"capabilities" : [ ],
"select" : [ ],
"update" : {
"*.yml" : "true"
},
"vars" : { },
"settings" : [ ],
"variants" : [ ],
"commands" : { },
"matatikaHidden" : false,
"_links" : {
"self" : {
"href" : "https://catalog.matatika.com/api/dataplugins/14518e68-ecda-48c9-9c93-155453d89ef2"
}
}
}, {
"id" : "dbf87b80-6eb6-483a-90bb-b7a8c094fb3a",
"pluginType" : "FILE",
"name" : "analyze-solarvista",
"namespace" : "tap_solarvista",
"variant" : "matatika",
"label" : "Solarvista Insights",
"description" : "Instant insights on revenue, projects, work items, and engineer performance from Solarvista Live.",
"hidden" : false,
"pipUrl" : "git+https://github.com/Matatika/[email protected]",
"repo" : "https://github.com/Matatika/analyze-solarvista",
"capabilities" : [ ],
"select" : [ ],
"update" : {
"*.yml" : "true"
},
"vars" : { },
"settings" : [ ],
"variants" : [ ],
"commands" : { },
"matatikaHidden" : false,
"_links" : {
"self" : {
"href" : "https://catalog.matatika.com/api/dataplugins/dbf87b80-6eb6-483a-90bb-b7a8c094fb3a"
}
}
}, {
"id" : "ffd26c88-aa25-4e04-913c-8dd0b22762d1",
"pluginType" : "FILE",
"name" : "analyze-trello",
"namespace" : "tap_trello",
"variant" : "matatika",
"label" : "Trello Insights",
"description" : "Instant insights on members, cards, boards, and actions from Trello.",
"hidden" : false,
"pipUrl" : "git+https://github.com/Matatika/[email protected]",
"repo" : "https://github.com/Matatika/analyze-trello",
"capabilities" : [ ],
"select" : [ ],
"update" : {
"*.yml" : "true"
},
"vars" : { },
"settings" : [ ],
"variants" : [ ],
"commands" : { },
"matatikaHidden" : false,
"_links" : {
"self" : {
"href" : "https://catalog.matatika.com/api/dataplugins/ffd26c88-aa25-4e04-913c-8dd0b22762d1"
}
}
}, {
"id" : "931124c6-882f-4f0d-b0ca-6db09f1e1948",
"pluginType" : "EXTRACTOR",
"name" : "tap-matatika-sit",
"namespace" : "tap_matatika_sit",
"variant" : "matatika",
"label" : "Matatika SIT",
"description" : "Test extractor based on tap-spreadsheets-anywhere used during Matatika SIT runs",
"logoUrl" : "/assets/images/datasource/tap-matatika-sit.svg",
"hidden" : false,
"docs" : "https://meltano.com/plugins/extractors/spreadsheets-anywhere.html",
"pipUrl" : "git+https://github.com/ets/tap-spreadsheets-anywhere.git",
"repo" : "https://github.com/ets/tap-spreadsheets-anywhere",
"executable" : "tap-spreadsheets-anywhere",
"capabilities" : [ "catalog", "state", "discover" ],
"select" : [ ],
"update" : { },
"vars" : { },
"settings" : [ {
"name" : "tables",
"aliases" : [ ],
"value" : "[{\"path\":\"https://raw.githubusercontent.com/Matatika/matatika-examples/master/example_data\",\"name\":\"gitflixusers\",\"pattern\":\"GitFlixUsers.csv\",\"start_date\":\"2021-01-01T00:00:00Z\",\"key_properties\":[\"id\"],\"format\":\"csv\"}]",
"kind" : "ARRAY",
"description" : "An array holding json objects that each describe a set of targeted source files. See docs for details.",
"protected" : false
} ],
"variants" : [ ],
"commands" : { },
"matatikaHidden" : false,
"_links" : {
"self" : {
"href" : "https://catalog.matatika.com/api/dataplugins/931124c6-882f-4f0d-b0ca-6db09f1e1948"
}
}
}, {
"id" : "58c0c3c2-94a0-46d9-bfda-3b8905ddb3ab",
"pluginType" : "EXTRACTOR",
"name" : "tap-shopify",
"namespace" : "tap_shopify",
"variant" : "matatika",
"label" : "Shopify",
"description" : "E-commerce platform for online stores and retail point-of-sale systems.\n\n## Prerequisites\n\nBefore importing data from Shopify, you or your store admin may need to perform some\ninitial setup to grant read access.\n\n[Prerequisites](https://www.matatika.com/docs/data-sources/tap-shopify/prerequisites)\n\n## Settings\n\n### Access Token\n\nThe access token to authenticate with the Shopify API. See Prerequisites.\n\n\n### Store\n\nShopify store id, use the prefix of your admin url.\n\ne.g. https://<your store id>.myshopify.com/admin\n\n\n### Start Date\n\nProvide a start date to control the amount of historical data.\n\nFormat: `YYYY-MM-DDTHH:MM:SSZ`\nExample: `2020-10-01T13:12:11Z`\n\n\n## Learn more\n\nLearn more about the Shopify data source\n\n[Matatika Shopify Data Source Documentation](https://www.matatika.com/docs/data-sources/tap-shopify)",
"logoUrl" : "/assets/images/datasource/tap-shopify.png",
"hidden" : false,
"docs" : "https://www.matatika.com/docs/data-sources/tap-shopify/",
"pipUrl" : "git+https://github.com/Matatika/[email protected]",
"repo" : "https://github.com/Matatika/tap-shopify",
"capabilities" : [ "catalog", "state", "discover" ],
"select" : [ ],
"update" : { },
"vars" : { },
"settings" : [ {
"name" : "access_token",
"aliases" : [ ],
"label" : "Access Token",
"kind" : "PASSWORD",
"required" : "true",
"protected" : false
}, {
"name" : "store",
"aliases" : [ ],
"label" : "Store",
"kind" : "STRING",
"required" : "true",
"protected" : false
}, {
"name" : "start_date",
"aliases" : [ ],
"label" : "Start Date",
"kind" : "DATE_ISO8601",
"protected" : false
} ],
"variants" : [ ],
"commands" : { },
"matatikaHidden" : false,
"_links" : {
"self" : {
"href" : "https://catalog.matatika.com/api/dataplugins/58c0c3c2-94a0-46d9-bfda-3b8905ddb3ab"
}
}
}, {
"id" : "c439b543-9822-4294-b183-d20a13490fb2",
"pluginType" : "EXTRACTOR",
"name" : "tap-meltano",
"namespace" : "tap_meltano",
"variant" : "matatika",
"label" : "Meltano",
"description" : "Meltano is a data pipeline and modular open source data ops platform.\n\nWhen you run a Meltano job and you have a `MELTANO_DATABASE_URI` set, data about that job run is stored in that database.\n\nBy using this data import, and providing an existing `MELTANO_DATABASE_URI`, you will get insights on Meltano job runs.\n\n## Prerequisites\n\n[Meltano Prerequisites](https://www.matatika.com/docs/data-sources/tap-meltano/prerequisites)\n\n## Data source settings\n\n### Meltano Database URI\n\nYour target Meltano project's database URI. By default, (if left blank), this value will be set to your workspace's Meltano database URI.\n\nExample database URI: `postgresql://<username>:<password>@<host>:<port>/<database>?options=-csearch_path%3D<schema>`\n\n## Learn more\n\n[Matatika Meltano Data Source Documentation](https://www.matatika.com/docs/data-sources/tap-meltano)\n\n[Meltano Hub Meltano Documentation](https://hub.meltano.com/taps/meltano)",
"logoUrl" : "/assets/images/datasource/tap-meltano.png",
"hidden" : false,
"docs" : "https://www.matatika.com/docs/data-sources/tap-meltano/",
"pipUrl" : "git+https://github.com/Matatika/[email protected]",
"repo" : "https://github.com/Matatika/tap-meltano",
"capabilities" : [ "catalog", "state", "discover" ],
"select" : [ ],
"update" : { },
"vars" : { },
"settings" : [ {
"name" : "meltano_database_uri",
"aliases" : [ ],
"value" : "$MELTANO_DATABASE_URI",
"kind" : "PASSWORD",
"required" : "true",
"protected" : false
} ],
"variants" : [ ],
"commands" : { },
"matatikaHidden" : false,
"_links" : {
"self" : {
"href" : "https://catalog.matatika.com/api/dataplugins/c439b543-9822-4294-b183-d20a13490fb2"
}
}
}, {
"id" : "51bcb7cb-13ab-4847-bda4-4db40bacf553",
"pluginType" : "EXTRACTOR",
"name" : "tap-google-analytics",
"namespace" : "tap_google_analytics",
"variant" : "matatika",
"label" : "Google Analytics",
"description" : "App and website analytics platform hosted by Google\n\n## Prerequisites\n\nBefore importing data from Google Analytics you or your webmaster will need to have performed some initial setup to grant\nyourself access to the Google Analytics for your website.\n\n[Google Analytics Prerequisites](https://www.matatika.com/docs/data-sources/tap-google-analytics/prerequisites)\n\n## Data source settings\n### View ID\n\nOnce you have connected with your google account, the `View ID` drop down will allow you to choose which view you want to get your data from.\n\n### Start Date\n\nProviding a start date allows you to set how much historical data you wish to import into you workspace.\n\nThis date has to be in the ISO-8601 date format, see the example below.\n\nFormat: `YYYY-MM-DDTHH:MM:SSZ`\n\nExample: `2020-10-01T13:12:11Z`\n\n### End Date\n\nProviding an end date allows you to limit how much data you wish to import into you workspace.\n\nThis date has to be in the ISO-8601 date format, see the example below.\n\nFormat: `YYYY-MM-DDTHH:MM:SSZ`\n\nExample: `2020-10-01T13:12:11Z`\n\n### Reports\n\nProviding the path to a reports json file allows you to supply the tap with a user defined definition of tables and information you want to sync from Google Analytics.\nThis does require you to commit a file to your workspace repository so that Google Analytics can find it during run time, and may also break the dbt transforms that are required to have our included datasets work.\nFor more information on how to define a custom reports file see the [Meltano Hub Documentation](https://hub.meltano.com/extractors/google-analytics#reports).\n\n## Learn more\n\n[Matatika Google Analytics Data Source Documentation](https://www.matatika.com/docs/data-sources/tap-google-analytics)\n\n[Meltano Hub Google Analytics Documentation](https://hub.meltano.com/extractors/google-analytics)",
"logoUrl" : "/assets/images/datasource/tap-google-analytics.svg",
"hidden" : false,
"docs" : "https://www.matatika.com/docs/data-sources/tap-google-analytics/",
"pipUrl" : "git+https://github.com/Matatika/[email protected]",
"repo" : "https://github.com/Matatika/tap-google-analytics",
"capabilities" : [ "catalog", "state", "discover" ],
"select" : [ ],
"update" : { },
"vars" : { },
"settings" : [ {
"name" : "oauth_credentials.authorization_url",
"aliases" : [ ],
"label" : "OAuth identity provider authorization endpoint used create and refresh tokens",
"value" : "https://oauth2.googleapis.com/token",
"kind" : "HIDDEN",
"required" : "true",
"protected" : false
}, {
"name" : "oauth_credentials.scope",
"aliases" : [ ],
"label" : "OAuth scopes we need to request access to",
"value" : "profile email https://www.googleapis.com/auth/analytics.readonly",
"kind" : "HIDDEN",
"required" : "true",
"protected" : false
}, {
"name" : "oauth_credentials.access_token",
"aliases" : [ ],
"label" : "Access Token",
"kind" : "HIDDEN",
"required" : "true",
"protected" : false
}, {
"name" : "oauth_credentials.refresh_token",
"aliases" : [ ],
"label" : "OAuth Refresh Token",
"kind" : "HIDDEN",
"required" : "true",
"protected" : false
}, {
"name" : "oauth_credentials.refresh_proxy_url",
"aliases" : [ ],
"label" : "Optional - will be called with 'oauth_credentials.refresh_token' to refresh the access token",
"kind" : "HIDDEN",
"protected" : false
}, {
"name" : "oauth_credentials.refresh_proxy_url_auth",
"aliases" : [ ],
"label" : "Optional - Sets Authorization header on 'oauth_credentials.refresh_url' request",
"kind" : "HIDDEN",
"protected" : false
}, {
"name" : "oauth_credentials.client_id",
"aliases" : [ ],
"label" : "Optional - OAuth Client ID used if refresh_proxy_url not supplied",
"kind" : "HIDDEN",
"protected" : false
}, {
"name" : "oauth_credentials.client_secret",
"aliases" : [ ],
"label" : "Optional - OAuth Client Secret used if refresh_proxy_url not supplied",
"kind" : "HIDDEN",
"protected" : false
}, {
"name" : "view_id",
"aliases" : [ ],
"label" : "View ID",
"placeholder" : "Ex. 198343027",
"kind" : "STRING",
"required" : "true",
"protected" : false
}, {
"name" : "reports",
"aliases" : [ ],
"label" : "Reports",
"placeholder" : "Ex. my_report_definition.json",
"kind" : "STRING",
"protected" : false
}, {
"name" : "start_date",
"aliases" : [ ],
"kind" : "DATE_ISO8601",
"protected" : false
}, {
"name" : "end_date",
"aliases" : [ ],
"kind" : "DATE_ISO8601",
"protected" : false
} ],
"variants" : [ ],
"commands" : { },
"matatikaHidden" : false,
"_links" : {
"self" : {
"href" : "https://catalog.matatika.com/api/dataplugins/51bcb7cb-13ab-4847-bda4-4db40bacf553"
}
}
}, {
"id" : "2798a08b-dec4-4f73-bac6-cfaf567302e5",
"pluginType" : "EXTRACTOR",
"name" : "tap-spotify",
"namespace" : "tap_spotify",
"variant" : "matatika",
"label" : "Spotify",
"description" : "Spotify is a digital music streaming service.\n\nBy using this data import you will see insights into your music trends.\n\n## Prerequisites\n\nTo get you Spotify credentials you can follow the steps listed in the [Spotify Prerequisites](https://www.matatika.com/docs/data-sources/tap-spotify/prerequisites).\n\n## Data source settings\n\n### Client ID\n\nYour Spotify Client ID.\n\n### Client Secret\n\nYour Spotify Client Secret.\n\n### Refresh Token\n\nYour Spotify Client Refresh Token.\n\n## Learn more\n\n[Matatika Spotify Data Source Documentation](https://www.matatika.com/docs/data-sources/tap-spotify)\n\n[Meltano Hub Spotify Documentation](https://hub.meltano.com/taps/spotify)",
"logoUrl" : "/assets/images/datasource/tap-spotify.png",
"hidden" : false,
"docs" : "https://www.matatika.com/docs/data-sources/tap-spotify/",
"pipUrl" : "git+https://github.com/Matatika/[email protected]",
"repo" : "https://github.com/Matatika/tap-spotify",
"capabilities" : [ "catalog", "state", "discover" ],
"select" : [ ],
"update" : { },
"vars" : { },
"settings" : [ {
"name" : "client_id",
"aliases" : [ ],
"kind" : "PASSWORD",
"required" : "true",
"protected" : false
}, {
"name" : "client_secret",
"aliases" : [ ],
"kind" : "PASSWORD",
"required" : "true",
"protected" : false
}, {
"name" : "refresh_token",
"aliases" : [ ],
"kind" : "PASSWORD",
"required" : "true",
"protected" : false
} ],
"variants" : [ ],
"commands" : { },
"matatikaHidden" : false,
"_links" : {
"self" : {
"href" : "https://catalog.matatika.com/api/dataplugins/2798a08b-dec4-4f73-bac6-cfaf567302e5"
}
}
}, {
"id" : "0879ca90-e5ba-49b9-8435-c68676133ac7",
"pluginType" : "FILE",
"name" : "analyze-meltano",
"namespace" : "tap_meltano",
"variant" : "matatika",
"label" : "Meltano Insights",
"description" : "Instant insights on jobs from Meltano.",
"hidden" : false,
"pipUrl" : "git+https://github.com/Matatika/[email protected]",
"repo" : "https://github.com/Matatika/analyze-meltano",
"capabilities" : [ ],
"select" : [ ],
"update" : {
"*.yml" : "true"
},
"vars" : { },
"settings" : [ ],
"variants" : [ ],
"commands" : { },
"matatikaHidden" : false,
"_links" : {
"self" : {
"href" : "https://catalog.matatika.com/api/dataplugins/0879ca90-e5ba-49b9-8435-c68676133ac7"
}
}
}, {
"id" : "5a4a153f-0ec0-4d55-8b04-2e04e008dca1",
"pluginType" : "EXTRACTOR",
"name" : "tap-google-sheets",
"namespace" : "tap_google_sheets",
"variant" : "matatika",
"label" : "Google Sheets",
"description" : "Google Sheets is a web-based application that enables users to create, update and modify spreadsheets and share the data online in real time.\n\n## Data source settings\n### Sheet ID - `Required`\n\nOnce you have connected with your google account, the `Sheet ID` drop down will allow you to choose which sheet you want to get your data from.\n\n### Stream Name - `Optional`\n\nSet the stream name to control the table or file name output by the target. By default this will be the file name of your Google Sheet.\n\n### Child Sheet Name - `Optional`\n\nPick a different sheet within your Google Sheet to sync data from. By default we sync data from the first visible sheet in your Google Sheet.\n\n## Learn more\n\n[Matatika Google Sheets Data Source Documentation](https://www.matatika.com/docs/data-sources/tap-google-sheets)",
"logoUrl" : "/assets/images/datasource/tap-google-sheets.png",
"hidden" : false,
"docs" : "https://www.matatika.com/docs/data-sources/tap-google-sheets/",
"pipUrl" : "git+https://github.com/Matatika/[email protected]",
"repo" : "https://github.com/Matatika/tap-google-sheets",
"capabilities" : [ "catalog", "state", "discover" ],
"select" : [ ],
"update" : { },
"vars" : { },
"settings" : [ {
"name" : "oauth_credentials.authorization_url",
"aliases" : [ ],
"label" : "OAuth identity provider authorization endpoint used create and refresh tokens",
"value" : "https://oauth2.googleapis.com/token",
"kind" : "HIDDEN",
"required" : "true",
"protected" : false
}, {
"name" : "oauth_credentials.scope",
"aliases" : [ ],
"label" : "OAuth scopes we need to request access to",
"value" : "https://www.googleapis.com/auth/spreadsheets.readonly https://www.googleapis.com/auth/drive.readonly",
"kind" : "HIDDEN",
"required" : "true",
"protected" : false
}, {
"name" : "oauth_credentials.access_token",
"aliases" : [ ],
"label" : "Access Token",
"kind" : "HIDDEN",
"required" : "true",
"protected" : false
}, {
"name" : "oauth_credentials.refresh_token",
"aliases" : [ ],
"label" : "OAuth Refresh Token",
"kind" : "HIDDEN",
"required" : "true",
"protected" : false
}, {
"name" : "oauth_credentials.refresh_proxy_url",
"aliases" : [ ],
"label" : "Optional - will be called with 'oauth_credentials.refresh_token' to refresh the access token",
"kind" : "HIDDEN",
"protected" : false
}, {
"name" : "oauth_credentials.refresh_proxy_url_auth",
"aliases" : [ ],
"label" : "Optional - Sets Authorization header on 'oauth_credentials.refresh_url' request",
"kind" : "HIDDEN",
"protected" : false
}, {
"name" : "oauth_credentials.client_id",
"aliases" : [ ],
"label" : "Optional - OAuth Client ID used if refresh_proxy_url not supplied",
"kind" : "HIDDEN",
"protected" : false
}, {
"name" : "oauth_credentials.client_secret",
"aliases" : [ ],
"label" : "Optional - OAuth Client Secret used if refresh_proxy_url not supplied",
"kind" : "HIDDEN",
"protected" : false
}, {
"name" : "sheet_id",
"aliases" : [ ],
"label" : "Sheet ID",
"kind" : "STRING",
"required" : "true",
"protected" : false
}, {
"name" : "stream_name",
"aliases" : [ ],
"label" : "Stream Name (Optional)",
"kind" : "STRING",
"protected" : false
}, {
"name" : "child_sheet_name",
"aliases" : [ ],
"label" : "Child Sheet Name (Optional)",
"kind" : "STRING",
"protected" : false
} ],
"variants" : [ ],
"commands" : { },
"matatikaHidden" : false,
"_links" : {
"self" : {
"href" : "https://catalog.matatika.com/api/dataplugins/5a4a153f-0ec0-4d55-8b04-2e04e008dca1"
}
}
}, {
"id" : "0b2aeb17-7cfa-490c-81c8-9dd34fef6f52",
"pluginType" : "EXTRACTOR",
"name" : "tap-postgres",
"namespace" : "tap_postgres",
"variant" : "transferwise",
"label" : "PostgreSQL",
"description" : "PostgreSQL database extractor",
"logoUrl" : "/assets/logos/extractors/postgres.png",
"hidden" : false,
"pipUrl" : "pipelinewise-tap-postgres",
"repo" : "https://github.com/transferwise/pipelinewise-tap-postgres",
"capabilities" : [ "catalog", "state", "discover" ],
"select" : [ ],
"update" : { },
"vars" : { },
"settings" : [ {
"name" : "host",
"aliases" : [ ],
"label" : "Host",
"value" : "localhost",
"kind" : "STRING",
"description" : "PostgreSQL host",
"required" : "true",
"protected" : false
}, {
"name" : "port",
"aliases" : [ ],
"label" : "Port",
"value" : "5432",
"kind" : "INTEGER",
"description" : "PostgreSQL port",
"required" : "true",
"protected" : false
}, {
"name" : "user",
"aliases" : [ ],
"label" : "User",
"kind" : "STRING",
"description" : "PostgreSQL user",
"required" : "true",
"protected" : false
}, {
"name" : "password",
"aliases" : [ ],
"label" : "Password",
"kind" : "PASSWORD",
"description" : "PostgreSQL password",
"required" : "true",
"protected" : false
}, {
"name" : "dbname",
"aliases" : [ ],
"label" : "Database Name",
"kind" : "STRING",
"description" : "PostgreSQL database name",
"required" : "true",
"protected" : false
}, {
"name" : "ssl",
"aliases" : [ ],
"label" : "SSL",
"value" : "false",
"kind" : "BOOLEAN",
"description" : "Using SSL via postgres `sslmode='require'` option. If the server does not accept SSL connections or the client certificate is not recognized the connection will fail",
"protected" : false,
"value_post_processor" : "STRINGIFY"
}, {
"name" : "filter_schemas",
"aliases" : [ ],
"label" : "Filter Schemas",
"kind" : "STRING",
"description" : "Scan only the specified comma-separated schemas to improve the performance of data extraction",
"protected" : false
}, {
"name" : "default_replication_method",
"aliases" : [ ],
"label" : "Default Replication Method",
"kind" : "OPTIONS",
"description" : "The prefered replication method.\n\nNote:\n- Key-based incremental (`INCREMENTAL`) requires configuring a `replication_key` column within the\n catalog's stream definitions for each supported stream.\n- Requirements and setup instructions for log-based incremental (`LOG_BASED`) are documented\n [here](https://github.com/transferwise/pipelinewise-tap-postgres#log-based-replication-requirements).\n",
"protected" : false
}, {
"name" : "max_run_seconds",
"aliases" : [ ],
"label" : "Max Run Seconds",
"value" : "43200",
"kind" : "INTEGER",
"description" : "Stop running the tap after certain number of seconds",
"protected" : false
}, {
"name" : "logical_poll_total_seconds",
"aliases" : [ ],
"label" : "Logical Poll Total Seconds",
"value" : "10800",
"kind" : "INTEGER",
"description" : "Stop running the tap when no data received from wal after certain number of seconds",
"protected" : false
}, {
"name" : "break_at_end_lsn",
"aliases" : [ ],
"label" : "Break At End LSN",
"value" : "true",
"kind" : "BOOLEAN",
"description" : "Stop running the tap if the newly received lsn is after the max lsn that was detected when the tap started",
"protected" : false
} ],
"variants" : [ ],
"commands" : { },
"matatikaHidden" : false,
"_links" : {
"self" : {
"href" : "https://catalog.matatika.com/api/dataplugins/0b2aeb17-7cfa-490c-81c8-9dd34fef6f52"
}
}
}, {
"id" : "a2ac4e7b-b259-43ba-bf95-19227c398f51",
"pluginType" : "EXTRACTOR",
"name" : "tap-trello",
"namespace" : "tap_trello",
"variant" : "matatika",
"label" : "Trello",
"description" : "Trello is a collaboration tool that organizes your projects into boards.\n\n## Prerequisites\n\n[Trello Prerequisites](https://www.matatika.com/docs/data-sources/tap-trello/prerequisites)\n\n## Data source settings\n\n### Developer API Key\n\nTo get your Developer API Key:\n\n1. Go to this link: [https://trello.com/app-key](https://trello.com/app-key) and log in if needed.\n2. The Developer API Key will be displayed below the title of the page.\n\n### Access Token\n\nTo get your Access Token:\n\n1. Go to this link: [https://trello.com/app-key](https://trello.com/app-key) and log in if needed.\n2. Below the Developer API Key, click the generate a Token link.\n3. You will now be show a screen about what permissions you are granting to your API key. Click allow at the bottom of the page.\n4. You will then be redirected to a page with a token, this is your Access Token.\n\n## Learn more\n\n[Matatika Trello Data Source Documentation](https://www.matatika.com/docs/data-sources/tap-trello)\n\n[Meltano Hub Trello Documentation](https://hub.meltano.com/taps/trello--matatika)",
"logoUrl" : "/assets/images/datasource/tap-trello.png",
"hidden" : false,
"docs" : "https://www.matatika.com/docs/data-sources/tap-trello/",
"pipUrl" : "git+https://github.com/Matatika/[email protected]",
"repo" : "https://github.com/Matatika/tap-trello",
"capabilities" : [ "catalog", "state", "discover" ],
"select" : [ ],
"update" : { },
"vars" : { },
"settings" : [ {
"name" : "developer_api_key",
"aliases" : [ ],
"label" : "Developer API Key",
"kind" : "PASSWORD",
"required" : "true",
"protected" : false
}, {
"name" : "access_token",
"aliases" : [ ],
"label" : "Access Token",
"kind" : "PASSWORD",
"required" : "true",
"protected" : false
}, {
"name" : "start_date",
"aliases" : [ ],
"kind" : "STRING",
"protected" : false
} ],
"variants" : [ ],
"commands" : { },
"matatikaHidden" : false,
"_links" : {
"self" : {
"href" : "https://catalog.matatika.com/api/dataplugins/a2ac4e7b-b259-43ba-bf95-19227c398f51"
}
}
}, {
"id" : "aec1159a-2110-4a51-b7a7-8e40e98b865f",
"pluginType" : "LOADER",
"name" : "target-postgres",
"namespace" : "postgres_transferwise",
"variant" : "transferwise",
"label" : "Postgres Warehouse",
"description" : "Loader for Postgres Data Warehouse\n\nThe World's Most Advanced Open Source Relational Database\n\n## Settings\n\n### User\nPostgres user\n\n### Password\nPostgres password\n\n### Host\nPostgres host\nExample: `test-host.db.elephantsql.com`\n\n### Port\nPostgres port\nExample: `5432`\n\n### Database Name\nPostgres database name\n\n### Default Target Schema\nName of the schema where tables will be created (no database prefix)",
"logoUrl" : "/assets/logos/loaders/postgres.png",
"hidden" : false,
"pipUrl" : "git+https://github.com/Matatika/[email protected]",
"repo" : "git+https://github.com/Matatika/[email protected]",
"capabilities" : [ ],
"select" : [ ],
"update" : { },
"vars" : { },
"settings" : [ {
"name" : "user",
"aliases" : [ "username" ],
"label" : "User",
"kind" : "STRING",
"description" : "PostgreSQL user",
"required" : "true",
"protected" : false
}, {
"name" : "password",
"aliases" : [ ],
"label" : "Password",
"kind" : "PASSWORD",
"description" : "PostgreSQL password",
"required" : "true",
"protected" : false
}, {
"name" : "host",
"aliases" : [ "address" ],
"label" : "Host",
"kind" : "STRING",
"description" : "PostgreSQL host",
"required" : "true",
"protected" : false
}, {
"name" : "port",
"aliases" : [ ],
"label" : "Port",
"value" : "5432",
"kind" : "INTEGER",
"description" : "PostgreSQL port",
"required" : "true",
"protected" : false
}, {
"name" : "dbname",
"aliases" : [ "database" ],
"label" : "Database Name",
"kind" : "STRING",
"description" : "PostgreSQL database name",
"required" : "true",
"protected" : false
}, {
"name" : "default_target_schema",
"aliases" : [ ],
"label" : "Default Target Schema",
"value" : "analytics",
"kind" : "STRING",
"description" : "Name of the schema where the tables will be created.",
"required" : "true",
"protected" : false
}, {
"name" : "ssl",
"aliases" : [ ],
"label" : "SSL",
"value" : "false",
"kind" : "HIDDEN",
"protected" : false,
"value_post_processor" : "STRINGIFY"
}, {
"name" : "batch_size_rows",
"aliases" : [ ],
"label" : "Batch Size Rows",
"value" : "100000",
"kind" : "INTEGER",
"description" : "Maximum number of rows in each batch. At the end of each batch, the rows in the batch are loaded into Postgres.",
"protected" : false
}, {
"name" : "underscore_camel_case_fields",
"aliases" : [ ],
"label" : "Underscore Camel Case Fields",
"value" : "true",
"kind" : "HIDDEN",
"description" : "Enable underscoring camel case field names, make the resulting postgres column names the same as from the Meltano variant of target-postgres.",
"protected" : false
}, {
"name" : "flush_all_streams",
"aliases" : [ ],
"label" : "Flush All Streams",
"value" : "false",
"kind" : "HIDDEN",
"description" : "Flush and load every stream into Postgres when one batch is full. Warning: This may trigger the COPY command to use files with low number of records.",
"protected" : false
}, {
"name" : "parallelism",
"aliases" : [ ],
"label" : "Parallelism",
"value" : "0",
"kind" : "HIDDEN",
"description" : "The number of threads used to flush tables. 0 will create a thread for each stream, up to parallelism_max. -1 will create a thread for each CPU core. Any other positive number will create that number of threads, up to parallelism_max.",
"protected" : false
}, {
"name" : "parallelism_max",
"aliases" : [ ],
"label" : "Max Parallelism",
"value" : "16",
"kind" : "HIDDEN",
"description" : "Max number of parallel threads to use when flushing tables.",
"protected" : false
}, {
"name" : "default_target_schema_select_permission",
"aliases" : [ ],
"label" : "Default Target Schema Select Permission",
"kind" : "HIDDEN",
"description" : "Grant USAGE privilege on newly created schemas and grant SELECT privilege on newly created tables to a specific role or a list of roles. If `schema_mapping` is not defined then every stream sent by the tap is granted accordingly.",
"protected" : false
}, {
"name" : "schema_mapping",
"aliases" : [ ],
"label" : "Schema Mapping",
"kind" : "HIDDEN",
"description" : "Useful if you want to load multiple streams from one tap to multiple Postgres schemas.\n\nIf the tap sends the `stream_id` in `<schema_name>-<table_name>` format then this option overwrites the `default_target_schema` value.\nNote, that using `schema_mapping` you can overwrite the `default_target_schema_select_permission` value to grant SELECT permissions to different groups per schemas or optionally you can create indices automatically for the replicated tables.\n",
"protected" : false
}, {
"name" : "add_metadata_columns",
"aliases" : [ ],
"label" : "Add Metadata Columns",
"value" : "true",
"kind" : "HIDDEN",
"description" : "Metadata columns add extra row level information about data ingestions, (i.e. when was the row read in source, when was inserted or deleted in postgres etc.) Metadata columns are creating automatically by adding extra columns to the tables with a column prefix `_SDC_`. The column names are following the stitch naming conventions documented at https://www.stitchdata.com/docs/data-structure/integration-schemas#sdc-columns. Enabling metadata columns will flag the deleted rows by setting the `_SDC_DELETED_AT` metadata column. Without the `add_metadata_columns` option the deleted rows from singer taps will not be recongisable in Postgres.",
"protected" : false
}, {
"name" : "hard_delete",
"aliases" : [ ],
"label" : "Hard Delete",
"value" : "false",
"kind" : "HIDDEN",
"description" : "When `hard_delete` option is true then DELETE SQL commands will be performed in Postgres to delete rows in tables. It's achieved by continuously checking the `_SDC_DELETED_AT` metadata column sent by the singer tap. Due to deleting rows requires metadata columns, `hard_delete` option automatically enables the `add_metadata_columns` option as well.",
"protected" : false
}, {
"name" : "data_flattening_max_level",
"aliases" : [ ],
"label" : "Data Flattening Max Level",
"value" : "10",
"kind" : "HIDDEN",
"description" : "Object type RECORD items from taps can be transformed to flattened columns by creating columns automatically. When value is 0 (default) then flattening functionality is turned off.",
"protected" : false
}, {
"name" : "primary_key_required",
"aliases" : [ ],
"label" : "Primary Key Required",
"value" : "false",
"kind" : "BOOLEAN",
"description" : "Log based and Incremental replications on tables with no Primary Key cause duplicates when merging UPDATE events. When set to true, stop loading data if no Primary Key is defined.",
"protected" : false
}, {
"name" : "validate_records",
"aliases" : [ ],
"label" : "Validate Records",
"value" : "false",
"kind" : "BOOLEAN",
"description" : "Validate every single record message to the corresponding JSON schema. This option is disabled by default and invalid RECORD messages will fail only at load time by Postgres. Enabling this option will detect invalid records earlier but could cause performance degradation.",
"protected" : false
}, {
"name" : "temp_dir",
"aliases" : [ ],
"label" : "Temporary Directory",
"kind" : "HIDDEN",
"description" : "(Default: platform-dependent) Directory of temporary CSV files with RECORD messages.",
"protected" : false
} ],
"variants" : [ ],
"commands" : { },
"matatikaHidden" : false,
"_links" : {
"self" : {
"href" : "https://catalog.matatika.com/api/dataplugins/aec1159a-2110-4a51-b7a7-8e40e98b865f"
}
}
}, {
"id" : "168356ff-a397-4038-8cde-f8c04011887e",
"pluginType" : "EXTRACTOR",
"name" : "tap-github",
"namespace" : "tap_github",
"variant" : "matatika",
"label" : "GitHub",
"description" : "GitHub is an online software development platform. It's used for storing, tracking, and collaborating on software projects.\n## Personal Access Token\nGithub token to authenticate with. This token will need to be a Github Classic `Personal Access Token` scoped with everything you need access to.\n\nThese scopes can range from being able to access public endpoints, for example for public repositories, to private repository and endpoint permissions.\n\nFor a quick start if you are syncing a public repository (like the default Repositories value) the only required scope is `public_repo`.\n\nFor help with this you can follow the Github [Creating a Classic Personal Access Token Docs](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/creating-a-personal-access-token#creating-a-personal-access-token-classic)\n## Repositories\nAn array of strings containing the github repos to be included.\nExample: `[\"Matatika/matatika-ce\"]`\n## Learn more\n[Matatika GitHub Data Source Documentation](https://www.matatika.com/docs/data-sources/tap-github)\n[Meltano Hub GitHub Documentation](https://hub.meltano.com/extractors/tap-github/)",
"logoUrl" : "/assets/images/datasource/tap-github.png",
"hidden" : false,
"docs" : "https://www.matatika.com/docs/data-sources/tap-github/",
"pipUrl" : "git+https://github.com/MeltanoLabs/tap-github.git",
"repo" : "https://github.com/MeltanoLabs/tap-github",
"capabilities" : [ "catalog", "about", "state", "discover" ],
"select" : [ ],
"update" : { },
"vars" : { },
"settings" : [ {
"name" : "auth_token",
"aliases" : [ ],
"label" : "Personal Access Token",
"kind" : "PASSWORD",
"required" : "true",
"protected" : false
}, {
"name" : "repositories",
"aliases" : [ ],
"label" : "Repositories",
"kind" : "ARRAY",
"required" : "true",
"protected" : false
} ],
"variants" : [ ],
"commands" : { },
"matatikaHidden" : false,
"_links" : {
"self" : {
"href" : "https://catalog.matatika.com/api/dataplugins/168356ff-a397-4038-8cde-f8c04011887e"
}
}
}, {
"id" : "e8484f4d-d247-43e3-8b71-aa3e867533e2",
"pluginType" : "UTILITY",
"name" : "gdrive",
"namespace" : "utility_gdrive",
"variant" : "matatika",
"label" : "Google Drive",
"description" : "Utility for downloading files from Google Drive. We recommend using an extractor to move the local files to your destination.\n\nPlease note that these files are not commit to your workspace's repository, so you need to fully process the files you want in a single pipeline run.\n\n## Utility settings\n### GDrive File or Folder ID\n\nThe file or folder ID that you want to sync from Google Drive. If you select a folder it will get all files in the top level of that folder, skipping sub folders.\n\nYou can search for files by typing in this setting field, and the results list updates accordingly.\n\nYou will need to select the file or folder from the list after you have found the one you are looking for, as the results in the list contain additional information that the utility needs.\n\n### Output Path\n\nOptionally provide an output path for the downloaded files to go to.\n\n## Learn more\n\n[Google Drive Utility Documentation](https://github.com/Matatika/utility-gdrive/blob/master/README.md)",
"logoUrl" : "/assets/images/utility/gdrive.png",
"hidden" : false,
"pipUrl" : "git+https://github.com/Matatika/[email protected]",
"capabilities" : [ ],
"select" : [ ],
"update" : { },
"vars" : { },
"settings" : [ {
"name" : "file_id",
"aliases" : [ ],
"label" : "GDrive File or Folder ID",
"kind" : "STRING",
"required" : "true",
"protected" : false
}, {
"name" : "output_path",
"aliases" : [ ],
"label" : "Output Path",
"kind" : "STRING",
"env" : "GDRIVE_OUTPUT_PATH",
"protected" : false
}, {
"name" : "oauth_credentials.authorization_url",
"aliases" : [ ],
"label" : "OAuth identity provider authorization endpoint used create and refresh tokens",
"value" : "https://oauth2.googleapis.com/token",
"kind" : "HIDDEN",
"required" : "true",
"protected" : false
}, {
"name" : "oauth_credentials.scope",
"aliases" : [ ],
"label" : "OAuth scopes we need to request access to",
"value" : "https://www.googleapis.com/auth/drive.readonly",
"kind" : "HIDDEN",
"required" : "true",
"protected" : false
}, {
"name" : "oauth_credentials.refresh_proxy_url",
"aliases" : [ ],
"label" : "Optional - will be called with 'oauth_credentials.refresh_token' to refresh the access token",
"kind" : "HIDDEN",
"protected" : false
}, {
"name" : "oauth_credentials.refresh_proxy_url_auth",
"aliases" : [ ],
"label" : "Optional - Sets Authorization header on 'oauth_credentials.refresh_url' request",
"kind" : "HIDDEN",
"protected" : false
}, {
"name" : "oauth_credentials.client_id",
"aliases" : [ ],
"label" : "Optional - OAuth Client ID used if refresh_proxy_url not supplied",
"kind" : "HIDDEN",
"env" : "GDRIVE_CLIENT_ID",
"protected" : false
}, {
"name" : "oauth_credentials.client_secret",
"aliases" : [ ],
"label" : "Optional - OAuth Client Secret used if refresh_proxy_url not supplied",
"kind" : "HIDDEN",
"env" : "GDRIVE_CLIENT_SECRET",
"protected" : false
}, {
"name" : "oauth_credentials.refresh_token",
"aliases" : [ ],
"label" : "OAuth Refresh Token",
"kind" : "HIDDEN",
"env" : "GDRIVE_REFRESH_TOKEN",
"protected" : false
}, {
"name" : "oauth_credentials.access_token",
"aliases" : [ ],
"label" : "Access Token",
"kind" : "HIDDEN",
"env" : "GDRIVE_ACCESS_TOKEN",
"required" : "true",
"protected" : false
} ],
"variants" : [ ],
"commands" : {
"run" : {
"args" : "$GDRIVE_FILE_ID",
"description" : "Run utility-gdrive to download file or folder of files."
}
},
"matatikaHidden" : false,
"_links" : {
"self" : {
"href" : "https://catalog.matatika.com/api/dataplugins/e8484f4d-d247-43e3-8b71-aa3e867533e2"
}
}
} ]
},
"_links" : {
"first" : {
"href" : "https://catalog.matatika.com/api/dataplugins?page=0&size=20"
},
"self" : {
"href" : "https://catalog.matatika.com/api/dataplugins?page=0&size=20"
},
"next" : {
"href" : "https://catalog.matatika.com/api/dataplugins?page=1&size=20"
},
"last" : {
"href" : "https://catalog.matatika.com/api/dataplugins?page=2&size=20"
}
},
"page" : {
"size" : 20,
"totalElements" : 48,
"totalPages" : 3,
"number" : 0
}
}
View the Matatika discovery.yml
GET
/api/discovery.yml
Returns a Meltano discovery.yml containing all dataplugins supported by Matatika.
Request
Example Snippets
- cURL
curl -H "Authorization: Bearer $ACCESS_TOKEN" 'https://catalog.matatika.com:443/api/discovery.yml' -i -X GET \
-H 'Accept: application/json, application/javascript, text/javascript, text/json' \
-H 'Content-Type: application/json'
- Python (requests)
import requests
url = "https://catalog.matatika.com:443/api/discovery.yml"
headers = {
'Authorization': ACCESS_TOKEN
}
response = requests.request("GET", url, headers=headers)
print(response.text.encode('utf8'))
Response
200 OK
version: 20
extractors: []
loaders: []
transformers: []
files: []
utilities: []
View all workspace dataplugins
GET
/api/workspaces/{workspace-id}/dataplugins
Returns all dataplugins available to the workspace {workspace-id}.
Prerequisites
- Workspace
{workspace-id}must exist
Request
Example Snippets
- cURL
curl -H "Authorization: Bearer $ACCESS_TOKEN" 'https://catalog.matatika.com:443/api/workspaces/c8bd869f-87e5-4545-a75d-f29b178079ea/dataplugins' -i -X GET \
-H 'Accept: application/json, application/javascript, text/javascript, text/json' \
-H 'Content-Type: application/json'
- Python (requests)
import requests
url = "https://catalog.matatika.com:443/api/workspaces/c8bd869f-87e5-4545-a75d-f29b178079ea/dataplugins"
headers = {
'Authorization': ACCESS_TOKEN
}
response = requests.request("GET", url, headers=headers)
print(response.text.encode('utf8'))
Response
200 OK
Dataplugin collection with HAL links.
{
"_embedded" : {
"dataplugins" : [ {
"id" : "115f4bd2-6897-479b-816d-c2f9d18fb7cd",
"pluginType" : "LOADER",
"name" : "target-postgres",
"namespace" : "postgres_transferwise",
"variant" : "transferwise",
"label" : "Postgres Warehouse",
"description" : "Loader for Postgres Data Warehouse\n\nThe World's Most Advanced Open Source Relational Database\n\n## Settings\n\n### User\nPostgres user\n\n### Password\nPostgres password\n\n### Host\nPostgres host\nExample: `test-host.db.elephantsql.com`\n\n### Port\nPostgres port\nExample: `5432`\n\n### Database Name\nPostgres database name\n\n### Default Target Schema\nName of the schema where tables will be created (no database prefix)",
"logoUrl" : "/assets/logos/loaders/postgres.png",
"hidden" : false,
"pipUrl" : "git+https://github.com/Matatika/[email protected]",
"repo" : "git+https://github.com/Matatika/[email protected]",
"capabilities" : [ ],
"select" : [ ],
"update" : { },
"vars" : { },
"settings" : [ {
"name" : "user",
"aliases" : [ "username" ],
"label" : "User",
"kind" : "STRING",
"description" : "PostgreSQL user",
"required" : "true",
"protected" : false
}, {
"name" : "password",
"aliases" : [ ],
"label" : "Password",
"kind" : "PASSWORD",
"description" : "PostgreSQL password",
"required" : "true",
"protected" : false
}, {
"name" : "host",
"aliases" : [ "address" ],
"label" : "Host",
"kind" : "STRING",
"description" : "PostgreSQL host",
"required" : "true",
"protected" : false
}, {
"name" : "port",
"aliases" : [ ],
"label" : "Port",
"value" : "5432",
"kind" : "INTEGER",
"description" : "PostgreSQL port",
"required" : "true",
"protected" : false
}, {
"name" : "dbname",
"aliases" : [ "database" ],
"label" : "Database Name",
"kind" : "STRING",
"description" : "PostgreSQL database name",
"required" : "true",
"protected" : false
}, {
"name" : "default_target_schema",
"aliases" : [ ],
"label" : "Default Target Schema",
"value" : "analytics",
"kind" : "STRING",
"description" : "Name of the schema where the tables will be created.",
"required" : "true",
"protected" : false
}, {
"name" : "ssl",
"aliases" : [ ],
"label" : "SSL",
"value" : "false",
"kind" : "HIDDEN",
"protected" : false,
"value_post_processor" : "STRINGIFY"
}, {
"name" : "batch_size_rows",
"aliases" : [ ],
"label" : "Batch Size Rows",
"value" : "100000",
"kind" : "INTEGER",
"description" : "Maximum number of rows in each batch. At the end of each batch, the rows in the batch are loaded into Postgres.",
"protected" : false
}, {
"name" : "underscore_camel_case_fields",
"aliases" : [ ],
"label" : "Underscore Camel Case Fields",
"value" : "true",
"kind" : "HIDDEN",
"description" : "Enable underscoring camel case field names, make the resulting postgres column names the same as from the Meltano variant of target-postgres.",
"protected" : false
}, {
"name" : "flush_all_streams",
"aliases" : [ ],
"label" : "Flush All Streams",
"value" : "false",
"kind" : "HIDDEN",
"description" : "Flush and load every stream into Postgres when one batch is full. Warning: This may trigger the COPY command to use files with low number of records.",
"protected" : false
}, {
"name" : "parallelism",
"aliases" : [ ],
"label" : "Parallelism",
"value" : "0",
"kind" : "HIDDEN",
"description" : "The number of threads used to flush tables. 0 will create a thread for each stream, up to parallelism_max. -1 will create a thread for each CPU core. Any other positive number will create that number of threads, up to parallelism_max.",
"protected" : false
}, {
"name" : "parallelism_max",
"aliases" : [ ],
"label" : "Max Parallelism",
"value" : "16",
"kind" : "HIDDEN",
"description" : "Max number of parallel threads to use when flushing tables.",
"protected" : false
}, {
"name" : "default_target_schema_select_permission",
"aliases" : [ ],
"label" : "Default Target Schema Select Permission",
"kind" : "HIDDEN",
"description" : "Grant USAGE privilege on newly created schemas and grant SELECT privilege on newly created tables to a specific role or a list of roles. If `schema_mapping` is not defined then every stream sent by the tap is granted accordingly.",
"protected" : false
}, {
"name" : "schema_mapping",
"aliases" : [ ],
"label" : "Schema Mapping",
"kind" : "HIDDEN",
"description" : "Useful if you want to load multiple streams from one tap to multiple Postgres schemas.\n\nIf the tap sends the `stream_id` in `<schema_name>-<table_name>` format then this option overwrites the `default_target_schema` value.\nNote, that using `schema_mapping` you can overwrite the `default_target_schema_select_permission` value to grant SELECT permissions to different groups per schemas or optionally you can create indices automatically for the replicated tables.\n",
"protected" : false
}, {
"name" : "add_metadata_columns",
"aliases" : [ ],
"label" : "Add Metadata Columns",
"value" : "true",
"kind" : "HIDDEN",
"description" : "Metadata columns add extra row level information about data ingestions, (i.e. when was the row read in source, when was inserted or deleted in postgres etc.) Metadata columns are creating automatically by adding extra columns to the tables with a column prefix `_SDC_`. The column names are following the stitch naming conventions documented at https://www.stitchdata.com/docs/data-structure/integration-schemas#sdc-columns. Enabling metadata columns will flag the deleted rows by setting the `_SDC_DELETED_AT` metadata column. Without the `add_metadata_columns` option the deleted rows from singer taps will not be recongisable in Postgres.",
"protected" : false
}, {
"name" : "hard_delete",
"aliases" : [ ],
"label" : "Hard Delete",
"value" : "false",
"kind" : "HIDDEN",
"description" : "When `hard_delete` option is true then DELETE SQL commands will be performed in Postgres to delete rows in tables. It's achieved by continuously checking the `_SDC_DELETED_AT` metadata column sent by the singer tap. Due to deleting rows requires metadata columns, `hard_delete` option automatically enables the `add_metadata_columns` option as well.",
"protected" : false
}, {
"name" : "data_flattening_max_level",
"aliases" : [ ],
"label" : "Data Flattening Max Level",
"value" : "10",
"kind" : "HIDDEN",
"description" : "Object type RECORD items from taps can be transformed to flattened columns by creating columns automatically. When value is 0 (default) then flattening functionality is turned off.",
"protected" : false
}, {
"name" : "primary_key_required",
"aliases" : [ ],
"label" : "Primary Key Required",
"value" : "false",
"kind" : "BOOLEAN",
"description" : "Log based and Incremental replications on tables with no Primary Key cause duplicates when merging UPDATE events. When set to true, stop loading data if no Primary Key is defined.",
"protected" : false
}, {
"name" : "validate_records",
"aliases" : [ ],
"label" : "Validate Records",
"value" : "false",
"kind" : "BOOLEAN",
"description" : "Validate every single record message to the corresponding JSON schema. This option is disabled by default and invalid RECORD messages will fail only at load time by Postgres. Enabling this option will detect invalid records earlier but could cause performance degradation.",
"protected" : false
}, {
"name" : "temp_dir",
"aliases" : [ ],
"label" : "Temporary Directory",
"kind" : "HIDDEN",
"description" : "(Default: platform-dependent) Directory of temporary CSV files with RECORD messages.",
"protected" : false
} ],
"variants" : [ ],
"commands" : { },
"matatikaHidden" : false,
"_links" : {
"self" : {
"href" : "https://catalog.matatika.com/api/dataplugins/115f4bd2-6897-479b-816d-c2f9d18fb7cd"
},
"update dataplugin" : {
"href" : "https://catalog.matatika.com/api/workspaces/c8bd869f-87e5-4545-a75d-f29b178079ea/dataplugins/115f4bd2-6897-479b-816d-c2f9d18fb7cd",
"type" : "PUT"
}
}
}, {
"id" : "86e79c55-eb2c-4394-b26a-9e92eaa4a1d5",
"pluginType" : "TRANSFORMER",
"name" : "dbt",
"namespace" : "dbt",
"variant" : "dbt-labs",
"label" : "dbt",
"logoUrl" : "/assets/images/transformer/dbt.png",
"hidden" : false,
"docs" : "https://meltano.com/docs/transforms.html",
"pipUrl" : "dbt-core~=1.3.0 dbt-postgres~=1.3.0 dbt-snowflake~=1.3.0\n",
"repo" : "https://github.com/dbt-labs/dbt-core",
"capabilities" : [ ],
"select" : [ ],
"update" : { },
"vars" : { },
"settings" : [ {
"name" : "project_dir",
"aliases" : [ ],
"value" : "$MELTANO_PROJECT_ROOT/transform",
"kind" : "STRING",
"protected" : false
}, {
"name" : "profiles_dir",
"aliases" : [ ],
"value" : "$MELTANO_PROJECT_ROOT/transform/profile",
"kind" : "STRING",
"env" : "DBT_PROFILES_DIR",
"protected" : false
}, {
"name" : "target",
"aliases" : [ ],
"value" : "$MELTANO_LOAD__DIALECT",
"kind" : "STRING",
"protected" : false
}, {
"name" : "source_schema",
"aliases" : [ ],
"value" : "$MELTANO_LOAD__TARGET_SCHEMA",
"kind" : "STRING",
"protected" : false
}, {
"name" : "target_schema",
"aliases" : [ ],
"value" : "analytics",
"kind" : "STRING",
"protected" : false
}, {
"name" : "models",
"aliases" : [ ],
"value" : "$MELTANO_TRANSFORM__PACKAGE_NAME $MELTANO_EXTRACTOR_NAMESPACE my_meltano_project",
"kind" : "STRING",
"protected" : false
} ],
"variants" : [ ],
"commands" : {
"compile" : {
"args" : "compile",
"description" : "Generates executable SQL from source model, test, and analysis files. Compiled SQL files are written to the target/ directory."
},
"seed" : {
"args" : "seed",
"description" : "Load data from csv files into your data warehouse."
},
"test" : {
"args" : "test",
"description" : "Runs tests on data in deployed models."
},
"deps" : {
"args" : "deps",
"description" : "Pull the most recent version of the dependencies listed in packages.yml"
},
"run" : {
"args" : "run",
"description" : "Compile SQL and execute against the current target database."
},
"clean" : {
"args" : "clean",
"description" : "Delete all folders in the clean-targets list (usually the dbt_modules and target directories.)"
},
"snapshot" : {
"args" : "snapshot",
"description" : "Execute snapshots defined in your project."
}
},
"matatikaHidden" : false,
"_links" : {
"self" : {
"href" : "https://catalog.matatika.com/api/dataplugins/86e79c55-eb2c-4394-b26a-9e92eaa4a1d5"
},
"update dataplugin" : {
"href" : "https://catalog.matatika.com/api/workspaces/c8bd869f-87e5-4545-a75d-f29b178079ea/dataplugins/86e79c55-eb2c-4394-b26a-9e92eaa4a1d5",
"type" : "PUT"
}
}
}, {
"id" : "44e5a09c-72f7-4fe0-8017-251f62cb3196",
"pluginType" : "EXTRACTOR",
"name" : "tap-custom-test",
"variant" : "sit",
"label" : "Tap Custom Test",
"description" : "A dataplugin created during an SIT run",
"hidden" : false,
"pipUrl" : "git+https://github.com/Matatika/example-repository",
"capabilities" : [ ],
"select" : [ ],
"update" : { },
"vars" : { },
"settings" : [ {
"name" : "username",
"aliases" : [ ],
"label" : "Username",
"placeholder" : "username",
"kind" : "STRING",
"description" : "The username login credential.",
"protected" : false
}, {
"name" : "email",
"aliases" : [ ],
"label" : "Email",
"placeholder" : "[email protected]",
"kind" : "EMAIL",
"description" : "The email login credential.",
"protected" : false
}, {
"name" : "start_date",
"aliases" : [ ],
"label" : "Start Date",
"placeholder" : "2020-01-01T00:00:00Z",
"kind" : "DATE_ISO8601",
"description" : "The data to begin extracting data from, in ISO 8601 format.",
"protected" : false
} ],
"variants" : [ ],
"commands" : { },
"matatikaHidden" : false,
"_links" : {
"self" : {
"href" : "https://catalog.matatika.com/api/dataplugins/44e5a09c-72f7-4fe0-8017-251f62cb3196"
},
"update dataplugin" : {
"href" : "https://catalog.matatika.com/api/workspaces/c8bd869f-87e5-4545-a75d-f29b178079ea/dataplugins/44e5a09c-72f7-4fe0-8017-251f62cb3196",
"type" : "PUT"
},
"delete dataplugin" : {
"href" : "https://catalog.matatika.com/api/dataplugins/44e5a09c-72f7-4fe0-8017-251f62cb3196",
"type" : "DELETE"
}
}
}, {
"id" : "6dbee8cb-70de-4b0c-b63a-c4f17b54d0dd",
"pluginType" : "EXTRACTOR",
"name" : "tap-test",
"variant" : "sit",
"hidden" : false,
"capabilities" : [ ],
"select" : [ ],
"update" : { },
"vars" : { },
"settings" : [ ],
"variants" : [ ],
"commands" : { },
"matatikaHidden" : false,
"_links" : {
"self" : {
"href" : "https://catalog.matatika.com/api/dataplugins/6dbee8cb-70de-4b0c-b63a-c4f17b54d0dd"
},
"update dataplugin" : {
"href" : "https://catalog.matatika.com/api/workspaces/c8bd869f-87e5-4545-a75d-f29b178079ea/dataplugins/6dbee8cb-70de-4b0c-b63a-c4f17b54d0dd",
"type" : "PUT"
},
"delete dataplugin" : {
"href" : "https://catalog.matatika.com/api/dataplugins/6dbee8cb-70de-4b0c-b63a-c4f17b54d0dd",
"type" : "DELETE"
}
}
}, {
"id" : "6101d588-6820-4ee2-b2f3-3c140a959f5b",
"pluginType" : "LOADER",
"name" : "target-test",
"variant" : "sit",
"hidden" : false,
"capabilities" : [ ],
"select" : [ ],
"update" : { },
"vars" : { },
"settings" : [ ],
"variants" : [ ],
"commands" : { },
"matatikaHidden" : false,
"_links" : {
"self" : {
"href" : "https://catalog.matatika.com/api/dataplugins/6101d588-6820-4ee2-b2f3-3c140a959f5b"
},
"update dataplugin" : {
"href" : "https://catalog.matatika.com/api/workspaces/c8bd869f-87e5-4545-a75d-f29b178079ea/dataplugins/6101d588-6820-4ee2-b2f3-3c140a959f5b",
"type" : "PUT"
},
"delete dataplugin" : {
"href" : "https://catalog.matatika.com/api/dataplugins/6101d588-6820-4ee2-b2f3-3c140a959f5b",
"type" : "DELETE"
}
}
}, {
"id" : "ac2d246a-ece5-452a-856b-c7c0252751b5",
"pluginType" : "TRANSFORM",
"name" : "dbt-tap-test",
"variant" : "sit",
"hidden" : false,
"capabilities" : [ ],
"select" : [ ],
"update" : { },
"vars" : { },
"settings" : [ ],
"variants" : [ ],
"commands" : { },
"matatikaHidden" : false,
"_links" : {
"self" : {
"href" : "https://catalog.matatika.com/api/dataplugins/ac2d246a-ece5-452a-856b-c7c0252751b5"
},
"update dataplugin" : {
"href" : "https://catalog.matatika.com/api/workspaces/c8bd869f-87e5-4545-a75d-f29b178079ea/dataplugins/ac2d246a-ece5-452a-856b-c7c0252751b5",
"type" : "PUT"
},
"delete dataplugin" : {
"href" : "https://catalog.matatika.com/api/dataplugins/ac2d246a-ece5-452a-856b-c7c0252751b5",
"type" : "DELETE"
}
}
}, {
"id" : "a6a6b4a0-5ff6-477f-abd7-6207d05188ae",
"pluginType" : "FILE",
"name" : "analyze-test",
"variant" : "sit",
"hidden" : false,
"capabilities" : [ ],
"select" : [ ],
"update" : { },
"vars" : { },
"settings" : [ ],
"variants" : [ ],
"commands" : { },
"matatikaHidden" : false,
"_links" : {
"self" : {
"href" : "https://catalog.matatika.com/api/dataplugins/a6a6b4a0-5ff6-477f-abd7-6207d05188ae"
},
"update dataplugin" : {
"href" : "https://catalog.matatika.com/api/workspaces/c8bd869f-87e5-4545-a75d-f29b178079ea/dataplugins/a6a6b4a0-5ff6-477f-abd7-6207d05188ae",
"type" : "PUT"
},
"delete dataplugin" : {
"href" : "https://catalog.matatika.com/api/dataplugins/a6a6b4a0-5ff6-477f-abd7-6207d05188ae",
"type" : "DELETE"
}
}
} ]
},
"_links" : {
"self" : {
"href" : "https://catalog.matatika.com/api/workspaces/c8bd869f-87e5-4545-a75d-f29b178079ea/dataplugins?page=0&size=20"
}
},
"page" : {
"size" : 20,
"totalElements" : 7,
"totalPages" : 1,
"number" : 0
}
}
View a workspace discovery.yml
GET
/api/workspaces/{workspace-id}/discovery.yml
Returns a Meltano discovery.yml containing all dataplugins available to the workspace {workspace-id}.
Prerequisites
- Workspace
{workspace-id}must exist
Request
Example Snippets
- cURL
curl -H "Authorization: Bearer $ACCESS_TOKEN" 'https://catalog.matatika.com:443/api/workspaces/c8bd869f-87e5-4545-a75d-f29b178079ea/discovery.yml' -i -X GET \
-H 'Accept: application/json, application/javascript, text/javascript, text/json' \
-H 'Content-Type: application/json'
- Python (requests)
import requests
url = "https://catalog.matatika.com:443/api/workspaces/c8bd869f-87e5-4545-a75d-f29b178079ea/discovery.yml"
headers = {
'Authorization': ACCESS_TOKEN
}
response = requests.request("GET", url, headers=headers)
print(response.text.encode('utf8'))
Response
200 OK
version: 20
extractors:
- id: 6dbee8cb-70de-4b0c-b63a-c4f17b54d0dd
name: tap-test
variant: sit
hidden: false
- id: 44e5a09c-72f7-4fe0-8017-251f62cb3196
name: tap-custom-test
variant: sit
label: Tap Custom Test
description: A dataplugin created during an SIT run
hidden: false
pip_url: git+https://github.com/Matatika/example-repository
settings:
- name: username
label: Username
placeholder: username
kind: string
description: The username login credential.
protected: false
- name: email
label: Email
placeholder: [email protected]
kind: email
description: The email login credential.
protected: false
- name: start_date
label: Start Date
placeholder: 2020-01-01T00:00:00Z
kind: date_iso8601
description: "The data to begin extracting data from, in ISO 8601 format."
protected: false
- id: bdf19f6a-e898-49e6-bb59-8457b33907b1
name: tap-googleads
namespace: tap_googleads
variant: matatika
label: Google Ads
description: |-
App and website ads platform hosted by Google.
## Prerequisites
[Google Ads Prerequisites](https://www.matatika.com/docs/data-sources/tap-googleads/prerequisites)
## Data source settings
### Customer Id
Customer Id is the account id of your Google Ads account with the ads campaigns, and therefore data, that will get synced.
Your Customer Id is usually in the format '111-111-1111' but to make requests to the api we have to remove the dashes.
### Start Date
Providing a start date allows you to set how much historical data you wish to import into you workspace.
If you do not provide a start date the tap will set a date of 90 days ago.
This date has to be in the ISO-8601 date format, see the example below.
Format: `YYYY-MM-DDTHH:MM:SSZ`
Example: `2020-10-01T13:12:11Z`
### End Date
Providing a end date allows you to set how much historical data you wish to import into you workspace.
If you do not provide a end date the tap will sync all data up to the current date.
This date has to be in the ISO-8601 date format, see the example below.
Format: `YYYY-MM-DDTHH:MM:SSZ`
Example: `2020-10-01T13:12:11Z`
## Learn more
[Matatika Google Ads Data Source Documentation](https://www.matatika.com/docs/data-sources/tap-googleads)
[Meltano Hub Google Ads Documentation](https://hub.meltano.com/taps/googleads)
logo_url: /assets/images/datasource/tap-googleads.svg
hidden: false
docs: https://www.matatika.com/docs/data-sources/tap-googleads/
pip_url: git+https://github.com/Matatika/[email protected]0.3.3
repo: https://github.com/Matatika/tap-googleads
capabilities:
- catalog
- state
- discover
settings:
- name: oauth_credentials.authorization_url
label: OAuth identity provider authorization endpoint used create and refresh tokens
value: https://oauth2.googleapis.com/token
kind: hidden
required: "true"
protected: false
- name: oauth_credentials.scope
label: OAuth scopes we need to request access to
value: https://www.googleapis.com/auth/adwords
kind: hidden
required: "true"
protected: false
- name: oauth_credentials.access_token
label: Access Token
kind: hidden
required: "true"
protected: false
- name: oauth_credentials.refresh_token
label: OAuth Refresh Token
kind: hidden
required: "true"
protected: false
- name: oauth_credentials.refresh_proxy_url
label: Optional - will be called with 'oauth_credentials.refresh_token' to refresh the access token
kind: hidden
protected: false
- name: oauth_credentials.refresh_proxy_url_auth
label: Optional - Sets Authorization header on 'oauth_credentials.refresh_url' request
kind: hidden
protected: false
- name: oauth_credentials.client_id
label: Optional - OAuth Client ID used if refresh_proxy_url not supplied
kind: hidden
protected: false
- name: oauth_credentials.client_secret
label: Optional - OAuth Client Secret used if refresh_proxy_url not supplied
kind: hidden
protected: false
- name: start_date
label: Start Date
kind: date_iso8601
protected: false
- name: end_date
label: End Date
kind: date_iso8601
protected: false
- name: developer_token
label: Developer Token
value: DYSuW0qdfU5-jti8Zdh1HQ
kind: hidden
required: "true"
protected: false
- name: customer_id
label: Customer Id
kind: string
required: "true"
protected: false
- id: 51bcb7cb-13ab-4847-bda4-4db40bacf553
name: tap-google-analytics
namespace: tap_google_analytics
variant: matatika
label: Google Analytics
description: |-
App and website analytics platform hosted by Google
## Prerequisites
Before importing data from Google Analytics you or your webmaster will need to have performed some initial setup to grant
yourself access to the Google Analytics for your website.
[Google Analytics Prerequisites](https://www.matatika.com/docs/data-sources/tap-google-analytics/prerequisites)
## Data source settings
### View ID
Once you have connected with your google account, the `View ID` drop down will allow you to choose which view you want to get your data from.
### Start Date
Providing a start date allows you to set how much historical data you wish to import into you workspace.
This date has to be in the ISO-8601 date format, see the example below.
Format: `YYYY-MM-DDTHH:MM:SSZ`
Example: `2020-10-01T13:12:11Z`
### End Date
Providing an end date allows you to limit how much data you wish to import into you workspace.
This date has to be in the ISO-8601 date format, see the example below.
Format: `YYYY-MM-DDTHH:MM:SSZ`
Example: `2020-10-01T13:12:11Z`
### Reports
Providing the path to a reports json file allows you to supply the tap with a user defined definition of tables and information you want to sync from Google Analytics.
This does require you to commit a file to your workspace repository so that Google Analytics can find it during run time, and may also break the dbt transforms that are required to have our included datasets work.
For more information on how to define a custom reports file see the [Meltano Hub Documentation](https://hub.meltano.com/extractors/google-analytics#reports).
## Learn more
[Matatika Google Analytics Data Source Documentation](https://www.matatika.com/docs/data-sources/tap-google-analytics)
[Meltano Hub Google Analytics Documentation](https://hub.meltano.com/extractors/google-analytics)
logo_url: /assets/images/datasource/tap-google-analytics.svg
hidden: false
docs: https://www.matatika.com/docs/data-sources/tap-google-analytics/
pip_url: git+https://github.com/Matatika/[email protected]0.3.0
repo: https://github.com/Matatika/tap-google-analytics
capabilities:
- catalog
- state
- discover
settings:
- name: oauth_credentials.authorization_url
label: OAuth identity provider authorization endpoint used create and refresh tokens
value: https://oauth2.googleapis.com/token
kind: hidden
required: "true"
protected: false
- name: oauth_credentials.scope
label: OAuth scopes we need to request access to
value: profile email https://www.googleapis.com/auth/analytics.readonly
kind: hidden
required: "true"
protected: false
- name: oauth_credentials.access_token
label: Access Token
kind: hidden
required: "true"
protected: false
- name: oauth_credentials.refresh_token
label: OAuth Refresh Token
kind: hidden
required: "true"
protected: false
- name: oauth_credentials.refresh_proxy_url
label: Optional - will be called with 'oauth_credentials.refresh_token' to refresh the access token
kind: hidden
protected: false
- name: oauth_credentials.refresh_proxy_url_auth
label: Optional - Sets Authorization header on 'oauth_credentials.refresh_url' request
kind: hidden
protected: false
- name: oauth_credentials.client_id
label: Optional - OAuth Client ID used if refresh_proxy_url not supplied
kind: hidden
protected: false
- name: oauth_credentials.client_secret
label: Optional - OAuth Client Secret used if refresh_proxy_url not supplied
kind: hidden
protected: false
- name: view_id
label: View ID
placeholder: Ex. 198343027
kind: string
required: "true"
protected: false
- name: reports
label: Reports
placeholder: Ex. my_report_definition.json
kind: string
protected: false
- name: start_date
kind: date_iso8601
protected: false
- name: end_date
kind: date_iso8601
protected: false
- id: 58c0c3c2-94a0-46d9-bfda-3b8905ddb3ab
name: tap-shopify
namespace: tap_shopify
variant: matatika
label: Shopify
description: |-
E-commerce platform for online stores and retail point-of-sale systems.
## Prerequisites
Before importing data from Shopify, you or your store admin may need to perform some
initial setup to grant read access.
[Prerequisites](https://www.matatika.com/docs/data-sources/tap-shopify/prerequisites)
## Settings
### Access Token
The access token to authenticate with the Shopify API. See Prerequisites.
### Store
Shopify store id, use the prefix of your admin url.
e.g. https://<your store id>.myshopify.com/admin
### Start Date
Provide a start date to control the amount of historical data.
Format: `YYYY-MM-DDTHH:MM:SSZ`
Example: `2020-10-01T13:12:11Z`
## Learn more
Learn more about the Shopify data source
[Matatika Shopify Data Source Documentation](https://www.matatika.com/docs/data-sources/tap-shopify)
logo_url: /assets/images/datasource/tap-shopify.png
hidden: false
docs: https://www.matatika.com/docs/data-sources/tap-shopify/
pip_url: git+https://github.com/Matatika/[email protected]0.1.0
repo: https://github.com/Matatika/tap-shopify
capabilities:
- catalog
- state
- discover
settings:
- name: access_token
label: Access Token
kind: password
required: "true"
protected: false
- name: store
label: Store
kind: string
required: "true"
protected: false
- name: start_date
label: Start Date
kind: date_iso8601
protected: false
- id: 33444aa0-a5e9-4edb-927a-d0c15707baa0
name: tap-solarvista
namespace: tap_solarvista
variant: matatika
label: Solarvista Live
description: "Solarvista Live is a Field Service Management platform to manage, plan, and mobilise your engineers.\n\n## Prerequisites\n\nBefore importing data from Solarvista Live you will need to have performed the initial setup in Solarvista.\n\n[Solarvista Live Prerequisites](https://www.matatika.com/docs/data-sources/tap-solarvista/prerequisites)\n\n\n## Data source settings\n\n### Account\n\nSolarvista API `account`\n\nTo get your `account` id you will need to follow the steps that are linked in our [Prerequisites](https://www.matatika.com/docs/data-sources/tap-solarvista/prerequisites) section.\n\n### ClientId\n\nSolarvista API `clientId`\n\nTo get your `clientId` you will need to follow the steps that are linked in our [Prerequisites](https://www.matatika.com/docs/data-sources/tap-solarvista/prerequisites) section.\n\n### Code\n\nSolarvista API `code`\n\nTo get your `code` you will need to follow the steps that are linked in our [Prerequisites](https://www.matatika.com/docs/data-sources/tap-solarvista/prerequisites) section.\n\n### Datasources\n\nLeave this field blank to accept the default Solarvista datasources or customise the type of data to sync, we support:\n- `customer`\n- `site`\n- `equipment`\n- `work-item`\n- `work-item-history`\n- `activity`\n- `users`\n- `territory`\n- `project`\n- `appointment`\n- `skill`\n\n\n### Start Date\n\nProviding a start date allows you to set how much historical data you wish to import into your workspace. \nNote: this setting will only work on your first pipeline run, as from then on we will use your pipeline state to sync only new data for you.\n\nThis date has to be in the ISO-8601 date format, see the example below.\n\nFormat: `YYYY-MM-DDTHH:MM:SSZ`\nExample: `2020-10-01T13:12:11Z`\n\n### Force Start Date\n\nProviding a force start date will make your pipeline sync all data from a specific date, ignoring any previous pipeline state.\n\nThis date has to be in the ISO-8601 date format, see the example below.\n\nFormat: `YYYY-MM-DDTHH:MM:SSZ`\nExample: `2020-10-01T13:12:11Z`\n\n\n## Learn more\n\n[Matatika Solavista Live Data Source Documentation](https://www.matatika.com/docs/data-sources/tap-solarvista)\n\n[Meltano Hub Solarvista Live Documentation](https://hub.meltano.com/taps/solarvista)"
logo_url: /assets/images/datasource/tap-solarvista.png
hidden: false
docs: https://www.matatika.com/docs/data-sources/tap-solarvista/
pip_url: git+https://github.com/Matatika/[email protected]0.3.1
repo: https://github.com/Matatika/tap-solarvista
capabilities:
- state
settings:
- name: datasources
label: Datasources
kind: string
required: "true"
protected: false
- name: account
label: Account
kind: string
required: "true"
protected: false
- name: clientId
label: Client ID
kind: string
required: "true"
protected: false
- name: code
label: Code
kind: password
required: "true"
protected: false
- name: start_date
label: Start Date
kind: date_iso8601
protected: false
- name: force_start_date
label: Force Start Date
kind: date_iso8601
protected: false
- id: 6c5a07d0-8580-4bf3-a56e-fb87f7c24c09
name: tap-auth0
namespace: tap_auth0
variant: matatika
label: Auth0
description: |-
Auth0 is an identity and access management service, that allows you to manage the access of your users.
## Prerequisites
To find out how to get the required credentials for this data source please go to our documentation: [Auth0 Prerequisites](https://www.matatika.com/docs/data-sources/tap-auth0/prerequisites)
## Data source settings
### Client ID
Your Auth0 Machine-to-Machine application client id.
### Client Secret
Your Auth0 Machine-to-Machine application client secret.
### Domain
Your Auth0 tenant domain in the format `<TENANT_NAME>.<REGION_IDENTIFIER>.auth0.com``
## Learn more
[Matatika Auth0 Data Source Documentation](https://www.matatika.com/docs/data-sources/tap-auth0)
[Meltano Hub Auth0 Documentation](https://hub.meltano.com/taps/auth0)
logo_url: /assets/images/datasource/tap-auth0.png
hidden: false
docs: https://www.matatika.com/docs/data-sources/tap-auth0/
pip_url: git+https://github.com/Matatika/tap-auth0@v0.3.1
repo: https://github.com/Matatika/tap-auth0
capabilities:
- catalog
- state
- discover
settings:
- name: client_id
label: Client ID
kind: password
required: "true"
protected: false
- name: client_secret
label: Client Secret
kind: password
required: "true"
protected: false
- name: domain
label: Domain
kind: string
required: "true"
protected: false
- name: job_poll_interval_ms
value: 2000
kind: hidden
protected: false
- name: job_poll_max_count
value: 10
kind: hidden
protected: false
- id: 19946eef-9c56-45aa-bc6d-81594d71d00b
name: tap-govuk-weekly-road-fuel-prices
namespace: tap_govuk_weekly_road_fuel_prices
variant: matatika
label: Weekly road fuel prices
description: |-
Road fuel price statistics providing average UK retail 'pump' prices on a weekly basis.
## Learn more
[Weekly road fuel prices](https://www.gov.uk/government/statistics/weekly-road-fuel-prices)
logo_url: https://www.gov.uk/assets/government-frontend/national-statistics-21661e9982d6f605259af3673ea887a18ecd817763b1a62b154d3551a1f1d1b2.png
hidden: false
docs: https://www.matatika.com/docs/data-sources/tap-spreadsheets-anywhere/
pip_url: git+https://github.com/Matatika/[email protected]0.2.0
repo: https://github.com/Matatika/tap-spreadsheets-anywhere
executable: tap-spreadsheets-anywhere
capabilities:
- catalog
- state
- discover
settings:
- name: tables
value: |-
[{
"path":"https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/1140565/",
"name":"weekly_fuel_prices",
"pattern":"CSV_060323.csv",
"field_names":["Date","ULSP_per_litre","ULSD_per_litre","ULSP_duty","ULSD_duty","ULSP_vat_pc","ULSD_vat_pc"],
"skip_initial":18,
"start_date":"2003-09-06T00:00:00Z",
"key_properties":["date"],
"format":"csv"
}]
kind: array
description: Latest weekly road fuel prices.
required: "false"
protected: false
- id: 168356ff-a397-4038-8cde-f8c04011887e
name: tap-github
namespace: tap_github
variant: matatika
label: GitHub
description: |-
GitHub is an online software development platform. It's used for storing, tracking, and collaborating on software projects.
## Personal Access Token
Github token to authenticate with. This token will need to be a Github Classic `Personal Access Token` scoped with everything you need access to.
These scopes can range from being able to access public endpoints, for example for public repositories, to private repository and endpoint permissions.
For a quick start if you are syncing a public repository (like the default Repositories value) the only required scope is `public_repo`.
For help with this you can follow the Github [Creating a Classic Personal Access Token Docs](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/creating-a-personal-access-token#creating-a-personal-access-token-classic)
## Repositories
An array of strings containing the github repos to be included.
Example: `["Matatika/matatika-ce"]`
## Learn more
[Matatika GitHub Data Source Documentation](https://www.matatika.com/docs/data-sources/tap-github)
[Meltano Hub GitHub Documentation](https://hub.meltano.com/extractors/tap-github/)
logo_url: /assets/images/datasource/tap-github.png
hidden: false
docs: https://www.matatika.com/docs/data-sources/tap-github/
pip_url: git+https://github.com/MeltanoLabs/tap-github.git
repo: https://github.com/MeltanoLabs/tap-github
capabilities:
- catalog
- about
- state
- discover
settings:
- name: auth_token
label: Personal Access Token
kind: password
required: "true"
protected: false
- name: repositories
label: Repositories
kind: array
required: "true"
protected: false
- id: e93070bc-772f-457c-b62f-9a3b067d9b18
name: tap-spreadsheets-anywhere
namespace: tap_spreadsheets_anywhere
variant: matatika
label: Spreadsheets Anywhere
description: |-
Spreadsheets anywhere allows you to read data from spreadsheets (CSVs, Excel, JSON) from cloud or local storage.
## Data source settings
### Files
Files takes JSON objects inside an Array to describe one or more target files.
Example
```json
[{
"path":"https://meltano.com/files",
"name":"gitflixusers",
"pattern":"GitFlixUsers.csv",
"start_date":"2021-01-01T00:00:00Z",
"key_properties":["id"],
"format":"csv"
}]
```
Note no trailing `/` on the path, or prefixed `/` on the pattern.
Spreadsheets Anywhere will build the URL back up correctly, so in this case it will look for a file at `https://meltano.com/files/GitFlixUsers.csv`
Passing the above array containing JSON to our files field will sync the data of the file `GitFlixUsers.csv` into your datastore under the table name `gitflixusers`.
## Learn more
[Matatika Data Source Documentation](https://www.matatika.com/docs/data-sources/tap-spreadsheets-anywhere)
[Meltano Hub Documentation](https://hub.meltano.com/extractors/spreadsheets-anywhere)
logo_url: /assets/images/datasource/tap-spreadsheets-anywhere.png
hidden: false
docs: https://www.matatika.com/docs/data-sources/tap-spreadsheets-anywhere/
pip_url: git+https://github.com/Matatika/[email protected]0.2.0
repo: https://github.com/Matatika/tap-spreadsheets-anywhere
capabilities:
- catalog
- state
- discover
settings:
- name: tables
kind: array
description: An array holding json objects that each describe a set of targeted source files. See docs for details.
required: "true"
protected: false
- id: 376a80e6-9b03-4aaf-905a-d1f9bb2a48ab
name: tap-snowflake
namespace: tap_snowflake
variant: transferwise
label: Snowflake
description: "Extractor for a Snowflake database.\n\n## Settings\n\n### Account\n\nYour Snowflake account name (i.e. `rtXXXXX.eu-central-1`)\n\n### Database Name\n\nYour Snowflake database name that you want to extract\n\n### Snowflake user\n\nYour Snowflake user\n\n### Snowflake password\n\nYour Snowflake password\n\n### Warehouse\n\nSnowflake virtual warehouse name to be used\n\n### Tables\n\nA comma separated list of the table names that you want to sync. The table names should be fully qualified including the db and schema name. For example, syncing table1 and table2 would look like `db.schema.table1, db.schema.table2`\n\n### Role\n\nSnowflake role to use. If not defined then the user's default role will be used.\n\n### Default Replication Method\n\nThe prefered replication method.\n \n Note:\n - Key-based incremental (`INCREMENTAL`) \n - Full Table Replication (`FULL_TABLE`)\n \n### Private Key Path\n\nA path to your private key used for authenticating using Key Pair authentication instead of user/pass.\n\n### Private Key Passphrase\n\nThe private key passphrase used for authenticating using Key Pair authentication instead of user/pass.\n\n### Insecure Mode\n\nUse insecure mode to avoid \\\"Failed to get OCSP response\\\" warnings. Default value is `false`.\n\n## Learn more\n\n[Meltano Hub tap-snowflake Documentation](https://hub.meltano.com/extractors/tap-snowflake/)"
logo_url: /assets/logos/extractors/snowflake.png
hidden: false
pip_url: pipelinewise-tap-snowflake
repo: https://github.com/transferwise/pipelinewise-tap-snowflake
capabilities:
- catalog
- state
- discover
settings:
- name: account
label: Account
kind: string
description: Snowflake account name (i.e. `rtXXXXX.eu-central-1`)
required: "true"
protected: false
- name: dbname
label: Database Name
kind: string
description: Snowflake Database name
required: "true"
protected: false
- name: user
label: User
kind: string
description: Snowflake user
required: "true"
protected: false
- name: password
label: Password
kind: password
description: Snowflake password
required: "true"
protected: false
- name: warehouse
label: warehouse
kind: string
description: Snowflake virtual warehouse name
required: "true"
protected: false
- name: tables
label: Tables
kind: string
description: "A comma separated list of the table names that you want to sync. The table names should be fully qualified including the db and schema name. For example, syncing table1 and table2 would look like `db.schema.table1,db.schema.table2"
required: "true"
protected: false
- name: role
label: Role
kind: string
description: Snowflake role to use. If not defined then the user's default role will be used.
protected: false
- name: default_replication_method
label: Default Replication Method
value: FULL_TABLE
kind: options
description: |
The prefered replication method.
Note:
- Key-based incremental (`INCREMENTAL`) requires configuring a `replication_key` column within the
catalog's stream definitions for each supported stream.
- Requirements and setup instructions for log-based incremental (`LOG_BASED`) are documented
[here](https://github.com/transferwise/pipelinewise-tap-postgres#log-based-replication-requirements).
protected: false
- name: private_key_path
label: Private Key Path
kind: string
description: A path to your private key used for authenticating using Key Pair authentication instead of user/pass.
protected: false
- name: private_key_passphrase
label: Private Key Passphrase
kind: password
description: The private key passphrase used for authenticating using Key Pair authentication instead of user/pass.
protected: false
- name: insecure_mode
label: Insecure Mode
value: "false"
kind: string
description: Use insecure mode to avoid \"Failed to get OCSP response\" warnings.
protected: false
- id: 931124c6-882f-4f0d-b0ca-6db09f1e1948
name: tap-matatika-sit
namespace: tap_matatika_sit
variant: matatika
label: Matatika SIT
description: Test extractor based on tap-spreadsheets-anywhere used during Matatika SIT runs
logo_url: /assets/images/datasource/tap-matatika-sit.svg
hidden: false
docs: https://meltano.com/plugins/extractors/spreadsheets-anywhere.html
pip_url: git+https://github.com/ets/tap-spreadsheets-anywhere.git
repo: https://github.com/ets/tap-spreadsheets-anywhere
executable: tap-spreadsheets-anywhere
capabilities:
- catalog
- state
- discover
settings:
- name: tables
value: "[{\"path\":\"https://raw.githubusercontent.com/Matatika/matatika-examples/master/example_data\",\"name\":\"gitflixusers\",\"pattern\":\"GitFlixUsers.csv\",\"start_date\":\"2021-01-01T00:00:00Z\",\"key_properties\":[\"id\"],\"format\":\"csv\"}]"
kind: array
description: An array holding json objects that each describe a set of targeted source files. See docs for details.
protected: false
- id: a2ac4e7b-b259-43ba-bf95-19227c398f51
name: tap-trello
namespace: tap_trello
variant: matatika
label: Trello
description: |-
Trello is a collaboration tool that organizes your projects into boards.
## Prerequisites
[Trello Prerequisites](https://www.matatika.com/docs/data-sources/tap-trello/prerequisites)
## Data source settings
### Developer API Key
To get your Developer API Key:
1. Go to this link: [https://trello.com/app-key](https://trello.com/app-key) and log in if needed.
2. The Developer API Key will be displayed below the title of the page.
### Access Token
To get your Access Token:
1. Go to this link: [https://trello.com/app-key](https://trello.com/app-key) and log in if needed.
2. Below the Developer API Key, click the generate a Token link.
3. You will now be show a screen about what permissions you are granting to your API key. Click allow at the bottom of the page.
4. You will then be redirected to a page with a token, this is your Access Token.
## Learn more
[Matatika Trello Data Source Documentation](https://www.matatika.com/docs/data-sources/tap-trello)
[Meltano Hub Trello Documentation](https://hub.meltano.com/taps/trello--matatika)
logo_url: /assets/images/datasource/tap-trello.png
hidden: false
docs: https://www.matatika.com/docs/data-sources/tap-trello/
pip_url: git+https://github.com/Matatika/[email protected]0.3.0
repo: https://github.com/Matatika/tap-trello
capabilities:
- catalog
- state
- discover
settings:
- name: developer_api_key
label: Developer API Key
kind: password
required: "true"
protected: false
- name: access_token
label: Access Token
kind: password
required: "true"
protected: false
- name: start_date
kind: string
protected: false
- id: c439b543-9822-4294-b183-d20a13490fb2
name: tap-meltano
namespace: tap_meltano
variant: matatika
label: Meltano
description: |-
Meltano is a data pipeline and modular open source data ops platform.
When you run a Meltano job and you have a `MELTANO_DATABASE_URI` set, data about that job run is stored in that database.
By using this data import, and providing an existing `MELTANO_DATABASE_URI`, you will get insights on Meltano job runs.
## Prerequisites
[Meltano Prerequisites](https://www.matatika.com/docs/data-sources/tap-meltano/prerequisites)
## Data source settings
### Meltano Database URI
Your target Meltano project's database URI. By default, (if left blank), this value will be set to your workspace's Meltano database URI.
Example database URI: `postgresql://<username>:<password>@<host>:<port>/<database>?options=-csearch_path%3D<schema>`
## Learn more
[Matatika Meltano Data Source Documentation](https://www.matatika.com/docs/data-sources/tap-meltano)
[Meltano Hub Meltano Documentation](https://hub.meltano.com/taps/meltano)
logo_url: /assets/images/datasource/tap-meltano.png
hidden: false
docs: https://www.matatika.com/docs/data-sources/tap-meltano/
pip_url: git+https://github.com/Matatika/[email protected]0.3.0
repo: https://github.com/Matatika/tap-meltano
capabilities:
- catalog
- state
- discover
settings:
- name: meltano_database_uri
value: $MELTANO_DATABASE_URI
kind: password
required: "true"
protected: false
- id: 0b2aeb17-7cfa-490c-81c8-9dd34fef6f52
name: tap-postgres
namespace: tap_postgres
variant: transferwise
label: PostgreSQL
description: PostgreSQL database extractor
logo_url: /assets/logos/extractors/postgres.png
hidden: false
pip_url: pipelinewise-tap-postgres
repo: https://github.com/transferwise/pipelinewise-tap-postgres
capabilities:
- catalog
- state
- discover
settings:
- name: host
label: Host
value: localhost
kind: string
description: PostgreSQL host
required: "true"
protected: false
- name: port
label: Port
value: 5432
kind: integer
description: PostgreSQL port
required: "true"
protected: false
- name: user
label: User
kind: string
description: PostgreSQL user
required: "true"
protected: false
- name: password
label: Password
kind: password
description: PostgreSQL password
required: "true"
protected: false
- name: dbname
label: Database Name
kind: string
description: PostgreSQL database name
required: "true"
protected: false
- name: ssl
label: SSL
value: "false"
kind: boolean
description: Using SSL via postgres `sslmode='require'` option. If the server does not accept SSL connections or the client certificate is not recognized the connection will fail
protected: false
value_post_processor: stringify
- name: filter_schemas
label: Filter Schemas
kind: string
description: Scan only the specified comma-separated schemas to improve the performance of data extraction
protected: false
- name: default_replication_method
label: Default Replication Method
kind: options
description: |
The prefered replication method.
Note:
- Key-based incremental (`INCREMENTAL`) requires configuring a `replication_key` column within the
catalog's stream definitions for each supported stream.
- Requirements and setup instructions for log-based incremental (`LOG_BASED`) are documented
[here](https://github.com/transferwise/pipelinewise-tap-postgres#log-based-replication-requirements).
protected: false
- name: max_run_seconds
label: Max Run Seconds
value: 43200
kind: integer
description: Stop running the tap after certain number of seconds
protected: false
- name: logical_poll_total_seconds
label: Logical Poll Total Seconds
value: 10800
kind: integer
description: Stop running the tap when no data received from wal after certain number of seconds
protected: false
- name: break_at_end_lsn
label: Break At End LSN
value: "true"
kind: boolean
description: Stop running the tap if the newly received lsn is after the max lsn that was detected when the tap started
protected: false
- id: 2798a08b-dec4-4f73-bac6-cfaf567302e5
name: tap-spotify
namespace: tap_spotify
variant: matatika
label: Spotify
description: |-
Spotify is a digital music streaming service.
By using this data import you will see insights into your music trends.
## Prerequisites
To get you Spotify credentials you can follow the steps listed in the [Spotify Prerequisites](https://www.matatika.com/docs/data-sources/tap-spotify/prerequisites).
## Data source settings
### Client ID
Your Spotify Client ID.
### Client Secret
Your Spotify Client Secret.
### Refresh Token
Your Spotify Client Refresh Token.
## Learn more
[Matatika Spotify Data Source Documentation](https://www.matatika.com/docs/data-sources/tap-spotify)
[Meltano Hub Spotify Documentation](https://hub.meltano.com/taps/spotify)
logo_url: /assets/images/datasource/tap-spotify.png
hidden: false
docs: https://www.matatika.com/docs/data-sources/tap-spotify/
pip_url: git+https://github.com/Matatika/[email protected]0.3.0
repo: https://github.com/Matatika/tap-spotify
capabilities:
- catalog
- state
- discover
settings:
- name: client_id
kind: password
required: "true"
protected: false
- name: client_secret
kind: password
required: "true"
protected: false
- name: refresh_token
kind: password
required: "true"
protected: false
- id: 5a4a153f-0ec0-4d55-8b04-2e04e008dca1
name: tap-google-sheets
namespace: tap_google_sheets
variant: matatika
label: Google Sheets
description: |-
Google Sheets is a web-based application that enables users to create, update and modify spreadsheets and share the data online in real time.
## Data source settings
### Sheet ID - `Required`
Once you have connected with your google account, the `Sheet ID` drop down will allow you to choose which sheet you want to get your data from.
### Stream Name - `Optional`
Set the stream name to control the table or file name output by the target. By default this will be the file name of your Google Sheet.
### Child Sheet Name - `Optional`
Pick a different sheet within your Google Sheet to sync data from. By default we sync data from the first visible sheet in your Google Sheet.
## Learn more
[Matatika Google Sheets Data Source Documentation](https://www.matatika.com/docs/data-sources/tap-google-sheets)
logo_url: /assets/images/datasource/tap-google-sheets.png
hidden: false
docs: https://www.matatika.com/docs/data-sources/tap-google-sheets/
pip_url: git+https://github.com/Matatika/[email protected]0.2.0
repo: https://github.com/Matatika/tap-google-sheets
capabilities:
- catalog
- state
- discover
settings:
- name: oauth_credentials.authorization_url
label: OAuth identity provider authorization endpoint used create and refresh tokens
value: https://oauth2.googleapis.com/token
kind: hidden
required: "true"
protected: false
- name: oauth_credentials.scope
label: OAuth scopes we need to request access to
value: https://www.googleapis.com/auth/spreadsheets.readonly https://www.googleapis.com/auth/drive.readonly
kind: hidden
required: "true"
protected: false
- name: oauth_credentials.access_token
label: Access Token
kind: hidden
required: "true"
protected: false
- name: oauth_credentials.refresh_token
label: OAuth Refresh Token
kind: hidden
required: "true"
protected: false
- name: oauth_credentials.refresh_proxy_url
label: Optional - will be called with 'oauth_credentials.refresh_token' to refresh the access token
kind: hidden
protected: false
- name: oauth_credentials.refresh_proxy_url_auth
label: Optional - Sets Authorization header on 'oauth_credentials.refresh_url' request
kind: hidden
protected: false
- name: oauth_credentials.client_id
label: Optional - OAuth Client ID used if refresh_proxy_url not supplied
kind: hidden
protected: false
- name: oauth_credentials.client_secret
label: Optional - OAuth Client Secret used if refresh_proxy_url not supplied
kind: hidden
protected: false
- name: sheet_id
label: Sheet ID
kind: string
required: "true"
protected: false
- name: stream_name
label: Stream Name (Optional)
kind: string
protected: false
- name: child_sheet_name
label: Child Sheet Name (Optional)
kind: string
protected: false
loaders:
- id: 6101d588-6820-4ee2-b2f3-3c140a959f5b
name: target-test
variant: sit
hidden: false
- id: cb74863b-07d2-4b9a-912f-c7f8172ffc36
name: target-s3csv
namespace: pipelinewise_target_s3_csv
variant: transferwise
label: S3 CSV
description: "The target-s3-csv loader sends data into S3 AWS bucket as CSV after it was pulled from a source using an extractor\n\n## Settings\n### AWS Access Key Id\nYour AWS access key ID \n\n### AWS Secret Access Key\nYour AWS secret access key\n\n### AWS Session token\nYour AWS Session token\n\n### AWS endpoint URL\nAWS endpoint URL\n\n### AWS profile\nName for profile based authentication. If not provided, default profile will be used.\n\n### S3 Bucket name\nS3 Bucket name\n\n### S3 Key Prefix\n(Default - None) A static prefix before the generated S3 key names. Using prefixes you can diferentiate between multiple files.\n\n### Delimiter\n(Default - ',') A one-character string used to separate fields.\n\n### Quote Char\n(Default - '\\\"') A one-character string used to quote fields containing special characters, such as the delimiter or quotechar, or which contain new-line characters.\n\n### Add Metadata Columns\n(Default - False) Metadata columns add extra row level information about data ingestions, (i.e. when was the row read in source, when was inserted or deleted in snowflake etc.) Metadata columns are creating automatically by adding extra columns to the tables with a column prefix _SDC_. The column names are following the stitch naming conventions documented at https://www.stitchdata.com/docs/data-structure/integration-schemas#sdc-columns. Enabling metadata columns will flag the deleted rows by setting the _SDC_DELETED_AT metadata column. Without the add_metadata_columns option the deleted rows from singer taps will not be recongisable in Snowflake.\n\n### Encryption type\n(Default - 'none') The type of encryption to use. Current supported options are - 'none' and 'KMS'.\n\n### Encryption key\nA reference to the encryption key to use for data encryption. For KMS encryption, this should be the name of the KMS encryption key ID (e.g. '1234abcd-1234-1234-1234-1234abcd1234'). This field is ignored if 'encryption_type' is none or blank.\n\n### Compression\nThe type of compression to apply before uploading. Supported options are none (default) and gzip. For gzipped files, the file extension will automatically be changed to .csv.gz for all files.\n\n### Naming Convention\n(Default - None) Custom naming convention of the s3 key. Replaces tokens date, stream, and timestamp with the appropriate values. Supports \\\"folders\\\" in s3 keys e.g. folder/folder2/{stream}/export_date={date}/{timestamp}.csv. Honors the s3_key_prefix, if set, by prepending the \\\"filename\\\". E.g. naming_convention = folder1/my_file.csv and s3_key_prefix = prefix_ results in folder1/prefix_my_file.csv"
logo_url: /assets/logos/extractors/s3-csv.png
hidden: false
docs: https://hub.meltano.com/loaders/target-s3-csv--transferwise
pip_url: git+https://github.com/transferwise/pipelinewise-target-s3-csv.git
repo: https://github.com/transferwise/pipelinewise-target-s3-csv
settings:
- name: aws_access_key_id
label: AWS Access Key Id
kind: password
description: "If not provided, AWS_ACCESS_KEY_ID environment variable will be used."
required: "true"
protected: false
- name: aws_secret_access_key
label: AWS Secret Access Key
kind: password
description: "If not provided, AWS_SECRET_ACCESS_KEY environment variable will be used."
required: "true"
protected: false
- name: aws_session_token
label: AWS Session token
kind: password
description: "If not provided, AWS_SESSION_TOKEN environment variable will be used."
protected: false
- name: aws_endpoint_url
label: AWS endpoint URL
kind: string
description: AWS endpoint URL
protected: false
- name: aws_profile
label: AWS profile
kind: string
description: "TName for profile based authentication. If not provided, AWS_PROFILE environment variable will be used"
protected: false
- name: s3_bucket
label: S3 Bucket name
kind: string
description: S3 Bucket name
required: "true"
protected: false
- name: s3_key_prefix
label: S3 Key Prefix
kind: string
description: (Default - None) A static prefix before the generated S3 key names. Using prefixes you can
protected: false
- name: delimiter
label: delimiter
kind: string
description: "(Default - ',') A one-character string used to separate fields."
protected: false
- name: quotechar
label: Quote Char
kind: string
description: "(Default - '\\\"') A one-character string used to quote fields containing special characters, such as the delimiter or quotechar, or which contain new-line characters."
protected: false
- name: add_metadata_columns
label: Add Metadata Columns
kind: boolean
description: "(Default - False) Metadata columns add extra row level information about data ingestions, (i.e. when was the row read in source, when was inserted or deleted in snowflake etc.) Metadata columns are creating automatically by adding extra columns to the tables with a column prefix _SDC_. The column names are following the stitch naming conventions documented at https://www.stitchdata.com/docs/data-structure/integration-schemas#sdc-columns. Enabling metadata columns will flag the deleted rows by setting the _SDC_DELETED_AT metadata column. Without the add_metadata_columns option the deleted rows from singer taps will not be recongisable in Snowflake."
protected: false
- name: encryption_type
label: Encryption Type
kind: string
description: (Default - 'none') The type of encryption to use. Current supported options are - 'none' and 'KMS'.
protected: false
- name: compression
label: Compression
kind: string
description: "The type of compression to apply before uploading. Supported options are none (default) and gzip. For gzipped files, the file extension will automatically be changed to .csv.gz for all files."
protected: false
- name: naming_convention
label: Naming Convention
kind: string
description: "(Default - None) Custom naming convention of the s3 key. Replaces tokens date, stream, and timestamp with the appropriate values. Supports \\\"folders\\\" in s3 keys e.g. folder/folder2/{stream}/export_date={date}/{timestamp}.csv. Honors the s3_key_prefix, if set, by prepending the \\\"filename\\\". E.g. naming_convention = folder1/my_file.csv and s3_key_prefix = prefix_ results in folder1/prefix_my_file.csv"
protected: false
- id: aec1159a-2110-4a51-b7a7-8e40e98b865f
name: target-postgres
namespace: postgres_transferwise
variant: transferwise
label: Postgres Warehouse
description: |-
Loader for Postgres Data Warehouse
The World's Most Advanced Open Source Relational Database
## Settings
### User
Postgres user
### Password
Postgres password
### Host
Postgres host
Example: `test-host.db.elephantsql.com`
### Port
Postgres port
Example: `5432`
### Database Name
Postgres database name
### Default Target Schema
Name of the schema where tables will be created (no database prefix)
logo_url: /assets/logos/loaders/postgres.png
hidden: false
pip_url: git+https://github.com/Matatika/pipelinewise-t[email protected]0.1.0
repo: git+https://github.com/Matatika/[email protected]0.1.0
settings:
- name: user
aliases:
- username
label: User
kind: string
description: PostgreSQL user
required: "true"
protected: false
- name: password
label: Password
kind: password
description: PostgreSQL password
required: "true"
protected: false
- name: host
aliases:
- address
label: Host
kind: string
description: PostgreSQL host
required: "true"
protected: false
- name: port
label: Port
value: 5432
kind: integer
description: PostgreSQL port
required: "true"
protected: false
- name: dbname
aliases:
- database
label: Database Name
kind: string
description: PostgreSQL database name
required: "true"
protected: false
- name: default_target_schema
label: Default Target Schema
value: analytics
kind: string
description: Name of the schema where the tables will be created.
required: "true"
protected: false
- name: ssl
label: SSL
value: "false"
kind: hidden
protected: false
value_post_processor: stringify
- name: batch_size_rows
label: Batch Size Rows
value: 100000
kind: integer
description: "Maximum number of rows in each batch. At the end of each batch, the rows in the batch are loaded into Postgres."
protected: false
- name: underscore_camel_case_fields
label: Underscore Camel Case Fields
value: "true"
kind: hidden
description: "Enable underscoring camel case field names, make the resulting postgres column names the same as from the Meltano variant of target-postgres."
protected: false
- name: flush_all_streams
label: Flush All Streams
value: "false"
kind: hidden
description: "Flush and load every stream into Postgres when one batch is full. Warning: This may trigger the COPY command to use files with low number of records."
protected: false
- name: parallelism
label: Parallelism
value: 0
kind: hidden
description: "The number of threads used to flush tables. 0 will create a thread for each stream, up to parallelism_max. -1 will create a thread for each CPU core. Any other positive number will create that number of threads, up to parallelism_max."
protected: false
- name: parallelism_max
label: Max Parallelism
value: 16
kind: hidden
description: Max number of parallel threads to use when flushing tables.
protected: false
- name: default_target_schema_select_permission
label: Default Target Schema Select Permission
kind: hidden
description: Grant USAGE privilege on newly created schemas and grant SELECT privilege on newly created tables to a specific role or a list of roles. If `schema_mapping` is not defined then every stream sent by the tap is granted accordingly.
protected: false
- name: schema_mapping
label: Schema Mapping
kind: hidden
description: |
Useful if you want to load multiple streams from one tap to multiple Postgres schemas.
If the tap sends the `stream_id` in `<schema_name>-<table_name>` format then this option overwrites the `default_target_schema` value.
Note, that using `schema_mapping` you can overwrite the `default_target_schema_select_permission` value to grant SELECT permissions to different groups per schemas or optionally you can create indices automatically for the replicated tables.
protected: false
- name: add_metadata_columns
label: Add Metadata Columns
value: "true"
kind: hidden
description: "Metadata columns add extra row level information about data ingestions, (i.e. when was the row read in source, when was inserted or deleted in postgres etc.) Metadata columns are creating automatically by adding extra columns to the tables with a column prefix `_SDC_`. The column names are following the stitch naming conventions documented at https://www.stitchdata.com/docs/data-structure/integration-schemas#sdc-columns. Enabling metadata columns will flag the deleted rows by setting the `_SDC_DELETED_AT` metadata column. Without the `add_metadata_columns` option the deleted rows from singer taps will not be recongisable in Postgres."
protected: false
- name: hard_delete
label: Hard Delete
value: "false"
kind: hidden
description: "When `hard_delete` option is true then DELETE SQL commands will be performed in Postgres to delete rows in tables. It's achieved by continuously checking the `_SDC_DELETED_AT` metadata column sent by the singer tap. Due to deleting rows requires metadata columns, `hard_delete` option automatically enables the `add_metadata_columns` option as well."
protected: false
- name: data_flattening_max_level
label: Data Flattening Max Level
value: 10
kind: hidden
description: Object type RECORD items from taps can be transformed to flattened columns by creating columns automatically. When value is 0 (default) then flattening functionality is turned off.
protected: false
- name: primary_key_required
label: Primary Key Required
value: "false"
kind: boolean
description: "Log based and Incremental replications on tables with no Primary Key cause duplicates when merging UPDATE events. When set to true, stop loading data if no Primary Key is defined."
protected: false
- name: validate_records
label: Validate Records
value: "false"
kind: boolean
description: Validate every single record message to the corresponding JSON schema. This option is disabled by default and invalid RECORD messages will fail only at load time by Postgres. Enabling this option will detect invalid records earlier but could cause performance degradation.
protected: false
- name: temp_dir
label: Temporary Directory
kind: hidden
description: "(Default: platform-dependent) Directory of temporary CSV files with RECORD messages."
protected: false
- id: f6707a9b-34b8-44ce-85ba-29cd5531e0ef
name: target-postgres
namespace: postgres_transferwise
variant: matatika
label: Postgres Warehouse
description: |-
Loader for Postgres Data Warehouse
The World's Most Advanced Open Source Relational Database
## Settings
### User
Postgres user
### Password
Postgres password
### Host
Postgres host
Example: `test-host.db.elephantsql.com`
### Port
Postgres port
Example: `5432`
### Database Name
Postgres database name
### Default Target Schema
Name of the schema where tables will be created (no database prefix)
logo_url: /assets/logos/loaders/postgres.png
hidden: false
pip_url: git+https://github.com/Matatika/[email protected]0.1.0
repo: git+https://github.com/Matatika/[email protected]0.1.0
settings:
- name: user
aliases:
- username
label: User
kind: string
description: PostgreSQL user
required: "true"
protected: false
- name: password
label: Password
kind: password
description: PostgreSQL password
required: "true"
protected: false
- name: host
aliases:
- address
label: Host
kind: string
description: PostgreSQL host
required: "true"
protected: false
- name: port
label: Port
value: 5432
kind: integer
description: PostgreSQL port
required: "true"
protected: false
- name: dbname
aliases:
- database
label: Database Name
kind: string
description: PostgreSQL database name
required: "true"
protected: false
- name: default_target_schema
label: Default Target Schema
value: analytics
kind: string
description: Name of the schema where the tables will be created.
required: "true"
protected: false
- name: ssl
label: SSL
value: "false"
kind: hidden
protected: false
value_post_processor: stringify
- name: batch_size_rows
label: Batch Size Rows
value: 100000
kind: integer
description: "Maximum number of rows in each batch. At the end of each batch, the rows in the batch are loaded into Postgres."
protected: false
- name: underscore_camel_case_fields
label: Underscore Camel Case Fields
value: "true"
kind: hidden
description: "Enable underscoring camel case field names, make the resulting postgres column names the same as from the Meltano variant of target-postgres."
protected: false
- name: flush_all_streams
label: Flush All Streams
value: "false"
kind: hidden
description: "Flush and load every stream into Postgres when one batch is full. Warning: This may trigger the COPY command to use files with low number of records."
protected: false
- name: parallelism
label: Parallelism
value: 0
kind: hidden
description: "The number of threads used to flush tables. 0 will create a thread for each stream, up to parallelism_max. -1 will create a thread for each CPU core. Any other positive number will create that number of threads, up to parallelism_max."
protected: false
- name: parallelism_max
label: Max Parallelism
value: 16
kind: hidden
description: Max number of parallel threads to use when flushing tables.
protected: false
- name: default_target_schema_select_permission
label: Default Target Schema Select Permission
kind: hidden
description: Grant USAGE privilege on newly created schemas and grant SELECT privilege on newly created tables to a specific role or a list of roles. If `schema_mapping` is not defined then every stream sent by the tap is granted accordingly.
protected: false
- name: schema_mapping
label: Schema Mapping
kind: hidden
description: |
Useful if you want to load multiple streams from one tap to multiple Postgres schemas.
If the tap sends the `stream_id` in `<schema_name>-<table_name>` format then this option overwrites the `default_target_schema` value.
Note, that using `schema_mapping` you can overwrite the `default_target_schema_select_permission` value to grant SELECT permissions to different groups per schemas or optionally you can create indices automatically for the replicated tables.
protected: false
- name: add_metadata_columns
label: Add Metadata Columns
value: "true"
kind: hidden
description: "Metadata columns add extra row level information about data ingestions, (i.e. when was the row read in source, when was inserted or deleted in postgres etc.) Metadata columns are creating automatically by adding extra columns to the tables with a column prefix `_SDC_`. The column names are following the stitch naming conventions documented at https://www.stitchdata.com/docs/data-structure/integration-schemas#sdc-columns. Enabling metadata columns will flag the deleted rows by setting the `_SDC_DELETED_AT` metadata column. Without the `add_metadata_columns` option the deleted rows from singer taps will not be recongisable in Postgres."
protected: false
- name: hard_delete
label: Hard Delete
value: "false"
kind: hidden
description: "When `hard_delete` option is true then DELETE SQL commands will be performed in Postgres to delete rows in tables. It's achieved by continuously checking the `_SDC_DELETED_AT` metadata column sent by the singer tap. Due to deleting rows requires metadata columns, `hard_delete` option automatically enables the `add_metadata_columns` option as well."
protected: false
- name: data_flattening_max_level
label: Data Flattening Max Level
value: 10
kind: hidden
description: Object type RECORD items from taps can be transformed to flattened columns by creating columns automatically. When value is 0 (default) then flattening functionality is turned off.
protected: false
- name: primary_key_required
label: Primary Key Required
value: "false"
kind: boolean
description: "Log based and Incremental replications on tables with no Primary Key cause duplicates when merging UPDATE events. When set to true, stop loading data if no Primary Key is defined."
protected: false
- name: validate_records
label: Validate Records
value: "false"
kind: boolean
description: Validate every single record message to the corresponding JSON schema. This option is disabled by default and invalid RECORD messages will fail only at load time by Postgres. Enabling this option will detect invalid records earlier but could cause performance degradation.
protected: false
- name: temp_dir
label: Temporary Directory
kind: hidden
description: "(Default: platform-dependent) Directory of temporary CSV files with RECORD messages."
protected: false
- id: 8c4592e0-fc0e-4f71-babd-b1694c466d01
name: target-snowflake
namespace: snowflake
variant: matatika
label: Snowflake
description: |-
Snowflake Data Warehouse
Accelerate your analytics with the data platform built to enable the modern cloud data warehouse.
## Settings
### Account
Snowflake account
Example: `XC93539`
### Username
Snowflake login username
### Password
Snowflake login password
### Role
Name of the role used for loading data - this role is granted to all created schemas and tables
Example: `PUBLIC`
### Database
Snowflake database name
### Warehouse
Snowflake virtual warehouse name
Example: `COMPUTE_WH`
### Schema
Name of the schema where tables will be created (no database prefix)
### Batch Size
Maximum number of rows in each [batch](https://docs.snowflake.com/en/sql-reference/external-functions-best-practices.html#process-one-row-at-a-time)
### Timestamp Column
Name of the column used to store timestamps corresponding to when data was loaded into the table
logo_url: /assets/images/datastore/target-snowflake.png
hidden: false
docs: https://hub.meltano.com/loaders/snowflake--meltano.html
pip_url: git+https://github.com/Matatika/[email protected]0.1.1
repo: https://github.com/Matatika/target-snowflake
settings:
- name: account
label: Account
kind: string
description: Account Name in Snowflake (https://XXXXX.snowflakecomputing.com)
required: "true"
protected: false
- name: username
label: Username
kind: string
description: The username you use for logging in
required: "true"
protected: false
- name: password
label: Password
kind: password
description: The password you use for logging in
required: "true"
protected: false
- name: role
label: Role
kind: string
description: "Role to be used for loading the data, e.g. `LOADER`. Also this role is GRANTed usage to all tables and schemas created"
required: "true"
protected: false
- name: database
label: Database
kind: string
description: The name of the Snowflake database you want to use
required: "true"
protected: false
- name: warehouse
label: Warehouse
kind: string
description: The name of the Snowflake warehouse you want to use
required: "true"
protected: false
- name: schema
label: Schema
value: analytics
kind: string
required: "true"
protected: false
value_processor: upcase_string
- name: batch_size
label: Batch Size
value: 5000
kind: integer
description: How many records are sent to Snowflake at a time?
protected: false
- name: timestamp_column
label: Timestamp Column
value: __loaded_at
kind: string
description: Name of the column used for recording the timestamp when Data are uploaded to Snowflake.
protected: false
transformers:
- id: 81ca6a43-b7bf-4e3d-b01f-7c9fff39b962
name: dbt
namespace: dbt
variant: dbt-labs
label: dbt
logo_url: /assets/images/transformer/dbt.png
hidden: false
docs: https://meltano.com/docs/transforms.html
pip_url: |
dbt-core~=1.3.0 dbt-postgres~=1.3.0 dbt-snowflake~=1.3.0
repo: https://github.com/dbt-labs/dbt-core
settings:
- name: project_dir
value: $MELTANO_PROJECT_ROOT/transform
kind: string
protected: false
- name: profiles_dir
value: $MELTANO_PROJECT_ROOT/transform/profile
kind: string
env: DBT_PROFILES_DIR
protected: false
- name: target
value: $MELTANO_LOAD__DIALECT
kind: string
protected: false
- name: source_schema
value: $MELTANO_LOAD__TARGET_SCHEMA
kind: string
protected: false
- name: target_schema
value: analytics
kind: string
protected: false
- name: models
value: $MELTANO_TRANSFORM__PACKAGE_NAME $MELTANO_EXTRACTOR_NAMESPACE my_meltano_project
kind: string
protected: false
commands:
compile:
args: compile
description: "Generates executable SQL from source model, test, and analysis files. Compiled SQL files are written to the target/ directory."
seed:
args: seed
description: Load data from csv files into your data warehouse.
test:
args: test
description: Runs tests on data in deployed models.
deps:
args: deps
description: Pull the most recent version of the dependencies listed in packages.yml
run:
args: run
description: Compile SQL and execute against the current target database.
clean:
args: clean
description: Delete all folders in the clean-targets list (usually the dbt_modules and target directories.)
snapshot:
args: snapshot
description: Execute snapshots defined in your project.
files:
- id: a6a6b4a0-5ff6-477f-abd7-6207d05188ae
name: analyze-test
variant: sit
hidden: false
- id: 88550834-321b-4231-bc85-60f4ed240917
name: files-tableau
namespace: tableau
variant: matatika
label: Tableau
description: |-
Tableau is a visual analytics platform transforming the way we use data to solve problems – empowering people and organisations to make the most of their data.
Below you can find the requirements and documentation about connecting Matatika to Tableau.
## Requirements
- Tableau
- Admin/Owner permissions in the Matatika workspace you want to connect to
## Creating the Connection
We recommend the following guide from Tableau on how to connect Tableau to your Matatika workspace database.
[Connecting your data to Tableau](https://help.tableau.com/current/pro/desktop/en-us/basicconnectoverview.htm)
To get the required connection credentials:
1. Go to your Matatika workspace
1. Go to the Lab, then Stores
1. Expand out the datastore you want to connect to, and copy across the required settings.
logo_url: /assets/images/files/tableau.png
hidden: false
pip_url: git+https://github.com/Matatika/analyze-empty
repo: https://github.com/Matatika/analyze-empty
update:
'*.yml': "true"
- id: ffd26c88-aa25-4e04-913c-8dd0b22762d1
name: analyze-trello
namespace: tap_trello
variant: matatika
label: Trello Insights
description: "Instant insights on members, cards, boards, and actions from Trello."
hidden: false
pip_url: git+https://github.com/Matatika/[email protected]0.5.0
repo: https://github.com/Matatika/analyze-trello
update:
'*.yml': "true"
- id: cb48c5e8-f09f-4954-91d5-a7be958a2a62
name: files-looker
namespace: looker
variant: matatika
label: Looker
description: "Looker is a business intelligence software and big data analytics platform that helps you explore, analyze and share real-time business analytics easily.\n\nBelow you can find the requirements and documentation about connecting Matatika to Looker.\n\n## Requirements\n\n- Looker\n- Admin/Owner permissions in the Matatika workspace you want to connect to\n\n## Creating the Connection\n\nWe recommend the following guide from Google on how to connect Looker to your Matatika workspace database. \n\n[Connecting Looker to your database](https://cloud.google.com/looker/docs/connecting-to-your-db#creating_a_new_database_connection)\n\nTo get the required connection credentials:\n\n1. Go to your Matatika workspace\n1. Go to the Lab, then Stores\n1. Expand out the datastore you want to connect to, and copy across the required settings."
logo_url: /assets/images/files/looker.png
hidden: false
pip_url: git+https://github.com/Matatika/analyze-empty
repo: https://github.com/Matatika/analyze-empty
update:
'*.yml': "true"
- id: ea5edd4d-9f41-4b62-a504-a477e0866a2e
name: analyze-spotify
namespace: tap_spotify
variant: matatika
label: Spotify Insights
description: "Instant insights on artists, genres, and tracks from Spotify."
hidden: false
pip_url: git+https://github.com/Matatika/[email protected]0.4.1
repo: https://github.com/Matatika/analyze-spotify
update:
'*.yml': "true"
- id: f3b55dc0-a64c-4b6d-ac05-a50688dbc3ab
name: analyze-shopify
namespace: tap_shopify
variant: matatika
label: Shopify Insights
description: Instant insights on orders from Shopify.
hidden: false
pip_url: git+https://github.com/Matatika/[email protected]0.1.0
repo: https://github.com/Matatika/analyze-shopify
update:
'*.yml': "true"
- id: 14518e68-ecda-48c9-9c93-155453d89ef2
name: analyze-auth0
namespace: tap_auth0
variant: matatika
label: Auth0 Insights
description: "Instant insights on users, logins and quotas from Auth0."
hidden: false
pip_url: git+https://github.com/Matatika/analyze-auth0@v0.5.0
repo: https://github.com/Matatika/analyze-auth0
update:
'*.yml': "true"
- id: 9a1c67dc-4c3a-421b-be3a-f4ccc934343d
name: analyze-snowflake
namespace: dbt-snowflake-monitoring
variant: matatika
label: Snowflake Insights
description: Instant insights on usage and costs from Snowflake.
hidden: false
pip_url: git+https://github.com/Matatika/[email protected]0.1.0
repo: https://github.com/Matatika/analyze-snowflake-monitoring
update:
'*.yml': "true"
- id: 0879ca90-e5ba-49b9-8435-c68676133ac7
name: analyze-meltano
namespace: tap_meltano
variant: matatika
label: Meltano Insights
description: Instant insights on jobs from Meltano.
hidden: false
pip_url: git+https://github.com/Matatika/[email protected]0.4.0
repo: https://github.com/Matatika/analyze-meltano
update:
'*.yml': "true"
- id: 6472b907-3f72-4456-9ce3-dd97236ba84f
name: analyze-google-analytics
namespace: tap_google_analytics
variant: matatika
label: Google Analytics Insights
description: "Instant insights on users, locations, sources, and sessions from Google Analytics."
hidden: false
pip_url: git+https://github.com/Matatika/[email protected]0.4.1
repo: https://github.com/Matatika/analyze-google-analytics
update:
'*.yml': "true"
- id: b17ebf30-37a4-4e4f-aefb-b46cba4f316a
name: analyze-github
namespace: tap_github
variant: matatika
label: Github Insights
description: "Instant insights on commits, issues and PRs from Github."
logo_url: /assets/images/datasource/tap-github.png
hidden: false
pip_url: git+https://github.com/Matatika/[email protected]0.1.0
repo: https://github.com/Matatika/analyze-github
update:
'*.yml': "true"
- id: 07736115-e671-4723-a2fa-abe3bd8fe9f4
name: analyze-googleads
namespace: tap_googleads
variant: matatika
label: Google Ads Insights
description: "Instant insights on campaigns, devices, costs, and clicks from Google Ads."
hidden: false
pip_url: git+https://github.com/Matatika/[email protected]0.5.0
repo: https://github.com/Matatika/analyze-googleads
update:
'*.yml': "true"
- id: dbf87b80-6eb6-483a-90bb-b7a8c094fb3a
name: analyze-solarvista
namespace: tap_solarvista
variant: matatika
label: Solarvista Insights
description: "Instant insights on revenue, projects, work items, and engineer performance from Solarvista Live."
hidden: false
pip_url: git+https://github.com/Matatika/[email protected]0.5.0
repo: https://github.com/Matatika/analyze-solarvista
update:
'*.yml': "true"
- id: 1149bda6-c93f-4db6-a22c-f95afd60d575
name: analyze-sit
namespace: tap_matatika_sit
variant: matatika
hidden: false
pip_url: git+https://github.com/Matatika/analyze-sit.git
repo: https://github.com/Matatika/analyze-sit
update:
analyze/datasets/tap-matatika-sit/user-ages.yml: "true"
analyze/datasets/tap-matatika-sit/user-genders.yml: "true"
- id: e6c1ad3d-ebf5-4c4a-b129-f68156b47555
name: files-dbt
namespace: dbt
variant: matatika
hidden: false
pip_url: git+https://github.com/Matatika/[email protected]1.0.x.1
repo: https://github.com/Matatika/files-dbt
update:
transform/profile/profiles.yml: "true"
- id: 31a05f02-5aeb-4030-aae2-e291c1b14286
name: files-power-bi
namespace: power-bi
variant: matatika
label: Power BI
description: |-
Microsoft Power BI is a business intelligence platform that provides nontechnical business users with tools for aggregating, analyzing, visualizing and sharing data.
Below you can find the requirements and documentation about connecting Matatika to Power BI.
## Requirements
- Power BI
- Admin/Owner permissions in the Matatika workspace you want to connect to
## Creating the Connection
We recommend the following guide from Microsoft on how to connect Power BI to your Matatika workspace database.
[Connect to data in Power BI](https://learn.microsoft.com/en-us/power-bi/connect-data/desktop-quickstart-connect-to-data)
To get the required connection credentials:
1. Go to your Matatika workspace
1. Go to the Lab, then Stores
1. Expand out the datastore you want to connect to, and copy across the required settings.
logo_url: /assets/images/files/power-bi.png
hidden: false
pip_url: git+https://github.com/Matatika/analyze-empty
repo: https://github.com/Matatika/analyze-empty
update:
'*.yml': "true"
utilities:
- id: ec833174-6cec-4a7d-9a6d-7a303ac60811
name: sendgrid
namespace: utility_sendgrid
variant: matatika
label: Sendgrid
description: |-
Utility for sending emails using Sendgrid.
## Utility settings
### API Key
Your Sendgrid API key.
### To Address(es)
One or more email addresses to send your email to.
Example: `[email protected]` or `[email protected], support2@matatika.com`
### From Address
The email address you are sending from. This email address needs to be verified in your Sendgrid [Sender Authentication](https://docs.sendgrid.com/glossary/sender-authentication).
### Title
The title of the email you are sending.
### Body
The body of the email you are sending.
### Attachments
One ore more attachments you want to send with your email.
Example: `path/to/file.pdf` or `path/to/file.pdf, file_two.pdf`
logo_url: /assets/images/utility/sendgrid.png
hidden: false
pip_url: git+https://github.com/Matatika/utility-sendgrid.git
settings:
- name: to_addresses
label: To Address(es)
kind: string
required: "true"
protected: false
- name: from_address
label: From Address
kind: string
required: "true"
protected: false
- name: title
label: Title
kind: string
required: "true"
protected: false
- name: body
label: Body
kind: string
required: "true"
protected: false
- name: attachments
label: Attachments
kind: string
protected: false
- name: api_key
label: API Key
kind: password
required: "true"
protected: false
commands:
send:
args: ' '
description: Sends sendgrid email.
- id: e8484f4d-d247-43e3-8b71-aa3e867533e2
name: gdrive
namespace: utility_gdrive
variant: matatika
label: Google Drive
description: |-
Utility for downloading files from Google Drive. We recommend using an extractor to move the local files to your destination.
Please note that these files are not commit to your workspace's repository, so you need to fully process the files you want in a single pipeline run.
## Utility settings
### GDrive File or Folder ID
The file or folder ID that you want to sync from Google Drive. If you select a folder it will get all files in the top level of that folder, skipping sub folders.
You can search for files by typing in this setting field, and the results list updates accordingly.
You will need to select the file or folder from the list after you have found the one you are looking for, as the results in the list contain additional information that the utility needs.
### Output Path
Optionally provide an output path for the downloaded files to go to.
## Learn more
[Google Drive Utility Documentation](https://github.com/Matatika/utility-gdrive/blob/master/README.md)
logo_url: /assets/images/utility/gdrive.png
hidden: false
pip_url: git+https://github.com/Matatika/[email protected]0.2.0
settings:
- name: file_id
label: GDrive File or Folder ID
kind: string
required: "true"
protected: false
- name: output_path
label: Output Path
kind: string
env: GDRIVE_OUTPUT_PATH
protected: false
- name: oauth_credentials.authorization_url
label: OAuth identity provider authorization endpoint used create and refresh tokens
value: https://oauth2.googleapis.com/token
kind: hidden
required: "true"
protected: false
- name: oauth_credentials.scope
label: OAuth scopes we need to request access to
value: https://www.googleapis.com/auth/drive.readonly
kind: hidden
required: "true"
protected: false
- name: oauth_credentials.refresh_proxy_url
label: Optional - will be called with 'oauth_credentials.refresh_token' to refresh the access token
kind: hidden
protected: false
- name: oauth_credentials.refresh_proxy_url_auth
label: Optional - Sets Authorization header on 'oauth_credentials.refresh_url' request
kind: hidden
protected: false
- name: oauth_credentials.client_id
label: Optional - OAuth Client ID used if refresh_proxy_url not supplied
kind: hidden
env: GDRIVE_CLIENT_ID
protected: false
- name: oauth_credentials.client_secret
label: Optional - OAuth Client Secret used if refresh_proxy_url not supplied
kind: hidden
env: GDRIVE_CLIENT_SECRET
protected: false
- name: oauth_credentials.refresh_token
label: OAuth Refresh Token
kind: hidden
env: GDRIVE_REFRESH_TOKEN
protected: false
- name: oauth_credentials.access_token
label: Access Token
kind: hidden
env: GDRIVE_ACCESS_TOKEN
required: "true"
protected: false
commands:
run:
args: $GDRIVE_FILE_ID
description: Run utility-gdrive to download file or folder of files.
- id: 22440cdb-6931-4bc1-b528-3243bc7613bc
name: matatika
namespace: utility_matatika
variant: matatika
label: Matatika CLI
description: |-
Utility for publishing Matatika datasets.
## Utility settings
### Dataset Path
The path to the Matatika dataset or Jupyter Notebook you want to publish.
### Auth Token
In a Matatika pipeline this setting will be automatically provided for you.
You can manually provide this setting by getting your Developer Token from the API Keys screen in your Matatika workspace.
### Workspace ID
In a Matatika pipeline this setting will be automatically provided for you.
You can manually provide this setting by getting the UUID from your Matatika workspace's URL.
If your workspace URL is `https://app.matatika.com/l/e3fc98d5-6b06-4b75-b845-34f02843f946/feed` your Workspace ID is `e3fc98d5-6b06-4b75-b845-34f02843f946`
### Endpoint URL
In a Matatika pipeline this setting will be automatically provided for you.
If you are running Matatika locally you should set this to: `http://localhost:8080/api`
If you are using the Matatika App online: `https://catalog.matatika.com/api`
logo_url: /assets/images/utility/matatika.png
hidden: false
pip_url: matatika==1.13.0
settings:
- name: dataset_path
label: Dataset Path
kind: string
required: "true"
protected: false
- name: auth_token
label: Auth Token
value: $AUTH_TOKEN
kind: string
env: AUTH_TOKEN
protected: false
- name: workspace_id
label: Workspace ID
value: $WORKSPACE_ID
kind: string
env: WORKSPACE_ID
protected: false
- name: endpoint_url
label: Endpoint URL
value: $ENDPOINT_URL
kind: string
env: ENDPOINT_URL
protected: false
commands:
publish:
args: publish $MATATIKA_DATASET_PATH
description: Publish a matatika dataset.
schedules:
args: schedules
description: Convert Meltano jobs and schedules into Matatika pipeline yamls.
- id: df41147d-d80a-4028-a84e-20120112af55
name: notebook
namespace: utility_notebook
variant: matatika
label: Notebook
description: |-
Utility for running and converting Jupyter notebooks.
## Utility settings
### Path
This setting is the location of the Jupyter notebook or directory of notebooks you want to run and or convert.
### Format
If you want to convert your Jupyter notebooks, you can provide a format.
Currently supported format: `pdf`
### Config
When converting your Jupyter notebooks you can provide additional config to customize the output.
We have one custom config: no-code. This excludes any code output from executed files and makes the output pdf generally neater.
You can pass completely custom configs into this setting, for more information go to the documentation link in the Learn more section below.
## Learn more
[Notebook Utility Documentation](https://github.com/Matatika/utility-notebook/blob/master/README.md)
logo_url: /assets/images/utility/notebook.png
hidden: false
pip_url: git+https://github.com/Matatika/utility-notebook.git
settings:
- name: path
label: Path
kind: string
required: "true"
protected: false
- name: format
label: Format
kind: string
protected: false
- name: config
label: Config
kind: string
protected: false
commands:
run-convert-config:
args: run $NOTEBOOK_PATH -f $NOTEBOOK_FORMAT -c $NOTEBOOK_CONFIG
description: Run and convert one/many notebooks to a selected format with a config.
run:
args: run $NOTEBOOK_PATH
description: "Run one/many notebooks, saving their executed state."
convert:
args: convert $NOTEBOOK_PATH -f $NOTEBOOK_FORMAT
description: Convert one/many notebooks to a selected format.
run-convert:
args: run $NOTEBOOK_PATH -f $NOTEBOOK_FORMAT
description: Run and convert one/many notebooks to a selected format.
- id: 1c8f871f-cd9e-4506-9786-a8f644b71170
name: postgres
namespace: postgres
variant: postgres
label: Postgres
description: The World's Most Advanced Open Source Relational Database.
logo_url: /assets/logos/loaders/postgres.png
hidden: false
docs: https://www.postgresql.org/
repo: https://github.com/postgres/postgres
commands:
start:
args: start
container_spec:
image: postgres:15
env:
PATH: /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/lib/postgresql/15/bin
POSTGRES_USER: warehouse
POSTGRES_PASSWORD: warehouse
POSTGRES_DB: warehouse
LANG: en_US.utf8
ports:
"5432": 5432/tcp
volumes:
- $MELTANO_PROJECT_ROOT/.meltano/utilities/postgres/data/:/var/lib/postgresql/data
transforms:
- id: ac2d246a-ece5-452a-856b-c7c0252751b5
name: dbt-tap-test
variant: sit
hidden: false
- id: a9ba6541-32a3-47ab-bb96-8c4aef3c4ab4
name: dbt-google-analytics
namespace: tap_google_analytics
variant: matatika
hidden: false
pip_url: https://github.com/Matatika/[email protected]0.4.0
repo: https://github.com/Matatika/dbt-tap-google-analytics
vars:
schema: ""
- id: 53c122a6-8924-49d9-a9cb-11a2c0b11ebb
name: dbt-googleads
namespace: tap_googleads
variant: matatika
hidden: false
pip_url: https://github.com/Matatika/[email protected]0.4.0
repo: https://github.com/Matatika/dbt-tap-googleads
vars:
schema: ""
- id: 4f3acdb4-898b-4ddf-a70f-1141f7b73129
name: dbt-solarvista
namespace: tap_solarvista
variant: matatika
hidden: false
pip_url: https://github.com/Matatika/[email protected]0.4.0
repo: https://github.com/Matatika/dbt-tap-solarvista
vars:
schema: ""
- id: 8688dd6b-e9b9-48f9-b1ae-747ef53b071b
name: dbt-meltano
namespace: tap_meltano
variant: matatika
hidden: false
pip_url: https://github.com/Matatika/[email protected]0.4.0
repo: https://github.com/Matatika/dbt-tap-meltano
vars:
schema: ""
- id: 39d26195-9359-453d-9f3a-834e8abefcf1
name: dbt-github
namespace: tap_github
variant: matatika
hidden: false
pip_url: https://github.com/Matatika/[email protected]0.1.0
repo: https://github.com/Matatika/dbt-tap-github
vars:
schema: ""
- id: cd8f6857-4598-4463-a73b-acae80404cb6
name: dbt-snowflake-monitoring
namespace: dbt-snowflake-monitoring
variant: get-select
hidden: false
pip_url: https://github.com/get-select/[email protected]1.1.0
repo: https://github.com/get-select/dbt-snowflake-monitoring
- id: 46bf04a9-fba5-4ba8-82d3-b04a250ec5d3
name: dbt-spotify
namespace: tap_spotify
variant: matatika
hidden: false
pip_url: https://github.com/Matatika/[email protected]0.4.0
repo: https://github.com/Matatika/dbt-tap-spotify
vars:
schema: ""
- id: 512c097b-df0e-4437-ba9a-3374557a30d9
name: dbt-tap-trello
namespace: tap_trello
variant: matatika
hidden: false
pip_url: https://github.com/Matatika/[email protected]0.4.0
repo: https://github.com/Matatika/dbt-tap-trello
vars:
schema: ""
View a dataplugin
GET
/api/dataplugins/{dataplugin-id}
Returns the dataplugin {dataplugin-id}.
Prerequisites
- Dataplugin
{dataplugin-id}must exist
Request
Example Snippets
- cURL
curl -H "Authorization: Bearer $ACCESS_TOKEN" 'https://catalog.matatika.com:443/api/dataplugins/115f4bd2-6897-479b-816d-c2f9d18fb7cd' -i -X GET \
-H 'Accept: application/json, application/javascript, text/javascript, text/json' \
-H 'Content-Type: application/json'
- Python (requests)
import requests
url = "https://catalog.matatika.com:443/api/dataplugins/115f4bd2-6897-479b-816d-c2f9d18fb7cd"
headers = {
'Authorization': ACCESS_TOKEN
}
response = requests.request("GET", url, headers=headers)
print(response.text.encode('utf8'))
Response
200 OK
Dataplugin with HAL links.
{
"id" : "115f4bd2-6897-479b-816d-c2f9d18fb7cd",
"pluginType" : "LOADER",
"name" : "target-postgres",
"namespace" : "postgres_transferwise",
"variant" : "transferwise",
"label" : "Postgres Warehouse",
"description" : "Loader for Postgres Data Warehouse\n\nThe World's Most Advanced Open Source Relational Database\n\n## Settings\n\n### User\nPostgres user\n\n### Password\nPostgres password\n\n### Host\nPostgres host\nExample: `test-host.db.elephantsql.com`\n\n### Port\nPostgres port\nExample: `5432`\n\n### Database Name\nPostgres database name\n\n### Default Target Schema\nName of the schema where tables will be created (no database prefix)",
"logoUrl" : "/assets/logos/loaders/postgres.png",
"hidden" : false,
"pipUrl" : "git+https://github.com/Matatika/[email protected]",
"repo" : "git+https://github.com/Matatika/[email protected]",
"capabilities" : [ ],
"select" : [ ],
"update" : { },
"vars" : { },
"settings" : [ {
"name" : "user",
"aliases" : [ "username" ],
"label" : "User",
"kind" : "STRING",
"description" : "PostgreSQL user",
"required" : "true",
"protected" : false
}, {
"name" : "password",
"aliases" : [ ],
"label" : "Password",
"kind" : "PASSWORD",
"description" : "PostgreSQL password",
"required" : "true",
"protected" : false
}, {
"name" : "host",
"aliases" : [ "address" ],
"label" : "Host",
"kind" : "STRING",
"description" : "PostgreSQL host",
"required" : "true",
"protected" : false
}, {
"name" : "port",
"aliases" : [ ],
"label" : "Port",
"value" : "5432",
"kind" : "INTEGER",
"description" : "PostgreSQL port",
"required" : "true",
"protected" : false
}, {
"name" : "dbname",
"aliases" : [ "database" ],
"label" : "Database Name",
"kind" : "STRING",
"description" : "PostgreSQL database name",
"required" : "true",
"protected" : false
}, {
"name" : "default_target_schema",
"aliases" : [ ],
"label" : "Default Target Schema",
"value" : "analytics",
"kind" : "STRING",
"description" : "Name of the schema where the tables will be created.",
"required" : "true",
"protected" : false
}, {
"name" : "ssl",
"aliases" : [ ],
"label" : "SSL",
"value" : "false",
"kind" : "HIDDEN",
"protected" : false,
"value_post_processor" : "STRINGIFY"
}, {
"name" : "batch_size_rows",
"aliases" : [ ],
"label" : "Batch Size Rows",
"value" : "100000",
"kind" : "INTEGER",
"description" : "Maximum number of rows in each batch. At the end of each batch, the rows in the batch are loaded into Postgres.",
"protected" : false
}, {
"name" : "underscore_camel_case_fields",
"aliases" : [ ],
"label" : "Underscore Camel Case Fields",
"value" : "true",
"kind" : "HIDDEN",
"description" : "Enable underscoring camel case field names, make the resulting postgres column names the same as from the Meltano variant of target-postgres.",
"protected" : false
}, {
"name" : "flush_all_streams",
"aliases" : [ ],
"label" : "Flush All Streams",
"value" : "false",
"kind" : "HIDDEN",
"description" : "Flush and load every stream into Postgres when one batch is full. Warning: This may trigger the COPY command to use files with low number of records.",
"protected" : false
}, {
"name" : "parallelism",
"aliases" : [ ],
"label" : "Parallelism",
"value" : "0",
"kind" : "HIDDEN",
"description" : "The number of threads used to flush tables. 0 will create a thread for each stream, up to parallelism_max. -1 will create a thread for each CPU core. Any other positive number will create that number of threads, up to parallelism_max.",
"protected" : false
}, {
"name" : "parallelism_max",
"aliases" : [ ],
"label" : "Max Parallelism",
"value" : "16",
"kind" : "HIDDEN",
"description" : "Max number of parallel threads to use when flushing tables.",
"protected" : false
}, {
"name" : "default_target_schema_select_permission",
"aliases" : [ ],
"label" : "Default Target Schema Select Permission",
"kind" : "HIDDEN",
"description" : "Grant USAGE privilege on newly created schemas and grant SELECT privilege on newly created tables to a specific role or a list of roles. If `schema_mapping` is not defined then every stream sent by the tap is granted accordingly.",
"protected" : false
}, {
"name" : "schema_mapping",
"aliases" : [ ],
"label" : "Schema Mapping",
"kind" : "HIDDEN",
"description" : "Useful if you want to load multiple streams from one tap to multiple Postgres schemas.\n\nIf the tap sends the `stream_id` in `<schema_name>-<table_name>` format then this option overwrites the `default_target_schema` value.\nNote, that using `schema_mapping` you can overwrite the `default_target_schema_select_permission` value to grant SELECT permissions to different groups per schemas or optionally you can create indices automatically for the replicated tables.\n",
"protected" : false
}, {
"name" : "add_metadata_columns",
"aliases" : [ ],
"label" : "Add Metadata Columns",
"value" : "true",
"kind" : "HIDDEN",
"description" : "Metadata columns add extra row level information about data ingestions, (i.e. when was the row read in source, when was inserted or deleted in postgres etc.) Metadata columns are creating automatically by adding extra columns to the tables with a column prefix `_SDC_`. The column names are following the stitch naming conventions documented at https://www.stitchdata.com/docs/data-structure/integration-schemas#sdc-columns. Enabling metadata columns will flag the deleted rows by setting the `_SDC_DELETED_AT` metadata column. Without the `add_metadata_columns` option the deleted rows from singer taps will not be recongisable in Postgres.",
"protected" : false
}, {
"name" : "hard_delete",
"aliases" : [ ],
"label" : "Hard Delete",
"value" : "false",
"kind" : "HIDDEN",
"description" : "When `hard_delete` option is true then DELETE SQL commands will be performed in Postgres to delete rows in tables. It's achieved by continuously checking the `_SDC_DELETED_AT` metadata column sent by the singer tap. Due to deleting rows requires metadata columns, `hard_delete` option automatically enables the `add_metadata_columns` option as well.",
"protected" : false
}, {
"name" : "data_flattening_max_level",
"aliases" : [ ],
"label" : "Data Flattening Max Level",
"value" : "10",
"kind" : "HIDDEN",
"description" : "Object type RECORD items from taps can be transformed to flattened columns by creating columns automatically. When value is 0 (default) then flattening functionality is turned off.",
"protected" : false
}, {
"name" : "primary_key_required",
"aliases" : [ ],
"label" : "Primary Key Required",
"value" : "false",
"kind" : "BOOLEAN",
"description" : "Log based and Incremental replications on tables with no Primary Key cause duplicates when merging UPDATE events. When set to true, stop loading data if no Primary Key is defined.",
"protected" : false
}, {
"name" : "validate_records",
"aliases" : [ ],
"label" : "Validate Records",
"value" : "false",
"kind" : "BOOLEAN",
"description" : "Validate every single record message to the corresponding JSON schema. This option is disabled by default and invalid RECORD messages will fail only at load time by Postgres. Enabling this option will detect invalid records earlier but could cause performance degradation.",
"protected" : false
}, {
"name" : "temp_dir",
"aliases" : [ ],
"label" : "Temporary Directory",
"kind" : "HIDDEN",
"description" : "(Default: platform-dependent) Directory of temporary CSV files with RECORD messages.",
"protected" : false
} ],
"variants" : [ ],
"commands" : { },
"matatikaHidden" : false,
"_links" : {
"self" : {
"href" : "https://catalog.matatika.com/api/dataplugins/115f4bd2-6897-479b-816d-c2f9d18fb7cd"
},
"update dataplugin" : {
"href" : "https://catalog.matatika.com/api/workspaces/c8bd869f-87e5-4545-a75d-f29b178079ea/dataplugins/115f4bd2-6897-479b-816d-c2f9d18fb7cd",
"type" : "PUT"
}
}
}
Initialise a new dataplugin
POST
/api/dataplugins
Initialises a new dataplugin.
Request
Example Snippets
- cURL
curl -H "Authorization: Bearer $ACCESS_TOKEN" 'https://catalog.matatika.com:443/api/workspaces/c8bd869f-87e5-4545-a75d-f29b178079ea/dataplugins' -i -X POST \
-H 'Accept: application/json, application/javascript, text/javascript, text/json' \
-H 'Content-Type: application/json'
- Python (requests)
import requests
url = "https://catalog.matatika.com:443/api/workspaces/c8bd869f-87e5-4545-a75d-f29b178079ea/dataplugins"
headers = {
'Authorization': ACCESS_TOKEN
}
response = requests.request("POST", url, headers=headers)
print(response.text.encode('utf8'))
Response
200 OK
Dataplugin with HAL links.
{
"id" : "44e5a09c-72f7-4fe0-8017-251f62cb3196",
"hidden" : false,
"capabilities" : [ ],
"select" : [ ],
"update" : { },
"vars" : { },
"settings" : [ ],
"variants" : [ ],
"commands" : { },
"matatikaHidden" : false,
"_links" : {
"self" : {
"href" : "https://catalog.matatika.com/api/dataplugins/44e5a09c-72f7-4fe0-8017-251f62cb3196"
},
"create dataplugin" : {
"href" : "https://catalog.matatika.com/api/workspaces/c8bd869f-87e5-4545-a75d-f29b178079ea/dataplugins/44e5a09c-72f7-4fe0-8017-251f62cb3196",
"type" : "PUT"
}
}
}
Publish dataplugins from a discovery.yml
POST
/api/workspaces/{workspace-id}/discovery.yml
Publishes dataplugins from a Meltano discovery.yml.
Prerequisites
- Workspace
{workspace-id}must exist
Request
Body
version: 19
extractors:
- name: tap-test
variant: sit
loaders:
- name: target-test
variant: sit
transforms:
- name: dbt-tap-test
variant: sit
files:
- name: analyze-test
variant: sit
Example Snippets
- cURL
curl -H "Authorization: Bearer $ACCESS_TOKEN" 'https://catalog.matatika.com:443/api/workspaces/c8bd869f-87e5-4545-a75d-f29b178079ea/discovery' -i -X POST \
-H 'Accept: application/json, application/javascript, text/javascript, text/json' \
-H 'Content-Type: application/json' \
-d 'version: 19
extractors:
- name: tap-test
variant: sit
loaders:
- name: target-test
variant: sit
transforms:
- name: dbt-tap-test
variant: sit
files:
- name: analyze-test
variant: sit
'
- Python (requests)
import requests
url = "https://catalog.matatika.com:443/api/workspaces/c8bd869f-87e5-4545-a75d-f29b178079ea/discovery"
data = version: 19
extractors:
- name: tap-test
variant: sit
loaders:
- name: target-test
variant: sit
transforms:
- name: dbt-tap-test
variant: sit
files:
- name: analyze-test
variant: sit
headers = {
'Authorization': ACCESS_TOKEN
}
response = requests.request("POST", url, headers=headers, data=data)
print(response.text.encode('utf8'))
Response
201 Created
Dataplugin collection with HAL links.
{
"_embedded" : {
"dataplugins" : [ {
"id" : "6dbee8cb-70de-4b0c-b63a-c4f17b54d0dd",
"pluginType" : "EXTRACTOR",
"name" : "tap-test",
"variant" : "sit",
"hidden" : false,
"capabilities" : [ ],
"select" : [ ],
"update" : { },
"vars" : { },
"settings" : [ ],
"variants" : [ ],
"commands" : { },
"matatikaHidden" : false,
"_links" : {
"self" : {
"href" : "https://catalog.matatika.com/api/dataplugins/6dbee8cb-70de-4b0c-b63a-c4f17b54d0dd"
},
"update dataplugin" : {
"href" : "https://catalog.matatika.com/api/workspaces/c8bd869f-87e5-4545-a75d-f29b178079ea/dataplugins/6dbee8cb-70de-4b0c-b63a-c4f17b54d0dd",
"type" : "PUT"
},
"delete dataplugin" : {
"href" : "https://catalog.matatika.com/api/dataplugins/6dbee8cb-70de-4b0c-b63a-c4f17b54d0dd",
"type" : "DELETE"
}
}
}, {
"id" : "6101d588-6820-4ee2-b2f3-3c140a959f5b",
"pluginType" : "LOADER",
"name" : "target-test",
"variant" : "sit",
"hidden" : false,
"capabilities" : [ ],
"select" : [ ],
"update" : { },
"vars" : { },
"settings" : [ ],
"variants" : [ ],
"commands" : { },
"matatikaHidden" : false,
"_links" : {
"self" : {
"href" : "https://catalog.matatika.com/api/dataplugins/6101d588-6820-4ee2-b2f3-3c140a959f5b"
},
"update dataplugin" : {
"href" : "https://catalog.matatika.com/api/workspaces/c8bd869f-87e5-4545-a75d-f29b178079ea/dataplugins/6101d588-6820-4ee2-b2f3-3c140a959f5b",
"type" : "PUT"
},
"delete dataplugin" : {
"href" : "https://catalog.matatika.com/api/dataplugins/6101d588-6820-4ee2-b2f3-3c140a959f5b",
"type" : "DELETE"
}
}
}, {
"id" : "ac2d246a-ece5-452a-856b-c7c0252751b5",
"pluginType" : "TRANSFORM",
"name" : "dbt-tap-test",
"variant" : "sit",
"hidden" : false,
"capabilities" : [ ],
"select" : [ ],
"update" : { },
"vars" : { },
"settings" : [ ],
"variants" : [ ],
"commands" : { },
"matatikaHidden" : false,
"_links" : {
"self" : {
"href" : "https://catalog.matatika.com/api/dataplugins/ac2d246a-ece5-452a-856b-c7c0252751b5"
},
"update dataplugin" : {
"href" : "https://catalog.matatika.com/api/workspaces/c8bd869f-87e5-4545-a75d-f29b178079ea/dataplugins/ac2d246a-ece5-452a-856b-c7c0252751b5",
"type" : "PUT"
},
"delete dataplugin" : {
"href" : "https://catalog.matatika.com/api/dataplugins/ac2d246a-ece5-452a-856b-c7c0252751b5",
"type" : "DELETE"
}
}
}, {
"id" : "a6a6b4a0-5ff6-477f-abd7-6207d05188ae",
"pluginType" : "FILE",
"name" : "analyze-test",
"variant" : "sit",
"hidden" : false,
"capabilities" : [ ],
"select" : [ ],
"update" : { },
"vars" : { },
"settings" : [ ],
"variants" : [ ],
"commands" : { },
"matatikaHidden" : false,
"_links" : {
"self" : {
"href" : "https://catalog.matatika.com/api/dataplugins/a6a6b4a0-5ff6-477f-abd7-6207d05188ae"
},
"update dataplugin" : {
"href" : "https://catalog.matatika.com/api/workspaces/c8bd869f-87e5-4545-a75d-f29b178079ea/dataplugins/a6a6b4a0-5ff6-477f-abd7-6207d05188ae",
"type" : "PUT"
},
"delete dataplugin" : {
"href" : "https://catalog.matatika.com/api/dataplugins/a6a6b4a0-5ff6-477f-abd7-6207d05188ae",
"type" : "DELETE"
}
}
} ]
},
"_links" : {
"self" : {
"href" : "https://catalog.matatika.com/api/dataplugins"
}
}
}
Create a dataplugin
PUT
/api/dataplugins/{dataplugin-id}
Creates the dataplugin {dataplugin-id}.
Request
Body
Dataplugin resource.
{
"name" : "tap-custom-test",
"variant" : "sit",
"pluginType" : "EXTRACTOR",
"label" : "Tap Custom Test",
"description" : "A dataplugin created during an SIT run",
"pipUrl" : "git+https://github.com/Matatika/example-repository",
"settings" : [ {
"name" : "username",
"kind" : "STRING",
"label" : "Username",
"placeholder" : "username",
"description" : "The username login credential.",
"protected" : false
}, {
"name" : "email",
"kind" : "EMAIL",
"label" : "Email",
"description" : "The email login credential.",
"placeholder" : "[email protected]",
"protected" : false
}, {
"name" : "start_date",
"kind" : "DATE_ISO8601",
"label" : "Start Date",
"description" : "The data to begin extracting data from, in ISO 8601 format.",
"placeholder" : "2020-01-01T00:00:00Z",
"protected" : false
} ]
}
Example Snippets
- cURL
curl -H "Authorization: Bearer $ACCESS_TOKEN" 'https://catalog.matatika.com:443/api/workspaces/c8bd869f-87e5-4545-a75d-f29b178079ea/dataplugins/44e5a09c-72f7-4fe0-8017-251f62cb3196' -i -X PUT \
-H 'Accept: application/json, application/javascript, text/javascript, text/json' \
-H 'Content-Type: application/hal+json; charset=ISO-8859-1' \
-d '{
"name" : "tap-custom-test",
"variant" : "sit",
"pluginType" : "EXTRACTOR",
"label" : "Tap Custom Test",
"description" : "A dataplugin created during an SIT run",
"pipUrl" : "git+https://github.com/Matatika/example-repository",
"settings" : [ {
"name" : "username",
"kind" : "STRING",
"label" : "Username",
"placeholder" : "username",
"description" : "The username login credential.",
"protected" : false
}, {
"name" : "email",
"kind" : "EMAIL",
"label" : "Email",
"description" : "The email login credential.",
"placeholder" : "[email protected]",
"protected" : false
}, {
"name" : "start_date",
"kind" : "DATE_ISO8601",
"label" : "Start Date",
"description" : "The data to begin extracting data from, in ISO 8601 format.",
"placeholder" : "2020-01-01T00:00:00Z",
"protected" : false
} ]
}'
- Python (requests)
import requests
url = "https://catalog.matatika.com:443/api/workspaces/c8bd869f-87e5-4545-a75d-f29b178079ea/dataplugins/44e5a09c-72f7-4fe0-8017-251f62cb3196"
data = {
"name" : "tap-custom-test",
"variant" : "sit",
"pluginType" : "EXTRACTOR",
"label" : "Tap Custom Test",
"description" : "A dataplugin created during an SIT run",
"pipUrl" : "git+https://github.com/Matatika/example-repository",
"settings" : [ {
"name" : "username",
"kind" : "STRING",
"label" : "Username",
"placeholder" : "username",
"description" : "The username login credential.",
"protected" : false
}, {
"name" : "email",
"kind" : "EMAIL",
"label" : "Email",
"description" : "The email login credential.",
"placeholder" : "[email protected]",
"protected" : false
}, {
"name" : "start_date",
"kind" : "DATE_ISO8601",
"label" : "Start Date",
"description" : "The data to begin extracting data from, in ISO 8601 format.",
"placeholder" : "2020-01-01T00:00:00Z",
"protected" : false
} ]
}
headers = {
'Authorization': ACCESS_TOKEN
}
response = requests.request("PUT", url, headers=headers, data=data)
print(response.text.encode('utf8'))
Response
201 Created
Dataplugin with HAL links.
{
"id" : "44e5a09c-72f7-4fe0-8017-251f62cb3196",
"pluginType" : "EXTRACTOR",
"name" : "tap-custom-test",
"variant" : "sit",
"label" : "Tap Custom Test",
"description" : "A dataplugin updated during an SIT run",
"hidden" : false,
"pipUrl" : "git+https://github.com/Matatika/example-repository-2",
"capabilities" : [ ],
"select" : [ ],
"update" : { },
"vars" : { },
"settings" : [ {
"name" : "username",
"aliases" : [ ],
"label" : "Username",
"placeholder" : "username",
"kind" : "STRING",
"description" : "The username login credential.",
"protected" : false
}, {
"name" : "email",
"aliases" : [ ],
"label" : "Email",
"placeholder" : "[email protected]",
"kind" : "EMAIL",
"description" : "The email login credential.",
"protected" : false
}, {
"name" : "start_date",
"aliases" : [ ],
"label" : "Start Date",
"placeholder" : "2020-01-01T00:00:00Z",
"kind" : "DATE_ISO8601",
"description" : "The data to begin extracting data from, in ISO 8601 format.",
"protected" : false
} ],
"variants" : [ ],
"commands" : { },
"matatikaHidden" : false,
"_links" : {
"self" : {
"href" : "https://catalog.matatika.com/api/dataplugins/44e5a09c-72f7-4fe0-8017-251f62cb3196"
},
"update dataplugin" : {
"href" : "https://catalog.matatika.com/api/workspaces/c8bd869f-87e5-4545-a75d-f29b178079ea/dataplugins/44e5a09c-72f7-4fe0-8017-251f62cb3196",
"type" : "PUT"
},
"delete dataplugin" : {
"href" : "https://catalog.matatika.com/api/dataplugins/44e5a09c-72f7-4fe0-8017-251f62cb3196",
"type" : "DELETE"
}
}
}
Update a dataplugin
PUT
/api/dataplugins/{dataplugin-id}
Updates the dataplugin {dataplugin-id}.
Prerequisites
- Dataplugin
{dataplugin-id}must exist
Request
Body
Dataplugin resource.
{
"name" : "tap-custom-test",
"variant" : "sit",
"pluginType" : "EXTRACTOR",
"label" : "Tap Custom Test",
"description" : "A dataplugin updated during an SIT run",
"pipUrl" : "git+https://github.com/Matatika/example-repository-2",
"settings" : [ {
"name" : "username",
"kind" : "STRING",
"label" : "Username",
"placeholder" : "username",
"description" : "The username login credential.",
"protected" : false
}, {
"name" : "email",
"kind" : "EMAIL",
"label" : "Email",
"description" : "The email login credential.",
"placeholder" : "[email protected]",
"protected" : false
}, {
"name" : "start_date",
"kind" : "DATE_ISO8601",
"label" : "Start Date",
"description" : "The data to begin extracting data from, in ISO 8601 format.",
"placeholder" : "2020-01-01T00:00:00Z",
"protected" : false
} ]
}
| Path | Type | Description | Constraints |
|---|---|---|---|
description |
String |
A description of the dataplugin | |
repositoryUrl |
String |
A URL to the dataplugin repository | |
settings |
Array of Settings |
The dataplugin settings |
Example Snippets
- cURL
curl -H "Authorization: Bearer $ACCESS_TOKEN" 'https://catalog.matatika.com:443/api/workspaces/c8bd869f-87e5-4545-a75d-f29b178079ea/dataplugins/44e5a09c-72f7-4fe0-8017-251f62cb3196' -i -X PUT \
-H 'Accept: application/json, application/javascript, text/javascript, text/json' \
-H 'Content-Type: application/hal+json; charset=ISO-8859-1' \
-d '{
"name" : "tap-custom-test",
"variant" : "sit",
"pluginType" : "EXTRACTOR",
"label" : "Tap Custom Test",
"description" : "A dataplugin updated during an SIT run",
"pipUrl" : "git+https://github.com/Matatika/example-repository-2",
"settings" : [ {
"name" : "username",
"kind" : "STRING",
"label" : "Username",
"placeholder" : "username",
"description" : "The username login credential.",
"protected" : false
}, {
"name" : "email",
"kind" : "EMAIL",
"label" : "Email",
"description" : "The email login credential.",
"placeholder" : "[email protected]",
"protected" : false
}, {
"name" : "start_date",
"kind" : "DATE_ISO8601",
"label" : "Start Date",
"description" : "The data to begin extracting data from, in ISO 8601 format.",
"placeholder" : "2020-01-01T00:00:00Z",
"protected" : false
} ]
}'
- Python (requests)
import requests
url = "https://catalog.matatika.com:443/api/workspaces/c8bd869f-87e5-4545-a75d-f29b178079ea/dataplugins/44e5a09c-72f7-4fe0-8017-251f62cb3196"
data = {
"name" : "tap-custom-test",
"variant" : "sit",
"pluginType" : "EXTRACTOR",
"label" : "Tap Custom Test",
"description" : "A dataplugin updated during an SIT run",
"pipUrl" : "git+https://github.com/Matatika/example-repository-2",
"settings" : [ {
"name" : "username",
"kind" : "STRING",
"label" : "Username",
"placeholder" : "username",
"description" : "The username login credential.",
"protected" : false
}, {
"name" : "email",
"kind" : "EMAIL",
"label" : "Email",
"description" : "The email login credential.",
"placeholder" : "[email protected]",
"protected" : false
}, {
"name" : "start_date",
"kind" : "DATE_ISO8601",
"label" : "Start Date",
"description" : "The data to begin extracting data from, in ISO 8601 format.",
"placeholder" : "2020-01-01T00:00:00Z",
"protected" : false
} ]
}
headers = {
'Authorization': ACCESS_TOKEN
}
response = requests.request("PUT", url, headers=headers, data=data)
print(response.text.encode('utf8'))
Response
200 OK
Dataplugin with HAL links.
{
"id" : "44e5a09c-72f7-4fe0-8017-251f62cb3196",
"pluginType" : "EXTRACTOR",
"name" : "tap-custom-test",
"variant" : "sit",
"label" : "Tap Custom Test",
"description" : "A dataplugin updated during an SIT run",
"hidden" : false,
"pipUrl" : "git+https://github.com/Matatika/example-repository-2",
"capabilities" : [ ],
"select" : [ ],
"update" : { },
"vars" : { },
"settings" : [ {
"name" : "username",
"aliases" : [ ],
"label" : "Username",
"placeholder" : "username",
"kind" : "STRING",
"description" : "The username login credential.",
"protected" : false
}, {
"name" : "email",
"aliases" : [ ],
"label" : "Email",
"placeholder" : "[email protected]",
"kind" : "EMAIL",
"description" : "The email login credential.",
"protected" : false
}, {
"name" : "start_date",
"aliases" : [ ],
"label" : "Start Date",
"placeholder" : "2020-01-01T00:00:00Z",
"kind" : "DATE_ISO8601",
"description" : "The data to begin extracting data from, in ISO 8601 format.",
"protected" : false
} ],
"variants" : [ ],
"commands" : { },
"matatikaHidden" : false,
"_links" : {
"self" : {
"href" : "https://catalog.matatika.com/api/dataplugins/44e5a09c-72f7-4fe0-8017-251f62cb3196"
},
"update dataplugin" : {
"href" : "https://catalog.matatika.com/api/workspaces/c8bd869f-87e5-4545-a75d-f29b178079ea/dataplugins/44e5a09c-72f7-4fe0-8017-251f62cb3196",
"type" : "PUT"
},
"delete dataplugin" : {
"href" : "https://catalog.matatika.com/api/dataplugins/44e5a09c-72f7-4fe0-8017-251f62cb3196",
"type" : "DELETE"
}
}
}
Delete a dataplugin
DELETE
/api/dataplugins/{dataplugin-id}
Deletes the dataplugin {dataplugin-id}.
Request
Example Snippets
- cURL
curl -H "Authorization: Bearer $ACCESS_TOKEN" 'https://catalog.matatika.com:443/api/dataplugins/44e5a09c-72f7-4fe0-8017-251f62cb3196' -i -X DELETE \
-H 'Accept: application/json, application/javascript, text/javascript, text/json' \
-H 'Content-Type: application/json'
- Python (requests)
import requests
url = "https://catalog.matatika.com:443/api/dataplugins/44e5a09c-72f7-4fe0-8017-251f62cb3196"
headers = {
'Authorization': ACCESS_TOKEN
}
response = requests.request("DELETE", url, headers=headers)
print(response.text.encode('utf8'))
Response
204 No Content
No response body provided.