Dataplugins
Dataplugins simply define a source of data from a given repository. Matatika provides a number of pre-configured platform-wide dataplugins out-the-box, as well as the ability to create custom dataplugins through the API. From these, pipeline jobs can be run to inject data into a workspace.
Objects
Dataplugin
| Path | Type | Format | Description |
|---|---|---|---|
id |
String |
Version 4 UUID | The dataplugin ID |
name |
String |
The dataplugin name | |
description |
String |
A description of the dataplugin | |
repositoryUrl |
String |
URL | The dataplugin repository URL |
settings |
Array of Setting |
The dataplugin settings |
{
"id" : "73b171be-27dc-4b4b-8cea-fce2492eb1ee",
"pluginType" : "LOADER",
"name" : "target-postgres",
"namespace" : "postgres_transferwise",
"variant" : "matatika",
"label" : "Postgres Warehouse",
"description" : "Postgres Warehouse is a data warehousing solution built on top of the Postgres database management system.\n\nPostgres Warehouse is designed to handle large volumes of data and complex queries, making it an ideal solution for businesses that need to store and analyze large amounts of data. It provides a number of features that are specifically tailored to data warehousing, such as columnar storage, parallel processing, and support for advanced analytics. Additionally, Postgres Warehouse is highly scalable, allowing businesses to easily add more resources as their data needs grow. Overall, Postgres Warehouse is a powerful and flexible data warehousing solution that can help businesses make better decisions by providing them with the insights they need to succeed.\n### Prerequisites\nThe process of obtaining the required settings for connecting to a Postgres Warehouse may vary depending on the specific setup and configuration of the database. However, here are some general ways to obtain each of the required settings:\n\n- User: The user is typically created when the database is set up. You can ask the database administrator or check the database documentation to find out the username.\n- Password: The password is also typically created when the database is set up. You can ask the database administrator or check the database documentation to find out the password.\n- Host: The host is the server where the database is located. You can ask the database administrator or check the database documentation to find out the host name or IP address.\n- Port: The port is the number that the database listens on for incoming connections. The default port for Postgres is 5432, but it may be different depending on the configuration. You can ask the database administrator or check the database documentation to find out the port number.\n- Database Name: The database name is the name of the specific database you want to connect to. You can ask the database administrator or check the database documentation to find out the database name.\n- Default Target Schema: The default target schema is the schema that you want to use as the default when connecting to the database. This may be set up by the database administrator or you may need to create it yourself. You can ask the database administrator or check the database documentation to find out the default target schema.",
"logoUrl" : "/assets/logos/loaders/postgres.png",
"hidden" : false,
"docs" : "https://www.matatika.com/data-details/target-postgres/",
"pipUrl" : "git+https://github.com/Matatika/[email protected]",
"repo" : "git+https://github.com/Matatika/[email protected]",
"capabilities" : [ ],
"select" : [ ],
"update" : { },
"vars" : { },
"settings" : [ {
"name" : "user",
"aliases" : [ "username" ],
"label" : "User",
"options" : [ ],
"kind" : "STRING",
"description" : "The username used to connect to the Postgres Warehouse.",
"hidden" : false,
"sensitive" : false,
"required" : "true",
"protected" : false
}, {
"name" : "password",
"aliases" : [ ],
"label" : "Password",
"options" : [ ],
"kind" : "STRING",
"description" : "The password used to authenticate the user.",
"hidden" : false,
"sensitive" : true,
"required" : "true",
"protected" : false
}, {
"name" : "host",
"aliases" : [ "address" ],
"label" : "Host",
"options" : [ ],
"kind" : "STRING",
"description" : "The hostname or IP address of the Postgres Warehouse server.",
"hidden" : false,
"sensitive" : false,
"required" : "true",
"protected" : false
}, {
"name" : "port",
"aliases" : [ ],
"label" : "Port",
"value" : "5432",
"options" : [ ],
"kind" : "INTEGER",
"description" : "The port number used to connect to the Postgres Warehouse server.",
"hidden" : false,
"sensitive" : false,
"required" : "true",
"protected" : false
}, {
"name" : "dbname",
"aliases" : [ "database" ],
"label" : "Database Name",
"options" : [ ],
"kind" : "STRING",
"description" : "The name of the database to connect to.",
"hidden" : false,
"sensitive" : false,
"required" : "true",
"protected" : false
}, {
"name" : "default_target_schema",
"aliases" : [ ],
"label" : "Default Target Schema",
"value" : "analytics",
"options" : [ ],
"kind" : "STRING",
"description" : "The default schema to use when writing data to the Postgres Warehouse.",
"hidden" : false,
"sensitive" : false,
"required" : "true",
"protected" : false
}, {
"name" : "ssl",
"aliases" : [ ],
"label" : "SSL",
"value" : "false",
"options" : [ ],
"kind" : "BOOLEAN",
"description" : "Whether or not to use SSL encryption when connecting to the Postgres Warehouse.",
"hidden" : true,
"sensitive" : false,
"protected" : false,
"value_post_processor" : "STRINGIFY"
}, {
"name" : "batch_size_rows",
"aliases" : [ ],
"label" : "Batch Size Rows",
"value" : "100000",
"options" : [ ],
"kind" : "INTEGER",
"description" : "The number of rows to write to the Postgres Warehouse in each batch.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "underscore_camel_case_fields",
"aliases" : [ ],
"label" : "Underscore Camel Case Fields",
"value" : "true",
"options" : [ ],
"kind" : "BOOLEAN",
"description" : "Whether or not to convert field names from camel case to underscore-separated format.",
"hidden" : true,
"sensitive" : false,
"protected" : false
}, {
"name" : "flush_all_streams",
"aliases" : [ ],
"label" : "Flush All Streams",
"value" : "false",
"options" : [ ],
"kind" : "BOOLEAN",
"description" : "Whether or not to flush all streams to the Postgres Warehouse before closing the connection.",
"hidden" : true,
"sensitive" : false,
"protected" : false
}, {
"name" : "parallelism",
"aliases" : [ ],
"label" : "Parallelism",
"value" : "0",
"options" : [ ],
"kind" : "INTEGER",
"description" : "The number of threads to use when writing data to the Postgres Warehouse.",
"hidden" : true,
"sensitive" : false,
"protected" : false
}, {
"name" : "parallelism_max",
"aliases" : [ ],
"label" : "Max Parallelism",
"value" : "16",
"options" : [ ],
"kind" : "INTEGER",
"description" : "The maximum number of threads to use when writing data to the Postgres Warehouse.",
"hidden" : true,
"sensitive" : false,
"protected" : false
}, {
"name" : "default_target_schema_select_permission",
"aliases" : [ ],
"label" : "Default Target Schema Select Permission",
"options" : [ ],
"kind" : "STRING",
"description" : "The permission level required to select data from the default target schema.",
"hidden" : true,
"sensitive" : false,
"protected" : false
}, {
"name" : "schema_mapping",
"aliases" : [ ],
"label" : "Schema Mapping",
"options" : [ ],
"kind" : "STRING",
"description" : "A mapping of source schema names to target schema names.",
"hidden" : true,
"sensitive" : false,
"protected" : false
}, {
"name" : "add_metadata_columns",
"aliases" : [ ],
"label" : "Add Metadata Columns",
"value" : "true",
"options" : [ ],
"kind" : "BOOLEAN",
"description" : "Whether or not to add metadata columns to the target table.",
"hidden" : true,
"sensitive" : false,
"protected" : false
}, {
"name" : "hard_delete",
"aliases" : [ ],
"label" : "Hard Delete",
"value" : "false",
"options" : [ ],
"kind" : "BOOLEAN",
"description" : "Whether or not to perform hard deletes when deleting data from the Postgres Warehouse.",
"hidden" : true,
"sensitive" : false,
"protected" : false
}, {
"name" : "data_flattening_max_level",
"aliases" : [ ],
"label" : "Data Flattening Max Level",
"value" : "10",
"options" : [ ],
"kind" : "INTEGER",
"description" : "The maximum level of nested data structures to flatten when writing data to the Postgres Warehouse.",
"hidden" : true,
"sensitive" : false,
"protected" : false
}, {
"name" : "primary_key_required",
"aliases" : [ ],
"label" : "Primary Key Required",
"value" : "false",
"options" : [ ],
"kind" : "BOOLEAN",
"description" : "Whether or not a primary key is required for the target table.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "validate_records",
"aliases" : [ ],
"label" : "Validate Records",
"value" : "false",
"options" : [ ],
"kind" : "BOOLEAN",
"description" : "Whether or not to validate records before writing them to the Postgres Warehouse.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "temp_dir",
"aliases" : [ ],
"label" : "Temporary Directory",
"options" : [ ],
"kind" : "STRING",
"description" : "The directory to use for temporary files when writing data to the Postgres Warehouse.",
"hidden" : true,
"sensitive" : false,
"protected" : false
} ],
"variants" : [ ],
"commands" : { },
"matatikaHidden" : false,
"requires" : [ ],
"fullDescription" : "Postgres Warehouse is a data warehousing solution built on top of the Postgres database management system.\n\nPostgres Warehouse is designed to handle large volumes of data and complex queries, making it an ideal solution for businesses that need to store and analyze large amounts of data. It provides a number of features that are specifically tailored to data warehousing, such as columnar storage, parallel processing, and support for advanced analytics. Additionally, Postgres Warehouse is highly scalable, allowing businesses to easily add more resources as their data needs grow. Overall, Postgres Warehouse is a powerful and flexible data warehousing solution that can help businesses make better decisions by providing them with the insights they need to succeed.\n### Prerequisites\nThe process of obtaining the required settings for connecting to a Postgres Warehouse may vary depending on the specific setup and configuration of the database. However, here are some general ways to obtain each of the required settings:\n\n- User: The user is typically created when the database is set up. You can ask the database administrator or check the database documentation to find out the username.\n- Password: The password is also typically created when the database is set up. You can ask the database administrator or check the database documentation to find out the password.\n- Host: The host is the server where the database is located. You can ask the database administrator or check the database documentation to find out the host name or IP address.\n- Port: The port is the number that the database listens on for incoming connections. The default port for Postgres is 5432, but it may be different depending on the configuration. You can ask the database administrator or check the database documentation to find out the port number.\n- Database Name: The database name is the name of the specific database you want to connect to. You can ask the database administrator or check the database documentation to find out the database name.\n- Default Target Schema: The default target schema is the schema that you want to use as the default when connecting to the database. This may be set up by the database administrator or you may need to create it yourself. You can ask the database administrator or check the database documentation to find out the default target schema.\n\n## Settings\n\n\n### User\n\nThe username used to connect to the Postgres Warehouse.\n\n### Password\n\nThe password used to authenticate the user.\n\n### Host\n\nThe hostname or IP address of the Postgres Warehouse server.\n\n### Port\n\nThe port number used to connect to the Postgres Warehouse server.\n\n### Database Name\n\nThe name of the database to connect to.\n\n### Default Target Schema\n\nThe default schema to use when writing data to the Postgres Warehouse.\n\n### Batch Size Rows\n\nThe number of rows to write to the Postgres Warehouse in each batch.\n\n### Primary Key Required\n\nWhether or not a primary key is required for the target table.\n\n### Validate Records\n\nWhether or not to validate records before writing them to the Postgres Warehouse.",
"_links" : {
"self" : {
"href" : "https://app.matatika.com/api/dataplugins/73b171be-27dc-4b4b-8cea-fce2492eb1ee"
},
"update dataplugin" : {
"href" : "https://app.matatika.com/api/workspaces/8b97a7ce-165e-45d7-b3e9-e18777312f8f/dataplugins/73b171be-27dc-4b4b-8cea-fce2492eb1ee",
"type" : "PUT"
}
}
}
Setting
| Path | Type | Format | Description |
|---|---|---|---|
name |
String |
The setting name | |
value |
String |
The setting default value | |
label |
String |
The setting label | |
protected |
Boolean |
The setting protection status | |
kind |
String |
Setting Kind | The setting kind |
description |
String |
A description of the setting | |
placeholder |
String |
The setting placeholder text | |
envAliases |
Array of String |
Environment variable aliases for the setting | |
documentation |
String |
URL | The setting documentation URL |
oauth |
OAuth |
The setting OAuth configuration | |
env |
String |
OAuth
| Path | Type | Format | Description |
|---|---|---|---|
provider |
String |
The OAuth provider |
Formats
Setting Kind
String
| Value | Description |
|---|---|
STRING |
String setting |
INTEGER |
Integer setting |
PASSWORD |
Password setting |
HIDDEN |
Hidden setting |
BOOLEAN |
Boolean setting |
DATE_ISO8601 |
ISO 8601 date setting |
EMAIL |
Email setting |
OAUTH |
OAuth setting |
FILE |
File setting |
ARRAY |
Array setting |
Requests
- View all supported dataplugins
- View the Matatika
discovery.yml - View all workspace dataplugins
- View a workspace
discovery.yml - View a dataplugin
- Initialise a new dataplugin
- Publish dataplugins from a
discovery.yml - Create a dataplugin
- Update a dataplugin
- Delete a dataplugin
View all supported dataplugins
GET
/api/dataplugins
Returns all dataplugins supported by Matatika.
Request
Example Snippets
cURL
curl -H "Authorization: Bearer $ACCESS_TOKEN" 'https://app.matatika.com/api/dataplugins' -i -X GET \
-H 'Accept: application/json, application/javascript, text/javascript, text/json' \
-H 'Content-Type: application/json'
Python (requests)
import requests
url = "https://app.matatika.com/api/dataplugins"
headers = {
'Authorization': ACCESS_TOKEN
}
response = requests.request("GET", url, headers=headers)
print(response.text.encode('utf8'))
Response
200 OK
Dataplugin collection with HAL links.
{
"_embedded" : {
"dataplugins" : [ {
"id" : "1149bda6-c93f-4db6-a22c-f95afd60d575",
"pluginType" : "FILE",
"name" : "analyze-sit",
"namespace" : "tap_matatika_sit",
"variant" : "matatika",
"hidden" : false,
"pipUrl" : "git+https://github.com/Matatika/analyze-sit.git",
"repo" : "https://github.com/Matatika/analyze-sit",
"capabilities" : [ ],
"select" : [ ],
"update" : {
"analyze/datasets/tap-matatika-sit/user-ages.yml" : "true",
"analyze/datasets/tap-matatika-sit/user-genders.yml" : "true"
},
"vars" : { },
"settings" : [ ],
"variants" : [ ],
"commands" : { },
"matatikaHidden" : true,
"requires" : [ {
"id" : "931124c6-882f-4f0d-b0ca-6db09f1e1948",
"pluginType" : "EXTRACTOR",
"name" : "tap-matatika-sit",
"namespace" : "tap_matatika_sit",
"variant" : "matatika",
"label" : "Matatika SIT",
"description" : "Test extractor based on tap-spreadsheets-anywhere used during Matatika SIT runs",
"logoUrl" : "/assets/images/datasource/tap-matatika-sit.svg",
"hidden" : false,
"docs" : "https://meltano.com/plugins/extractors/spreadsheets-anywhere.html",
"pipUrl" : "git+https://github.com/ets/tap-spreadsheets-anywhere.git",
"repo" : "https://github.com/ets/tap-spreadsheets-anywhere",
"executable" : "tap-spreadsheets-anywhere",
"capabilities" : [ "DISCOVER", "CATALOG", "STATE" ],
"select" : [ ],
"update" : { },
"vars" : { },
"settings" : [ {
"name" : "tables",
"aliases" : [ ],
"label" : "Tables",
"value" : "[{\"path\":\"https://raw.githubusercontent.com/Matatika/matatika-examples/master/example_data\",\"name\":\"gitflixusers\",\"pattern\":\"GitFlixUsers.csv\",\"start_date\":\"2021-01-01T00:00:00Z\",\"key_properties\":[\"id\"],\"format\":\"csv\"}]",
"options" : [ ],
"kind" : "ARRAY",
"description" : "A setting in Matatika SIT that allows users to view and manage tables of data.",
"hidden" : false,
"sensitive" : false,
"protected" : false
} ],
"variants" : [ ],
"commands" : { },
"matatikaHidden" : true,
"requires" : [ ],
"fullDescription" : "Test extractor based on tap-spreadsheets-anywhere used during Matatika SIT runs\n\n## Settings\n\n\n### Tables\n\nA setting in Matatika SIT that allows users to view and manage tables of data."
} ],
"fullDescription" : "",
"_links" : {
"self" : {
"href" : "https://app.matatika.com/api/dataplugins/1149bda6-c93f-4db6-a22c-f95afd60d575"
}
}
}, {
"id" : "3d0d16b1-6b79-441b-987c-d9cc41ee6e73",
"pluginType" : "LOADER",
"name" : "target-redshift",
"namespace" : "target_redshift",
"variant" : "transferwise",
"label" : "Amazon Redshift",
"description" : "Amazon Redshift is a cloud-based data warehousing service. \n\nAmazon Redshift allows businesses to store and analyze large amounts of data in a cost-effective and scalable way. It can handle petabyte-scale data warehouses and offers fast query performance using SQL. It also integrates with other AWS services such as S3, EMR, and Kinesis. With Redshift, businesses can easily manage their data and gain insights to make informed decisions.",
"logoUrl" : "/assets/logos/loaders/redshift.png",
"hidden" : false,
"docs" : "https://www.matatika.com/data-details/target-redshift/",
"pipUrl" : "pipelinewise-target-redshift",
"repo" : "https://github.com/transferwise/pipelinewise-target-redshift",
"executable" : "target-redshift",
"capabilities" : [ "HARD_DELETE", "ACTIVATE_VERSION", "RECORD_FLATTENING", "SOFT_DELETE", "DATATYPE_FAILSAFE" ],
"select" : [ ],
"update" : { },
"vars" : { },
"settings" : [ {
"name" : "host",
"aliases" : [ ],
"label" : "Host",
"options" : [ ],
"kind" : "STRING",
"description" : "The endpoint URL for the Amazon Redshift cluster.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "port",
"aliases" : [ ],
"label" : "Port",
"value" : "5439",
"options" : [ ],
"kind" : "INTEGER",
"description" : "The port number on which the Amazon Redshift cluster is listening.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "dbname",
"aliases" : [ ],
"label" : "Database Name",
"options" : [ ],
"kind" : "STRING",
"description" : "The name of the Amazon Redshift database to connect to.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "user",
"aliases" : [ ],
"label" : "User name",
"options" : [ ],
"kind" : "STRING",
"description" : "The user name to use when connecting to the Amazon Redshift cluster.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "password",
"aliases" : [ ],
"label" : "Password",
"options" : [ ],
"kind" : "STRING",
"description" : "The password to use when connecting to the Amazon Redshift cluster.",
"hidden" : false,
"sensitive" : true,
"protected" : false
}, {
"name" : "s3_bucket",
"aliases" : [ ],
"label" : "S3 Bucket name",
"options" : [ ],
"kind" : "STRING",
"description" : "The name of the Amazon S3 bucket where the data to be loaded into Amazon Redshift is stored.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "default_target_schema",
"aliases" : [ ],
"label" : "Default Target Schema",
"value" : "$MELTANO_EXTRACT__LOAD_SCHEMA",
"options" : [ ],
"kind" : "STRING",
"description" : "The default schema to use when loading data into Amazon Redshift.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "aws_profile",
"aliases" : [ ],
"label" : "AWS Profile Name",
"options" : [ ],
"kind" : "STRING",
"description" : "The name of the AWS profile to use when connecting to Amazon Redshift.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "aws_access_key_id",
"aliases" : [ ],
"label" : "AWS S3 Access Key ID",
"options" : [ ],
"kind" : "STRING",
"description" : "The access key ID for the AWS account that owns the Amazon S3 bucket.",
"hidden" : false,
"sensitive" : true,
"protected" : false
}, {
"name" : "aws_secret_access_key",
"aliases" : [ ],
"label" : "AWS S3 Secret Access Key",
"options" : [ ],
"kind" : "STRING",
"description" : "The secret access key for the AWS account that owns the Amazon S3 bucket.",
"hidden" : false,
"sensitive" : true,
"protected" : false
}, {
"name" : "aws_session_token",
"aliases" : [ ],
"label" : "AWS S3 Session Token",
"options" : [ ],
"kind" : "STRING",
"description" : "The session token for the AWS account that owns the Amazon S3 bucket.",
"hidden" : false,
"sensitive" : true,
"protected" : false
}, {
"name" : "aws_redshift_copy_role_arn",
"aliases" : [ ],
"label" : "AWS Redshift COPY role ARN",
"options" : [ ],
"kind" : "STRING",
"description" : "The ARN of the AWS Identity and Access Management (IAM) role to use when loading data into Amazon Redshift.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "s3_acl",
"aliases" : [ ],
"label" : "AWS S3 ACL",
"options" : [ ],
"kind" : "STRING",
"description" : "The access control list (ACL) to apply to the Amazon S3 objects being loaded into Amazon Redshift.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "s3_key_prefix",
"aliases" : [ ],
"label" : "S3 Key Prefix",
"options" : [ ],
"kind" : "STRING",
"description" : "The prefix to apply to the Amazon S3 object keys being loaded into Amazon Redshift.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "copy_options",
"aliases" : [ ],
"label" : "COPY options",
"value" : "EMPTYASNULL BLANKSASNULL TRIMBLANKS TRUNCATECOLUMNS TIMEFORMAT 'auto' COMPUPDATE OFF STATUPDATE OFF",
"options" : [ ],
"kind" : "STRING",
"description" : "Additional options to use when loading data into Amazon Redshift.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "batch_size_rows",
"aliases" : [ ],
"label" : "Batch Size Rows",
"value" : "100000",
"options" : [ ],
"kind" : "INTEGER",
"description" : "The number of rows to load into Amazon Redshift at a time.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "flush_all_streams",
"aliases" : [ ],
"label" : "Flush All Streams",
"value" : "false",
"options" : [ ],
"kind" : "BOOLEAN",
"description" : "Whether to flush all streams to Amazon Redshift before disconnecting.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "parallelism",
"aliases" : [ ],
"label" : "Parallelism",
"value" : "0",
"options" : [ ],
"kind" : "INTEGER",
"description" : "The number of streams to use when loading data into Amazon Redshift.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "max_parallelism",
"aliases" : [ ],
"label" : "Max Parallelism",
"value" : "16",
"options" : [ ],
"kind" : "INTEGER",
"description" : "The maximum number of streams to use when loading data into Amazon Redshift.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "default_target_schema_select_permissions",
"aliases" : [ ],
"label" : "Default Target Schema Select Permission",
"options" : [ ],
"kind" : "STRING",
"description" : "The permission to use when selecting data from the default target schema.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "schema_mapping",
"aliases" : [ ],
"label" : "Scema Mapping",
"options" : [ ],
"kind" : "OBJECT",
"description" : "A mapping of source schema names to target schema names.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "disable_table_cache",
"aliases" : [ ],
"label" : "Disable Table Cache",
"value" : "false",
"options" : [ ],
"kind" : "BOOLEAN",
"description" : "Whether to disable the table cache when loading data into Amazon Redshift.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "add_metadata_columns",
"aliases" : [ ],
"label" : "Add Metdata Columns",
"value" : "false",
"options" : [ ],
"kind" : "BOOLEAN",
"description" : "Whether to add metadata columns to the Amazon Redshift table being loaded.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "hard_delete",
"aliases" : [ ],
"label" : "Hard Delete",
"value" : "false",
"options" : [ ],
"kind" : "BOOLEAN",
"description" : "Whether to perform a hard delete when deleting data from Amazon Redshift.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "data_flattening_max_level",
"aliases" : [ ],
"label" : "Data Flattening Max Level",
"value" : "0",
"options" : [ ],
"kind" : "INTEGER",
"description" : "The maximum level of data flattening to perform when loading data into Amazon Redshift.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "primary_key_required",
"aliases" : [ ],
"label" : "Primary Key Required",
"value" : "true",
"options" : [ ],
"kind" : "BOOLEAN",
"description" : "Whether a primary key is required when loading data into Amazon Redshift.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "validate_records",
"aliases" : [ ],
"label" : "Validate Records",
"value" : "false",
"options" : [ ],
"kind" : "BOOLEAN",
"description" : "Whether to validate records before loading them into Amazon Redshift.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "skip_updates",
"aliases" : [ ],
"label" : "Skip Updates",
"value" : "false",
"options" : [ ],
"kind" : "BOOLEAN",
"description" : "Whether to skip updates when loading data into Amazon Redshift.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "compression",
"aliases" : [ ],
"label" : "Compression",
"options" : [ {
"label" : "gzip",
"value" : "gzip"
}, {
"label" : "bzip2",
"value" : "bzip2"
}, {
"label" : "None",
"value" : ""
} ],
"kind" : "OPTIONS",
"description" : "The compression type to use when loading data into Amazon Redshift.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "slices",
"aliases" : [ ],
"label" : "Slices",
"value" : "1",
"options" : [ ],
"kind" : "INTEGER",
"description" : "The number of slices to use when loading data into Amazon Redshift.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "temp_dir",
"aliases" : [ ],
"label" : "Temp Directory",
"options" : [ ],
"kind" : "STRING",
"description" : "The directory to use for temporary files when loading data into Amazon Redshift.",
"hidden" : false,
"sensitive" : false,
"protected" : false
} ],
"variants" : [ ],
"commands" : { },
"matatikaHidden" : false,
"requires" : [ ],
"fullDescription" : "Amazon Redshift is a cloud-based data warehousing service. \n\nAmazon Redshift allows businesses to store and analyze large amounts of data in a cost-effective and scalable way. It can handle petabyte-scale data warehouses and offers fast query performance using SQL. It also integrates with other AWS services such as S3, EMR, and Kinesis. With Redshift, businesses can easily manage their data and gain insights to make informed decisions.\n\n## Settings\n\n\n### Host\n\nThe endpoint URL for the Amazon Redshift cluster.\n\n### Port\n\nThe port number on which the Amazon Redshift cluster is listening.\n\n### Database Name\n\nThe name of the Amazon Redshift database to connect to.\n\n### User name\n\nThe user name to use when connecting to the Amazon Redshift cluster.\n\n### Password\n\nThe password to use when connecting to the Amazon Redshift cluster.\n\n### S3 Bucket name\n\nThe name of the Amazon S3 bucket where the data to be loaded into Amazon Redshift is stored.\n\n### Default Target Schema\n\nThe default schema to use when loading data into Amazon Redshift.\n\n### AWS Profile Name\n\nThe name of the AWS profile to use when connecting to Amazon Redshift.\n\n### AWS S3 Access Key ID\n\nThe access key ID for the AWS account that owns the Amazon S3 bucket.\n\n### AWS S3 Secret Access Key\n\nThe secret access key for the AWS account that owns the Amazon S3 bucket.\n\n### AWS S3 Session Token\n\nThe session token for the AWS account that owns the Amazon S3 bucket.\n\n### AWS Redshift COPY role ARN\n\nThe ARN of the AWS Identity and Access Management (IAM) role to use when loading data into Amazon Redshift.\n\n### AWS S3 ACL\n\nThe access control list (ACL) to apply to the Amazon S3 objects being loaded into Amazon Redshift.\n\n### S3 Key Prefix\n\nThe prefix to apply to the Amazon S3 object keys being loaded into Amazon Redshift.\n\n### COPY options\n\nAdditional options to use when loading data into Amazon Redshift.\n\n### Batch Size Rows\n\nThe number of rows to load into Amazon Redshift at a time.\n\n### Flush All Streams\n\nWhether to flush all streams to Amazon Redshift before disconnecting.\n\n### Parallelism\n\nThe number of streams to use when loading data into Amazon Redshift.\n\n### Max Parallelism\n\nThe maximum number of streams to use when loading data into Amazon Redshift.\n\n### Default Target Schema Select Permission\n\nThe permission to use when selecting data from the default target schema.\n\n### Scema Mapping\n\nA mapping of source schema names to target schema names.\n\n### Disable Table Cache\n\nWhether to disable the table cache when loading data into Amazon Redshift.\n\n### Add Metdata Columns\n\nWhether to add metadata columns to the Amazon Redshift table being loaded.\n\n### Hard Delete\n\nWhether to perform a hard delete when deleting data from Amazon Redshift.\n\n### Data Flattening Max Level\n\nThe maximum level of data flattening to perform when loading data into Amazon Redshift.\n\n### Primary Key Required\n\nWhether a primary key is required when loading data into Amazon Redshift.\n\n### Validate Records\n\nWhether to validate records before loading them into Amazon Redshift.\n\n### Skip Updates\n\nWhether to skip updates when loading data into Amazon Redshift.\n\n### Compression\n\nThe compression type to use when loading data into Amazon Redshift.\n\n### Slices\n\nThe number of slices to use when loading data into Amazon Redshift.\n\n### Temp Directory\n\nThe directory to use for temporary files when loading data into Amazon Redshift.",
"_links" : {
"self" : {
"href" : "https://app.matatika.com/api/dataplugins/3d0d16b1-6b79-441b-987c-d9cc41ee6e73"
}
}
}, {
"id" : "6472b907-3f72-4456-9ce3-dd97236ba84f",
"pluginType" : "FILE",
"name" : "analyze-google-analytics",
"namespace" : "tap_google_analytics",
"variant" : "matatika",
"label" : "Google Analytics Insights",
"description" : "Instant insights on users, locations, sources, and sessions from Google Analytics.",
"hidden" : false,
"pipUrl" : "git+https://github.com/Matatika/[email protected]",
"repo" : "https://github.com/Matatika/analyze-google-analytics",
"capabilities" : [ ],
"select" : [ ],
"update" : {
"*.yml" : "true"
},
"vars" : { },
"settings" : [ ],
"variants" : [ ],
"commands" : { },
"matatikaHidden" : false,
"requires" : [ {
"id" : "a9ba6541-32a3-47ab-bb96-8c4aef3c4ab4",
"pluginType" : "TRANSFORM",
"name" : "dbt-google-analytics",
"namespace" : "tap_google_analytics",
"variant" : "matatika",
"hidden" : false,
"pipUrl" : "https://github.com/Matatika/[email protected]",
"repo" : "https://github.com/Matatika/dbt-tap-google-analytics",
"capabilities" : [ ],
"select" : [ ],
"update" : { },
"vars" : {
"schema" : ""
},
"settings" : [ ],
"variants" : [ ],
"commands" : { },
"matatikaHidden" : false,
"requires" : [ ],
"fullDescription" : ""
} ],
"fullDescription" : "Instant insights on users, locations, sources, and sessions from Google Analytics.",
"_links" : {
"self" : {
"href" : "https://app.matatika.com/api/dataplugins/6472b907-3f72-4456-9ce3-dd97236ba84f"
}
}
}, {
"id" : "2b48567d-5b9d-4018-9b6f-a9015963f53b",
"pluginType" : "LOADER",
"name" : "target-s3-avro",
"namespace" : "target_s3_avro",
"variant" : "faumel",
"label" : "S3 Avro",
"description" : "S3 Avro is a software tool for converting data between Avro and JSON formats in Amazon S3.\n\nS3 Avro is a software tool that allows users to easily convert data between Avro and JSON formats in Amazon S3. This tool is particularly useful for those who work with large amounts of data and need to quickly and efficiently convert between these two formats. With S3 Avro, users can easily upload Avro files to S3, convert them to JSON, and then download the converted files back to their local machine. This tool is designed to be user-friendly and intuitive, making it accessible to users of all skill levels.",
"logoUrl" : "/assets/logos/loaders/s3-avro.png",
"hidden" : false,
"docs" : "https://www.matatika.com/data-details/target-s3-avro/",
"pipUrl" : "git+https://github.com/faumel/target-s3-avro.git",
"repo" : "https://github.com/faumel/target-s3-avro",
"capabilities" : [ ],
"select" : [ ],
"update" : { },
"vars" : { },
"settings" : [ {
"name" : "verify",
"aliases" : [ ],
"label" : "Verify",
"options" : [ ],
"kind" : "BOOLEAN",
"description" : "Boolean value indicating whether to verify SSL certificates for HTTPS requests.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "aws_session_token",
"aliases" : [ ],
"label" : "Aws Session Token",
"options" : [ ],
"kind" : "STRING",
"description" : "Temporary session token for AWS authentication.",
"hidden" : false,
"sensitive" : true,
"protected" : false
}, {
"name" : "api_version",
"aliases" : [ ],
"label" : "Api Version",
"options" : [ ],
"kind" : "STRING",
"description" : "Version of the S3 Avro API to use.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "endpoint_url",
"aliases" : [ ],
"label" : "Endpoint Url",
"options" : [ ],
"kind" : "STRING",
"description" : "URL for the S3 Avro API endpoint.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "aws_secret_access_key",
"aliases" : [ ],
"label" : "Aws Secret Access Key",
"options" : [ ],
"kind" : "STRING",
"description" : "Secret access key for AWS authentication.",
"hidden" : false,
"sensitive" : true,
"protected" : false
}, {
"name" : "aws_access_key_id",
"aliases" : [ ],
"label" : "Aws Access Key Id",
"options" : [ ],
"kind" : "STRING",
"description" : "Access key ID for AWS authentication.",
"hidden" : false,
"sensitive" : true,
"protected" : false
}, {
"name" : "flatten_delimiter",
"aliases" : [ ],
"label" : "Flatten Delimiter",
"options" : [ ],
"kind" : "STRING",
"description" : "Delimiter to use when flattening nested Avro records.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "region_name",
"aliases" : [ ],
"label" : "Region Name",
"options" : [ ],
"kind" : "STRING",
"description" : "Name of the AWS region where the S3 bucket is located.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "tmp_dir",
"aliases" : [ ],
"label" : "Tmp Dir",
"options" : [ ],
"kind" : "STRING",
"description" : "Directory to use for temporary files during Avro serialization.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "use_ssl",
"aliases" : [ ],
"label" : "Use SSL",
"options" : [ ],
"kind" : "BOOLEAN",
"description" : "Boolean value indicating whether to use SSL for HTTPS requests.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "target_schema_bucket_key",
"aliases" : [ ],
"label" : "Target Schema Bucket Key",
"options" : [ ],
"kind" : "STRING",
"description" : "Key for the Avro schema file in the S3 bucket.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "config",
"aliases" : [ ],
"label" : "Config",
"options" : [ ],
"kind" : "STRING",
"description" : "Additional configuration options for the S3 Avro API connection.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "target_bucket_key",
"aliases" : [ ],
"label" : "Target Bucket Key",
"options" : [ ],
"kind" : "STRING",
"description" : "Key for the target object in the S3 bucket.",
"hidden" : false,
"sensitive" : false,
"protected" : false
} ],
"variants" : [ ],
"commands" : { },
"matatikaHidden" : false,
"requires" : [ ],
"fullDescription" : "S3 Avro is a software tool for converting data between Avro and JSON formats in Amazon S3.\n\nS3 Avro is a software tool that allows users to easily convert data between Avro and JSON formats in Amazon S3. This tool is particularly useful for those who work with large amounts of data and need to quickly and efficiently convert between these two formats. With S3 Avro, users can easily upload Avro files to S3, convert them to JSON, and then download the converted files back to their local machine. This tool is designed to be user-friendly and intuitive, making it accessible to users of all skill levels.\n\n## Settings\n\n\n### Verify\n\nBoolean value indicating whether to verify SSL certificates for HTTPS requests.\n\n### Aws Session Token\n\nTemporary session token for AWS authentication.\n\n### Api Version\n\nVersion of the S3 Avro API to use.\n\n### Endpoint Url\n\nURL for the S3 Avro API endpoint.\n\n### Aws Secret Access Key\n\nSecret access key for AWS authentication.\n\n### Aws Access Key Id\n\nAccess key ID for AWS authentication.\n\n### Flatten Delimiter\n\nDelimiter to use when flattening nested Avro records.\n\n### Region Name\n\nName of the AWS region where the S3 bucket is located.\n\n### Tmp Dir\n\nDirectory to use for temporary files during Avro serialization.\n\n### Use SSL\n\nBoolean value indicating whether to use SSL for HTTPS requests.\n\n### Target Schema Bucket Key\n\nKey for the Avro schema file in the S3 bucket.\n\n### Config\n\nAdditional configuration options for the S3 Avro API connection.\n\n### Target Bucket Key\n\nKey for the target object in the S3 bucket.",
"_links" : {
"self" : {
"href" : "https://app.matatika.com/api/dataplugins/2b48567d-5b9d-4018-9b6f-a9015963f53b"
}
}
}, {
"id" : "4f3acdb4-898b-4ddf-a70f-1141f7b73129",
"pluginType" : "TRANSFORM",
"name" : "dbt-solarvista",
"namespace" : "tap_solarvista",
"variant" : "matatika",
"hidden" : false,
"pipUrl" : "https://github.com/Matatika/[email protected]",
"repo" : "https://github.com/Matatika/dbt-tap-solarvista",
"capabilities" : [ ],
"select" : [ ],
"update" : { },
"vars" : {
"schema" : ""
},
"settings" : [ ],
"variants" : [ ],
"commands" : { },
"matatikaHidden" : false,
"requires" : [ {
"id" : "81ca6a43-b7bf-4e3d-b01f-7c9fff39b962",
"pluginType" : "TRANSFORMER",
"name" : "dbt",
"namespace" : "dbt",
"variant" : "dbt-labs",
"label" : "dbt",
"description" : " Power your project transformations with dbt™, a SQL-first transformation tool that enables analytics engineers to develop transformations with code.\n\n***Version Control and CI/CD***\n\nUse Matatika to deploy and promote changes between dev, UAT, and production environments.\n\n***Test and Document***\n\nUse Matatika to develop and test every model prior to production release, and share dynamically generated documentation with all stakeholders.\n\n***Develop***\n\nWrite modular data transformations in .sql – Matatika together with dbt handles the chore of dependency management. ",
"logoUrl" : "/assets/images/transformer/dbt.png",
"hidden" : false,
"docs" : "https://www.matatika.com/data-details/dbt/",
"pipUrl" : "dbt-core~=1.3.0 dbt-postgres~=1.3.0 dbt-snowflake~=1.3.0\n",
"repo" : "https://github.com/dbt-labs/dbt-core",
"capabilities" : [ ],
"select" : [ ],
"update" : { },
"vars" : { },
"settings" : [ {
"name" : "project_dir",
"aliases" : [ ],
"value" : "$MELTANO_PROJECT_ROOT/transform",
"options" : [ ],
"kind" : "STRING",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "profiles_dir",
"aliases" : [ ],
"value" : "$MELTANO_PROJECT_ROOT/transform/profile",
"options" : [ ],
"kind" : "STRING",
"hidden" : false,
"sensitive" : false,
"env" : "DBT_PROFILES_DIR",
"protected" : false
}, {
"name" : "target",
"aliases" : [ ],
"value" : "$MELTANO_LOAD__DIALECT",
"options" : [ ],
"kind" : "STRING",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "source_schema",
"aliases" : [ ],
"value" : "$MELTANO_LOAD__TARGET_SCHEMA",
"options" : [ ],
"kind" : "STRING",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "target_schema",
"aliases" : [ ],
"value" : "analytics",
"options" : [ ],
"kind" : "STRING",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "models",
"aliases" : [ ],
"value" : "$MELTANO_TRANSFORM__PACKAGE_NAME $MELTANO_EXTRACTOR_NAMESPACE my_meltano_project",
"options" : [ ],
"kind" : "STRING",
"hidden" : false,
"sensitive" : false,
"protected" : false
} ],
"variants" : [ ],
"commands" : {
"compile" : {
"args" : "compile",
"description" : "Generates executable SQL from source model, test, and analysis files. Compiled SQL files are written to the target/ directory."
},
"seed" : {
"args" : "seed",
"description" : "Load data from csv files into your data warehouse."
},
"test" : {
"args" : "test",
"description" : "Runs tests on data in deployed models."
},
"docs-generate" : {
"args" : "docs generate",
"description" : "Generate documentation artifacts for your project."
},
"deps" : {
"args" : "deps",
"description" : "Pull the most recent version of the dependencies listed in packages.yml"
},
"run" : {
"args" : "run",
"description" : "Compile SQL and execute against the current target database."
},
"clean" : {
"args" : "clean",
"description" : "Delete all folders in the clean-targets list (usually the dbt_modules and target directories.)"
},
"snapshot" : {
"args" : "snapshot",
"description" : "Execute snapshots defined in your project."
}
},
"matatikaHidden" : false,
"requires" : [ ],
"fullDescription" : " Power your project transformations with dbt™, a SQL-first transformation tool that enables analytics engineers to develop transformations with code.\n\n***Version Control and CI/CD***\n\nUse Matatika to deploy and promote changes between dev, UAT, and production environments.\n\n***Test and Document***\n\nUse Matatika to develop and test every model prior to production release, and share dynamically generated documentation with all stakeholders.\n\n***Develop***\n\nWrite modular data transformations in .sql – Matatika together with dbt handles the chore of dependency management. "
}, {
"id" : "33444aa0-a5e9-4edb-927a-d0c15707baa0",
"pluginType" : "EXTRACTOR",
"name" : "tap-solarvista",
"namespace" : "tap_solarvista",
"variant" : "matatika",
"label" : "Solarvista Live",
"description" : "Solarvista Live is a software platform for field service management.\n\nSolarvista Live is a cloud-based software platform designed to help businesses manage their field service operations more efficiently. It provides a range of tools and features to help businesses schedule and dispatch technicians, track work orders, manage inventory, and more. With Solarvista Live, businesses can streamline their field service operations, reduce costs, and improve customer satisfaction. The platform is highly customizable and can be tailored to meet the specific needs of each business. It is also designed to be easy to use, with a user-friendly interface that makes it simple for technicians and other field service personnel to access the information they need to do their jobs effectively. Overall, Solarvista Live is a powerful tool for businesses looking to optimize their field service operations and improve their bottom line.\n### Prerequisites\n- Datasources: The datasources required to connect to Solarvista Live are specific to the organization and must be provided by the Solarvista Live administrator or IT department.\n- Account: The account information required to connect to Solarvista Live is specific to the user and must be provided by the Solarvista Live administrator or IT department.\n- Client ID: The client ID required to connect to Solarvista Live is specific to the organization and must be provided by the Solarvista Live administrator or IT department.\n- Code: The code required to connect to Solarvista Live is specific to the user and must be provided by the Solarvista Live administrator or IT department.",
"logoUrl" : "/assets/images/datasource/tap-solarvista.png",
"hidden" : false,
"docs" : "https://www.matatika.com/docs/instant-insights/tap-solarvista/",
"pipUrl" : "git+https://github.com/Matatika/[email protected]",
"repo" : "https://github.com/Matatika/tap-solarvista",
"capabilities" : [ "STATE" ],
"select" : [ ],
"update" : { },
"vars" : { },
"settings" : [ {
"name" : "datasources",
"aliases" : [ ],
"label" : "Datasources",
"options" : [ ],
"kind" : "STRING",
"description" : "The data sources to connect to in Solarvista Live.",
"hidden" : false,
"sensitive" : false,
"required" : "true",
"protected" : false
}, {
"name" : "account",
"aliases" : [ ],
"label" : "Account",
"options" : [ ],
"kind" : "STRING",
"description" : "The account name to use for authentication.",
"hidden" : false,
"sensitive" : false,
"required" : "true",
"protected" : false
}, {
"name" : "clientId",
"aliases" : [ ],
"label" : "Client ID",
"options" : [ ],
"kind" : "STRING",
"description" : "The client ID to use for authentication.",
"hidden" : false,
"sensitive" : false,
"required" : "true",
"protected" : false
}, {
"name" : "code",
"aliases" : [ ],
"label" : "Code",
"options" : [ ],
"kind" : "STRING",
"description" : "The code to use for authentication.",
"hidden" : false,
"sensitive" : true,
"required" : "true",
"protected" : false
}, {
"name" : "start_date",
"aliases" : [ ],
"label" : "Start Date",
"options" : [ ],
"kind" : "DATE_ISO8601",
"description" : "The date to start retrieving data from.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "force_start_date",
"aliases" : [ ],
"label" : "Force Start Date",
"options" : [ ],
"kind" : "DATE_ISO8601",
"description" : "A flag indicating whether to force the start date even if data already exists for that date.",
"hidden" : false,
"sensitive" : false,
"protected" : false
} ],
"variants" : [ ],
"commands" : { },
"matatikaHidden" : false,
"requires" : [ ],
"fullDescription" : "Solarvista Live is a software platform for field service management.\n\nSolarvista Live is a cloud-based software platform designed to help businesses manage their field service operations more efficiently. It provides a range of tools and features to help businesses schedule and dispatch technicians, track work orders, manage inventory, and more. With Solarvista Live, businesses can streamline their field service operations, reduce costs, and improve customer satisfaction. The platform is highly customizable and can be tailored to meet the specific needs of each business. It is also designed to be easy to use, with a user-friendly interface that makes it simple for technicians and other field service personnel to access the information they need to do their jobs effectively. Overall, Solarvista Live is a powerful tool for businesses looking to optimize their field service operations and improve their bottom line.\n### Prerequisites\n- Datasources: The datasources required to connect to Solarvista Live are specific to the organization and must be provided by the Solarvista Live administrator or IT department.\n- Account: The account information required to connect to Solarvista Live is specific to the user and must be provided by the Solarvista Live administrator or IT department.\n- Client ID: The client ID required to connect to Solarvista Live is specific to the organization and must be provided by the Solarvista Live administrator or IT department.\n- Code: The code required to connect to Solarvista Live is specific to the user and must be provided by the Solarvista Live administrator or IT department.\n\n## Settings\n\n\n### Datasources\n\nThe data sources to connect to in Solarvista Live.\n\n### Account\n\nThe account name to use for authentication.\n\n### Client ID\n\nThe client ID to use for authentication.\n\n### Code\n\nThe code to use for authentication.\n\n### Start Date\n\nThe date to start retrieving data from.\n\n### Force Start Date\n\nA flag indicating whether to force the start date even if data already exists for that date."
} ],
"fullDescription" : "",
"_links" : {
"self" : {
"href" : "https://app.matatika.com/api/dataplugins/4f3acdb4-898b-4ddf-a70f-1141f7b73129"
}
}
}, {
"id" : "14518e68-ecda-48c9-9c93-155453d89ef2",
"pluginType" : "FILE",
"name" : "analyze-auth0",
"namespace" : "tap_auth0",
"variant" : "matatika",
"label" : "Auth0 Insights",
"description" : "Instant insights on users, logins and quotas from Auth0.",
"hidden" : false,
"pipUrl" : "git+https://github.com/Matatika/[email protected]",
"repo" : "https://github.com/Matatika/analyze-auth0",
"capabilities" : [ ],
"select" : [ ],
"update" : {
"*.yml" : "true"
},
"vars" : { },
"settings" : [ ],
"variants" : [ ],
"commands" : { },
"matatikaHidden" : false,
"requires" : [ {
"id" : "6c5a07d0-8580-4bf3-a56e-fb87f7c24c09",
"pluginType" : "EXTRACTOR",
"name" : "tap-auth0",
"namespace" : "tap_auth0",
"variant" : "matatika",
"label" : "Auth0",
"description" : "Auth0 is an identity and access management platform.\n\nAuth0 is a cloud-based platform that provides a comprehensive set of tools and services for managing user authentication and authorization in web and mobile applications. It allows developers to easily add authentication and authorization capabilities to their applications, without having to build and maintain their own identity management system. Auth0 supports a wide range of authentication methods, including social login, multi-factor authentication, and passwordless authentication. It also provides features such as user management, role-based access control, and integration with third-party identity providers. With Auth0, developers can focus on building their applications, while leaving the complex task of identity management to the experts.\n### Prerequisites\nTo obtain the Client ID, Client Secret, and Domain for connecting to Auth0, you need to follow these steps:\n\n1. Log in to your Auth0 account.\n2. From the dashboard, click on the \"Applications\" tab.\n3. Click on the \"Create Application\" button.\n4. Choose the type of application you want to create (Single Page Application, Regular Web Application, etc.).\n5. Give your application a name and click on the \"Create\" button.\n6. Once your application is created, you will be redirected to the \"Settings\" tab.\n7. Here, you will find the Client ID and Client Secret.\n8. To obtain the Domain, go to the \"Settings\" tab of your Auth0 account and copy the value of the \"Domain\" field.\n\nNote: The exact steps may vary slightly depending on the version of Auth0 you are using.",
"logoUrl" : "/assets/images/datasource/tap-auth0.png",
"hidden" : false,
"docs" : "https://www.matatika.com/docs/instant-insights/tap-auth0/",
"pipUrl" : "git+https://github.com/Matatika/[email protected]",
"repo" : "https://github.com/Matatika/tap-auth0",
"capabilities" : [ "DISCOVER", "CATALOG", "STATE" ],
"select" : [ ],
"update" : { },
"vars" : { },
"settings" : [ {
"name" : "client_id",
"aliases" : [ ],
"label" : "Client ID",
"options" : [ ],
"kind" : "STRING",
"description" : "A unique identifier for the client application that is registered with Auth0.",
"hidden" : false,
"sensitive" : true,
"required" : "true",
"protected" : false
}, {
"name" : "client_secret",
"aliases" : [ ],
"label" : "Client Secret",
"options" : [ ],
"kind" : "STRING",
"description" : "A secret string that is used to authenticate the client application with Auth0.",
"hidden" : false,
"sensitive" : true,
"required" : "true",
"protected" : false
}, {
"name" : "domain",
"aliases" : [ ],
"label" : "Domain",
"options" : [ ],
"kind" : "STRING",
"description" : "The Auth0 domain associated with the tenant.",
"hidden" : false,
"sensitive" : false,
"required" : "true",
"protected" : false
}, {
"name" : "job_poll_interval_ms",
"aliases" : [ ],
"label" : "Job poll interval ms",
"value" : "2000",
"options" : [ ],
"kind" : "INTEGER",
"description" : "The interval in milliseconds at which to poll for the status of a long-running job.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "job_poll_max_count",
"aliases" : [ ],
"label" : "Job poll max count",
"value" : "10",
"options" : [ ],
"kind" : "INTEGER",
"description" : "The maximum number of times to poll for the status of a long-running job.",
"hidden" : false,
"sensitive" : false,
"protected" : false
} ],
"variants" : [ ],
"commands" : { },
"matatikaHidden" : false,
"requires" : [ ],
"fullDescription" : "Auth0 is an identity and access management platform.\n\nAuth0 is a cloud-based platform that provides a comprehensive set of tools and services for managing user authentication and authorization in web and mobile applications. It allows developers to easily add authentication and authorization capabilities to their applications, without having to build and maintain their own identity management system. Auth0 supports a wide range of authentication methods, including social login, multi-factor authentication, and passwordless authentication. It also provides features such as user management, role-based access control, and integration with third-party identity providers. With Auth0, developers can focus on building their applications, while leaving the complex task of identity management to the experts.\n### Prerequisites\nTo obtain the Client ID, Client Secret, and Domain for connecting to Auth0, you need to follow these steps:\n\n1. Log in to your Auth0 account.\n2. From the dashboard, click on the \"Applications\" tab.\n3. Click on the \"Create Application\" button.\n4. Choose the type of application you want to create (Single Page Application, Regular Web Application, etc.).\n5. Give your application a name and click on the \"Create\" button.\n6. Once your application is created, you will be redirected to the \"Settings\" tab.\n7. Here, you will find the Client ID and Client Secret.\n8. To obtain the Domain, go to the \"Settings\" tab of your Auth0 account and copy the value of the \"Domain\" field.\n\nNote: The exact steps may vary slightly depending on the version of Auth0 you are using.\n\n## Settings\n\n\n### Client ID\n\nA unique identifier for the client application that is registered with Auth0.\n\n### Client Secret\n\nA secret string that is used to authenticate the client application with Auth0.\n\n### Domain\n\nThe Auth0 domain associated with the tenant.\n\n### Job poll interval ms\n\nThe interval in milliseconds at which to poll for the status of a long-running job.\n\n### Job poll max count\n\nThe maximum number of times to poll for the status of a long-running job."
} ],
"fullDescription" : "Instant insights on users, logins and quotas from Auth0.",
"_links" : {
"self" : {
"href" : "https://app.matatika.com/api/dataplugins/14518e68-ecda-48c9-9c93-155453d89ef2"
}
}
}, {
"id" : "e5e317b4-ddfe-4617-8228-966feeb124ed",
"pluginType" : "EXTRACTOR",
"name" : "tap-autopilot",
"namespace" : "tap_autopilot",
"variant" : "singer-io",
"label" : "Autopilot",
"description" : "Autopilot is a marketing automation software. \n\nAutopilot is a cloud-based marketing automation software that helps businesses automate their marketing tasks and workflows, such as lead generation, email marketing, and customer journey mapping, to improve customer engagement and drive revenue growth. It offers a visual canvas for creating personalized customer journeys, as well as integrations with popular CRM and marketing tools. Autopilot also provides analytics and reporting features to track campaign performance and optimize marketing strategies.",
"logoUrl" : "/assets/logos/extractors/autopilot.png",
"hidden" : false,
"docs" : "https://www.matatika.com/data-details/tap-autopilot/",
"pipUrl" : "tap-autopilot",
"repo" : "https://github.com/singer-io/tap-autopilot",
"capabilities" : [ "DISCOVER", "CATALOG", "STATE" ],
"select" : [ ],
"update" : { },
"vars" : { },
"settings" : [ ],
"variants" : [ ],
"commands" : { },
"matatikaHidden" : false,
"requires" : [ ],
"fullDescription" : "Autopilot is a marketing automation software. \n\nAutopilot is a cloud-based marketing automation software that helps businesses automate their marketing tasks and workflows, such as lead generation, email marketing, and customer journey mapping, to improve customer engagement and drive revenue growth. It offers a visual canvas for creating personalized customer journeys, as well as integrations with popular CRM and marketing tools. Autopilot also provides analytics and reporting features to track campaign performance and optimize marketing strategies.",
"_links" : {
"self" : {
"href" : "https://app.matatika.com/api/dataplugins/e5e317b4-ddfe-4617-8228-966feeb124ed"
}
}
}, {
"id" : "dbf87b80-6eb6-483a-90bb-b7a8c094fb3a",
"pluginType" : "FILE",
"name" : "analyze-solarvista",
"namespace" : "tap_solarvista",
"variant" : "matatika",
"label" : "Solarvista Insights",
"description" : "Instant insights on revenue, projects, work items, and engineer performance from Solarvista Live.",
"hidden" : false,
"pipUrl" : "git+https://github.com/Matatika/[email protected]",
"repo" : "https://github.com/Matatika/analyze-solarvista",
"capabilities" : [ ],
"select" : [ ],
"update" : {
"*.yml" : "true"
},
"vars" : { },
"settings" : [ ],
"variants" : [ ],
"commands" : { },
"matatikaHidden" : false,
"requires" : [ {
"id" : "4f3acdb4-898b-4ddf-a70f-1141f7b73129",
"pluginType" : "TRANSFORM",
"name" : "dbt-solarvista",
"namespace" : "tap_solarvista",
"variant" : "matatika",
"hidden" : false,
"pipUrl" : "https://github.com/Matatika/[email protected]",
"repo" : "https://github.com/Matatika/dbt-tap-solarvista",
"capabilities" : [ ],
"select" : [ ],
"update" : { },
"vars" : {
"schema" : ""
},
"settings" : [ ],
"variants" : [ ],
"commands" : { },
"matatikaHidden" : false,
"requires" : [ ],
"fullDescription" : ""
} ],
"fullDescription" : "Instant insights on revenue, projects, work items, and engineer performance from Solarvista Live.",
"_links" : {
"self" : {
"href" : "https://app.matatika.com/api/dataplugins/dbf87b80-6eb6-483a-90bb-b7a8c094fb3a"
}
}
}, {
"id" : "ffd26c88-aa25-4e04-913c-8dd0b22762d1",
"pluginType" : "FILE",
"name" : "analyze-trello",
"namespace" : "tap_trello",
"variant" : "matatika",
"label" : "Trello Insights",
"description" : "Instant insights on members, cards, boards, and actions from Trello.",
"hidden" : false,
"pipUrl" : "git+https://github.com/Matatika/[email protected]",
"repo" : "https://github.com/Matatika/analyze-trello",
"capabilities" : [ ],
"select" : [ ],
"update" : {
"*.yml" : "true"
},
"vars" : { },
"settings" : [ ],
"variants" : [ ],
"commands" : { },
"matatikaHidden" : false,
"requires" : [ {
"id" : "512c097b-df0e-4437-ba9a-3374557a30d9",
"pluginType" : "TRANSFORM",
"name" : "dbt-tap-trello",
"namespace" : "tap_trello",
"variant" : "matatika",
"hidden" : false,
"pipUrl" : "https://github.com/Matatika/[email protected]",
"repo" : "https://github.com/Matatika/dbt-tap-trello",
"capabilities" : [ ],
"select" : [ ],
"update" : { },
"vars" : {
"schema" : ""
},
"settings" : [ ],
"variants" : [ ],
"commands" : { },
"matatikaHidden" : false,
"requires" : [ ],
"fullDescription" : ""
} ],
"fullDescription" : "Instant insights on members, cards, boards, and actions from Trello.",
"_links" : {
"self" : {
"href" : "https://app.matatika.com/api/dataplugins/ffd26c88-aa25-4e04-913c-8dd0b22762d1"
}
}
}, {
"id" : "c5c84dde-1880-494d-95c4-7c71f43528f5",
"pluginType" : "EXTRACTOR",
"name" : "tap-aftership",
"namespace" : "tap_aftership",
"variant" : "harrystech",
"label" : "AfterShip",
"description" : "AfterShip is a shipment tracking platform for online retailers and customers.\n\nAfterShip allows online retailers to track and manage their shipments across multiple carriers and provides customers with real-time updates on the status of their orders. The platform integrates with over 700 carriers worldwide and offers features such as branded tracking pages, delivery notifications, and analytics to help businesses improve their shipping performance. AfterShip also offers a mobile app for customers to track their packages on-the-go.",
"logoUrl" : "/assets/logos/extractors/aftership.png",
"hidden" : false,
"docs" : "https://www.matatika.com/data-details/tap-aftership/",
"pipUrl" : "git+https://github.com/harrystech/tap-aftership.git",
"repo" : "https://github.com/harrystech/tap-aftership",
"capabilities" : [ "DISCOVER", "STREAM_MAPS", "CATALOG", "STATE", "SCHEMA_FLATTENING", "ABOUT" ],
"select" : [ ],
"update" : { },
"vars" : { },
"settings" : [ {
"name" : "api_key",
"aliases" : [ ],
"label" : "Api Key",
"options" : [ ],
"kind" : "STRING",
"description" : "A unique identifier used to authenticate and authorize API requests.",
"hidden" : false,
"sensitive" : true,
"protected" : false
}, {
"name" : "start_date",
"aliases" : [ ],
"label" : "Start Date",
"options" : [ ],
"kind" : "DATE_ISO8601",
"description" : "The earliest date for which shipment tracking information should be retrieved.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "end_date",
"aliases" : [ ],
"label" : "End Date",
"options" : [ ],
"kind" : "DATE_ISO8601",
"description" : "The latest date for which shipment tracking information should be retrieved.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "stream_maps",
"aliases" : [ ],
"label" : "Stream Maps",
"options" : [ ],
"kind" : "OBJECT",
"description" : "A list of stream maps that define the structure of the response data.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "stream_map_config",
"aliases" : [ ],
"label" : "Stream Map Config",
"options" : [ ],
"kind" : "OBJECT",
"description" : "Additional configuration settings for the stream maps.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "flattening_enabled",
"aliases" : [ ],
"label" : "Flattening Enabled",
"options" : [ ],
"kind" : "BOOLEAN",
"description" : "A boolean value indicating whether or not the response data should be flattened.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "flattening_max_depth",
"aliases" : [ ],
"label" : "Flattening Max Depth",
"options" : [ ],
"kind" : "INTEGER",
"description" : "The maximum depth to which the response data should be flattened.",
"hidden" : false,
"sensitive" : false,
"protected" : false
} ],
"variants" : [ ],
"commands" : { },
"matatikaHidden" : false,
"requires" : [ ],
"fullDescription" : "AfterShip is a shipment tracking platform for online retailers and customers.\n\nAfterShip allows online retailers to track and manage their shipments across multiple carriers and provides customers with real-time updates on the status of their orders. The platform integrates with over 700 carriers worldwide and offers features such as branded tracking pages, delivery notifications, and analytics to help businesses improve their shipping performance. AfterShip also offers a mobile app for customers to track their packages on-the-go.\n\n## Settings\n\n\n### Api Key\n\nA unique identifier used to authenticate and authorize API requests.\n\n### Start Date\n\nThe earliest date for which shipment tracking information should be retrieved.\n\n### End Date\n\nThe latest date for which shipment tracking information should be retrieved.\n\n### Stream Maps\n\nA list of stream maps that define the structure of the response data.\n\n### Stream Map Config\n\nAdditional configuration settings for the stream maps.\n\n### Flattening Enabled\n\nA boolean value indicating whether or not the response data should be flattened.\n\n### Flattening Max Depth\n\nThe maximum depth to which the response data should be flattened.",
"_links" : {
"self" : {
"href" : "https://app.matatika.com/api/dataplugins/c5c84dde-1880-494d-95c4-7c71f43528f5"
}
}
}, {
"id" : "c0598af4-f633-4d21-8f56-80a60aea9140",
"pluginType" : "LOADER",
"name" : "target-s3-csv",
"namespace" : "pipelinewise_target_s3_csv",
"variant" : "transferwise",
"label" : "S3 CSV",
"description" : "S3 CSV is a tool for managing CSV files in Amazon S3.\n\nS3 CSV is a software tool that allows users to easily manage CSV files stored in Amazon S3. It provides features such as importing, exporting, and transforming CSV files, as well as querying and filtering data. S3 CSV also offers advanced functionality such as data validation, data cleansing, and data enrichment. With S3 CSV, users can streamline their CSV file management processes and improve the accuracy and quality of their data.",
"logoUrl" : "/assets/logos/loaders/pipelinewise-s3-csv.png",
"hidden" : false,
"docs" : "https://www.matatika.com/data-details/target-s3-csv/",
"pipUrl" : "git+https://github.com/transferwise/pipelinewise-target-s3-csv.git",
"repo" : "https://github.com/transferwise/pipelinewise-target-s3-csv",
"capabilities" : [ ],
"select" : [ ],
"update" : { },
"vars" : { },
"settings" : [ {
"name" : "aws_access_key_id",
"aliases" : [ ],
"label" : "S3 Access Key Id",
"options" : [ ],
"kind" : "STRING",
"description" : "The access key ID for the AWS account.",
"hidden" : false,
"sensitive" : true,
"protected" : false
}, {
"name" : "aws_secret_access_key",
"aliases" : [ ],
"label" : "S3 Secret Access Key",
"options" : [ ],
"kind" : "STRING",
"description" : "The secret access key for the AWS account.",
"hidden" : false,
"sensitive" : true,
"protected" : false
}, {
"name" : "aws_session_token",
"aliases" : [ ],
"label" : "AWS Session token",
"options" : [ ],
"kind" : "STRING",
"description" : "The session token for the AWS account.",
"hidden" : false,
"sensitive" : true,
"protected" : false
}, {
"name" : "aws_endpoint_url",
"aliases" : [ ],
"label" : "AWS endpoint URL",
"options" : [ ],
"kind" : "STRING",
"description" : "The endpoint URL for the AWS service.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "aws_profile",
"aliases" : [ ],
"label" : "AWS profile",
"options" : [ ],
"kind" : "STRING",
"description" : "The name of the AWS profile to use.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "s3_bucket",
"aliases" : [ ],
"label" : "S3 Bucket name",
"options" : [ ],
"kind" : "STRING",
"description" : "The name of the S3 bucket to connect to.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "s3_key_prefix",
"aliases" : [ ],
"label" : "S3 Key Prefix",
"options" : [ ],
"kind" : "STRING",
"description" : "The prefix to use when searching for files in the S3 bucket.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "delimiter",
"aliases" : [ ],
"label" : "Delimiter",
"options" : [ ],
"kind" : "STRING",
"description" : "The delimiter used in the CSV file.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "quotechar",
"aliases" : [ ],
"label" : "Quote Char",
"options" : [ ],
"kind" : "STRING",
"description" : "The character used to quote fields in the CSV file.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "add_metadata_columns",
"aliases" : [ ],
"label" : "Add Metadata Columns",
"options" : [ ],
"kind" : "BOOLEAN",
"description" : "Whether or not to add metadata columns to the output.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "encryption_type",
"aliases" : [ ],
"label" : "S3 Access Key Id",
"options" : [ ],
"kind" : "STRING",
"description" : "The encryption key to use for the CSV file.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "encryption_key",
"aliases" : [ ],
"label" : "Encryption Key",
"options" : [ ],
"kind" : "STRING",
"description" : "The compression algorithm to use for the CSV file.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "compression",
"aliases" : [ ],
"label" : "Compression",
"options" : [ ],
"kind" : "STRING",
"description" : "The naming convention to use for the CSV file.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "naming_convention",
"aliases" : [ ],
"label" : "Naming Convention",
"options" : [ ],
"kind" : "STRING",
"description" : "(Default - None) Custom naming convention of the s3 key. Replaces tokens date, stream, and timestamp with the appropriate values. Supports \"folders\" in s3 keys e.g. folder/folder2/{stream}/export_date={date}/{timestamp}.csv. Honors the s3_key_prefix, if set, by prepending the \"filename\". E.g. naming_convention = folder1/my_file.csv and s3_key_prefix = prefix_ results in folder1/prefix_my_file.csv",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "temp_dir",
"aliases" : [ ],
"label" : "S3 Access Key Id",
"options" : [ ],
"kind" : "STRING",
"description" : "(Default - platform-dependent) Directory of temporary CSV files with RECORD messages.",
"hidden" : false,
"sensitive" : false,
"protected" : false
} ],
"variants" : [ ],
"commands" : { },
"matatikaHidden" : false,
"requires" : [ ],
"fullDescription" : "S3 CSV is a tool for managing CSV files in Amazon S3.\n\nS3 CSV is a software tool that allows users to easily manage CSV files stored in Amazon S3. It provides features such as importing, exporting, and transforming CSV files, as well as querying and filtering data. S3 CSV also offers advanced functionality such as data validation, data cleansing, and data enrichment. With S3 CSV, users can streamline their CSV file management processes and improve the accuracy and quality of their data.\n\n## Settings\n\n\n### S3 Access Key Id\n\nThe access key ID for the AWS account.\n\n### S3 Secret Access Key\n\nThe secret access key for the AWS account.\n\n### AWS Session token\n\nThe session token for the AWS account.\n\n### AWS endpoint URL\n\nThe endpoint URL for the AWS service.\n\n### AWS profile\n\nThe name of the AWS profile to use.\n\n### S3 Bucket name\n\nThe name of the S3 bucket to connect to.\n\n### S3 Key Prefix\n\nThe prefix to use when searching for files in the S3 bucket.\n\n### Delimiter\n\nThe delimiter used in the CSV file.\n\n### Quote Char\n\nThe character used to quote fields in the CSV file.\n\n### Add Metadata Columns\n\nWhether or not to add metadata columns to the output.\n\n### S3 Access Key Id\n\nThe encryption key to use for the CSV file.\n\n### Encryption Key\n\nThe compression algorithm to use for the CSV file.\n\n### Compression\n\nThe naming convention to use for the CSV file.\n\n### Naming Convention\n\n(Default - None) Custom naming convention of the s3 key. Replaces tokens date, stream, and timestamp with the appropriate values. Supports \"folders\" in s3 keys e.g. folder/folder2/{stream}/export_date={date}/{timestamp}.csv. Honors the s3_key_prefix, if set, by prepending the \"filename\". E.g. naming_convention = folder1/my_file.csv and s3_key_prefix = prefix_ results in folder1/prefix_my_file.csv\n\n### S3 Access Key Id\n\n(Default - platform-dependent) Directory of temporary CSV files with RECORD messages.",
"_links" : {
"self" : {
"href" : "https://app.matatika.com/api/dataplugins/c0598af4-f633-4d21-8f56-80a60aea9140"
}
}
}, {
"id" : "24072fe8-2f1f-4a0c-be4a-97df8c5e5be7",
"pluginType" : "LOADER",
"name" : "target-s3-parquet",
"namespace" : "target_s3_parquet",
"variant" : "gupy-io",
"label" : "S3 Parquet",
"description" : "S3 Parquet is a file format for storing and processing large amounts of data in a distributed computing environment.\n\nS3 Parquet is a columnar storage format that allows for efficient compression and encoding of data, making it ideal for storing and processing large amounts of data in a distributed computing environment. It is designed to work seamlessly with Amazon S3 and other big data processing tools such as Apache Spark and Hadoop. S3 Parquet allows for faster data processing and analysis, as well as reduced storage costs, making it a popular choice for big data applications.",
"logoUrl" : "/assets/logos/loaders/s3-parquet.png",
"hidden" : false,
"docs" : "https://www.matatika.com/data-details/target-s3-parquet/",
"pipUrl" : "git+https://github.com/gupy-io/target-s3-parquet.git",
"repo" : "https://github.com/gupy-io/target-s3-parquet",
"capabilities" : [ "STREAM_MAPS", "RECORD_FLATTENING", "ABOUT" ],
"select" : [ ],
"update" : { },
"vars" : { },
"settings" : [ {
"name" : "s3_path",
"aliases" : [ ],
"label" : "S3 Path",
"options" : [ ],
"kind" : "STRING",
"description" : "The path to the S3 bucket and object where the Parquet data is stored.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "aws_access_key_id",
"aliases" : [ ],
"label" : "AWS Access Key Id",
"options" : [ ],
"kind" : "STRING",
"description" : "The access key ID for the AWS account that has permission to access the S3 bucket.",
"hidden" : false,
"sensitive" : true,
"protected" : false
}, {
"name" : "aws_secret_access_key",
"aliases" : [ ],
"label" : "AWS Secret Access Key",
"options" : [ ],
"kind" : "STRING",
"description" : "The secret access key for the AWS account that has permission to access the S3 bucket.",
"hidden" : false,
"sensitive" : true,
"protected" : false
}, {
"name" : "athena_database",
"aliases" : [ ],
"label" : "Athena Database",
"options" : [ ],
"kind" : "STRING",
"description" : "The name of the Athena database where the Parquet data will be queried.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "add_record_metadata",
"aliases" : [ ],
"label" : "Add Record Metadata",
"options" : [ ],
"kind" : "BOOLEAN",
"description" : "Whether or not to add metadata to each record in the Parquet data.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "stringify_schema",
"aliases" : [ ],
"label" : "Stringify Schema",
"options" : [ ],
"kind" : "BOOLEAN",
"description" : "Whether or not to convert the schema of the Parquet data to a string format.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "stream_maps",
"aliases" : [ ],
"label" : "Stream Maps",
"options" : [ ],
"kind" : "OBJECT",
"description" : "A mapping of column names to stream names for the Parquet data.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "stream_map_config",
"aliases" : [ ],
"label" : "Stream Map Config",
"options" : [ ],
"kind" : "OBJECT",
"description" : "Configuration options for the stream maps.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "flattening_enabled",
"aliases" : [ ],
"label" : "Flattening Enabled",
"options" : [ ],
"kind" : "BOOLEAN",
"description" : "Whether or not to flatten nested structures in the Parquet data.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "flattening_max_depth",
"aliases" : [ ],
"label" : "Flattening Max Depth",
"options" : [ ],
"kind" : "INTEGER",
"description" : "The maximum depth to which nested structures will be flattened.",
"hidden" : false,
"sensitive" : false,
"protected" : false
} ],
"variants" : [ ],
"commands" : { },
"matatikaHidden" : false,
"requires" : [ ],
"fullDescription" : "S3 Parquet is a file format for storing and processing large amounts of data in a distributed computing environment.\n\nS3 Parquet is a columnar storage format that allows for efficient compression and encoding of data, making it ideal for storing and processing large amounts of data in a distributed computing environment. It is designed to work seamlessly with Amazon S3 and other big data processing tools such as Apache Spark and Hadoop. S3 Parquet allows for faster data processing and analysis, as well as reduced storage costs, making it a popular choice for big data applications.\n\n## Settings\n\n\n### S3 Path\n\nThe path to the S3 bucket and object where the Parquet data is stored.\n\n### AWS Access Key Id\n\nThe access key ID for the AWS account that has permission to access the S3 bucket.\n\n### AWS Secret Access Key\n\nThe secret access key for the AWS account that has permission to access the S3 bucket.\n\n### Athena Database\n\nThe name of the Athena database where the Parquet data will be queried.\n\n### Add Record Metadata\n\nWhether or not to add metadata to each record in the Parquet data.\n\n### Stringify Schema\n\nWhether or not to convert the schema of the Parquet data to a string format.\n\n### Stream Maps\n\nA mapping of column names to stream names for the Parquet data.\n\n### Stream Map Config\n\nConfiguration options for the stream maps.\n\n### Flattening Enabled\n\nWhether or not to flatten nested structures in the Parquet data.\n\n### Flattening Max Depth\n\nThe maximum depth to which nested structures will be flattened.",
"_links" : {
"self" : {
"href" : "https://app.matatika.com/api/dataplugins/24072fe8-2f1f-4a0c-be4a-97df8c5e5be7"
}
}
}, {
"id" : "931124c6-882f-4f0d-b0ca-6db09f1e1948",
"pluginType" : "EXTRACTOR",
"name" : "tap-matatika-sit",
"namespace" : "tap_matatika_sit",
"variant" : "matatika",
"label" : "Matatika SIT",
"description" : "Test extractor based on tap-spreadsheets-anywhere used during Matatika SIT runs",
"logoUrl" : "/assets/images/datasource/tap-matatika-sit.svg",
"hidden" : false,
"docs" : "https://meltano.com/plugins/extractors/spreadsheets-anywhere.html",
"pipUrl" : "git+https://github.com/ets/tap-spreadsheets-anywhere.git",
"repo" : "https://github.com/ets/tap-spreadsheets-anywhere",
"executable" : "tap-spreadsheets-anywhere",
"capabilities" : [ "DISCOVER", "CATALOG", "STATE" ],
"select" : [ ],
"update" : { },
"vars" : { },
"settings" : [ {
"name" : "tables",
"aliases" : [ ],
"label" : "Tables",
"value" : "[{\"path\":\"https://raw.githubusercontent.com/Matatika/matatika-examples/master/example_data\",\"name\":\"gitflixusers\",\"pattern\":\"GitFlixUsers.csv\",\"start_date\":\"2021-01-01T00:00:00Z\",\"key_properties\":[\"id\"],\"format\":\"csv\"}]",
"options" : [ ],
"kind" : "ARRAY",
"description" : "A setting in Matatika SIT that allows users to view and manage tables of data.",
"hidden" : false,
"sensitive" : false,
"protected" : false
} ],
"variants" : [ ],
"commands" : { },
"matatikaHidden" : true,
"requires" : [ ],
"fullDescription" : "Test extractor based on tap-spreadsheets-anywhere used during Matatika SIT runs\n\n## Settings\n\n\n### Tables\n\nA setting in Matatika SIT that allows users to view and manage tables of data.",
"_links" : {
"self" : {
"href" : "https://app.matatika.com/api/dataplugins/931124c6-882f-4f0d-b0ca-6db09f1e1948"
}
}
}, {
"id" : "4fa46eaa-9d17-42c1-9f59-8998bf10a71e",
"pluginType" : "EXTRACTOR",
"name" : "tap-anaplan",
"namespace" : "tap_anaplan",
"variant" : "matthew-skinner",
"label" : "Anaplan",
"description" : "Anaplan is a cloud-based platform for enterprise planning and performance management.\n\nAnaplan provides a centralized platform for businesses to plan, forecast, and analyze their financial and operational data in real-time. It allows users to create and customize models for budgeting, forecasting, sales planning, workforce planning, and more. Anaplan's platform is designed to be flexible and scalable, allowing businesses to adapt to changing market conditions and make data-driven decisions. It also offers collaboration tools, data visualization, and reporting capabilities to help teams work together more efficiently and effectively.",
"logoUrl" : "/assets/logos/extractors/anaplan.png",
"hidden" : false,
"docs" : "https://www.matatika.com/data-details/tap-anaplan/",
"pipUrl" : "git+https://github.com/matthew-skinner/tap-anaplan.git",
"repo" : "https://github.com/matthew-skinner/tap-anaplan",
"capabilities" : [ "DISCOVER", "CATALOG", "STATE" ],
"select" : [ ],
"update" : { },
"vars" : { },
"settings" : [ ],
"variants" : [ ],
"commands" : { },
"matatikaHidden" : false,
"requires" : [ ],
"fullDescription" : "Anaplan is a cloud-based platform for enterprise planning and performance management.\n\nAnaplan provides a centralized platform for businesses to plan, forecast, and analyze their financial and operational data in real-time. It allows users to create and customize models for budgeting, forecasting, sales planning, workforce planning, and more. Anaplan's platform is designed to be flexible and scalable, allowing businesses to adapt to changing market conditions and make data-driven decisions. It also offers collaboration tools, data visualization, and reporting capabilities to help teams work together more efficiently and effectively.",
"_links" : {
"self" : {
"href" : "https://app.matatika.com/api/dataplugins/4fa46eaa-9d17-42c1-9f59-8998bf10a71e"
}
}
}, {
"id" : "cb74863b-07d2-4b9a-912f-c7f8172ffc36",
"pluginType" : "LOADER",
"name" : "target-s3csv",
"namespace" : "pipelinewise_target_s3_csv",
"variant" : "transferwise",
"label" : "S3 CSV",
"description" : "S3 CSV is a file format used for storing data in Amazon S3.\n\nAmazon S3 is a cloud-based storage service that allows users to store and retrieve data from anywhere on the web. S3 CSV is a file format used for storing data in S3 that is organized in rows and columns, similar to a spreadsheet. This format is commonly used for storing large amounts of data that can be easily accessed and analyzed using various tools and applications. S3 CSV files can be easily imported and exported to other applications, making it a popular choice for data storage and analysis in the cloud.\n### Prerequisites\nTo obtain the AWS Access Key Id and AWS Secret Access Key, you need to go to the AWS Management Console, navigate to the IAM service, and create an IAM user with programmatic access. During the user creation process, you will be provided with the Access Key Id and Secret Access Key.\n\nTo obtain the S3 Bucket name, you need to navigate to the S3 service in the AWS Management Console and select the bucket that contains the CSV file you want to connect to. The name of the bucket will be displayed in the bucket details page.",
"logoUrl" : "/assets/logos/extractors/s3-csv.png",
"hidden" : false,
"docs" : "https://www.matatika.com/data-details/target-s3csv/",
"pipUrl" : "git+https://github.com/transferwise/pipelinewise-target-s3-csv.git",
"repo" : "https://github.com/transferwise/pipelinewise-target-s3-csv",
"capabilities" : [ ],
"select" : [ ],
"update" : { },
"vars" : { },
"settings" : [ {
"name" : "aws_access_key_id",
"aliases" : [ ],
"label" : "AWS Access Key Id",
"options" : [ ],
"kind" : "STRING",
"description" : "The access key ID for the AWS account.",
"hidden" : false,
"sensitive" : true,
"required" : "true",
"protected" : false
}, {
"name" : "aws_secret_access_key",
"aliases" : [ ],
"label" : "AWS Secret Access Key",
"options" : [ ],
"kind" : "STRING",
"description" : "The secret access key for the AWS account.",
"hidden" : false,
"sensitive" : true,
"required" : "true",
"protected" : false
}, {
"name" : "aws_session_token",
"aliases" : [ ],
"label" : "AWS Session token",
"options" : [ ],
"kind" : "STRING",
"description" : "The session token for the AWS account.",
"hidden" : false,
"sensitive" : true,
"protected" : false
}, {
"name" : "aws_endpoint_url",
"aliases" : [ ],
"label" : "AWS endpoint URL",
"options" : [ ],
"kind" : "STRING",
"description" : "The endpoint URL for the S3 bucket.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "aws_profile",
"aliases" : [ ],
"label" : "AWS profile",
"options" : [ ],
"kind" : "STRING",
"description" : "The name of the AWS profile to use.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "s3_bucket",
"aliases" : [ ],
"label" : "S3 Bucket name",
"options" : [ ],
"kind" : "STRING",
"description" : "The name of the S3 bucket to connect to.",
"hidden" : false,
"sensitive" : false,
"required" : "true",
"protected" : false
}, {
"name" : "s3_key_prefix",
"aliases" : [ ],
"label" : "S3 Key Prefix",
"options" : [ ],
"kind" : "STRING",
"description" : "The prefix for the S3 keys to read.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "delimiter",
"aliases" : [ ],
"label" : "delimiter",
"options" : [ ],
"kind" : "STRING",
"description" : "The delimiter used in the CSV file.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "quotechar",
"aliases" : [ ],
"label" : "Quote Char",
"options" : [ ],
"kind" : "STRING",
"description" : "The character used to quote fields in the CSV file.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "add_metadata_columns",
"aliases" : [ ],
"label" : "Add Metadata Columns",
"options" : [ ],
"kind" : "BOOLEAN",
"description" : "Whether to add metadata columns to the output.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "encryption_type",
"aliases" : [ ],
"label" : "Encryption Type",
"options" : [ ],
"kind" : "STRING",
"description" : "The type of encryption used for the S3 bucket.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "compression",
"aliases" : [ ],
"label" : "Compression",
"options" : [ ],
"kind" : "STRING",
"description" : "The compression type used for the CSV file.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "naming_convention",
"aliases" : [ ],
"label" : "Naming Convention",
"options" : [ ],
"kind" : "STRING",
"description" : "The naming convention used for the output files.",
"hidden" : false,
"sensitive" : false,
"protected" : false
} ],
"variants" : [ ],
"commands" : { },
"matatikaHidden" : false,
"requires" : [ ],
"fullDescription" : "S3 CSV is a file format used for storing data in Amazon S3.\n\nAmazon S3 is a cloud-based storage service that allows users to store and retrieve data from anywhere on the web. S3 CSV is a file format used for storing data in S3 that is organized in rows and columns, similar to a spreadsheet. This format is commonly used for storing large amounts of data that can be easily accessed and analyzed using various tools and applications. S3 CSV files can be easily imported and exported to other applications, making it a popular choice for data storage and analysis in the cloud.\n### Prerequisites\nTo obtain the AWS Access Key Id and AWS Secret Access Key, you need to go to the AWS Management Console, navigate to the IAM service, and create an IAM user with programmatic access. During the user creation process, you will be provided with the Access Key Id and Secret Access Key.\n\nTo obtain the S3 Bucket name, you need to navigate to the S3 service in the AWS Management Console and select the bucket that contains the CSV file you want to connect to. The name of the bucket will be displayed in the bucket details page.\n\n## Settings\n\n\n### AWS Access Key Id\n\nThe access key ID for the AWS account.\n\n### AWS Secret Access Key\n\nThe secret access key for the AWS account.\n\n### AWS Session token\n\nThe session token for the AWS account.\n\n### AWS endpoint URL\n\nThe endpoint URL for the S3 bucket.\n\n### AWS profile\n\nThe name of the AWS profile to use.\n\n### S3 Bucket name\n\nThe name of the S3 bucket to connect to.\n\n### S3 Key Prefix\n\nThe prefix for the S3 keys to read.\n\n### delimiter\n\nThe delimiter used in the CSV file.\n\n### Quote Char\n\nThe character used to quote fields in the CSV file.\n\n### Add Metadata Columns\n\nWhether to add metadata columns to the output.\n\n### Encryption Type\n\nThe type of encryption used for the S3 bucket.\n\n### Compression\n\nThe compression type used for the CSV file.\n\n### Naming Convention\n\nThe naming convention used for the output files.",
"_links" : {
"self" : {
"href" : "https://app.matatika.com/api/dataplugins/cb74863b-07d2-4b9a-912f-c7f8172ffc36"
}
}
}, {
"id" : "0879ca90-e5ba-49b9-8435-c68676133ac7",
"pluginType" : "FILE",
"name" : "analyze-meltano",
"namespace" : "tap_meltano",
"variant" : "matatika",
"label" : "Meltano Insights",
"description" : "Instant insights on jobs from Meltano.",
"hidden" : false,
"pipUrl" : "git+https://github.com/Matatika/[email protected]",
"repo" : "https://github.com/Matatika/analyze-meltano",
"capabilities" : [ ],
"select" : [ ],
"update" : {
"*.yml" : "true"
},
"vars" : { },
"settings" : [ ],
"variants" : [ ],
"commands" : { },
"matatikaHidden" : false,
"requires" : [ {
"id" : "8688dd6b-e9b9-48f9-b1ae-747ef53b071b",
"pluginType" : "TRANSFORM",
"name" : "dbt-meltano",
"namespace" : "tap_meltano",
"variant" : "matatika",
"description" : " Meltano is an open source project that manages data plugins and python virtual environments.\nMatatika extracts the data from Meltano and creates out of the box insights with this plugin.\n",
"hidden" : false,
"pipUrl" : "https://github.com/Matatika/[email protected]",
"repo" : "https://github.com/Matatika/dbt-tap-meltano",
"capabilities" : [ ],
"select" : [ ],
"update" : { },
"vars" : {
"schema" : ""
},
"settings" : [ ],
"variants" : [ ],
"commands" : { },
"matatikaHidden" : false,
"requires" : [ ],
"fullDescription" : " Meltano is an open source project that manages data plugins and python virtual environments.\nMatatika extracts the data from Meltano and creates out of the box insights with this plugin.\n"
} ],
"fullDescription" : "Instant insights on jobs from Meltano.",
"_links" : {
"self" : {
"href" : "https://app.matatika.com/api/dataplugins/0879ca90-e5ba-49b9-8435-c68676133ac7"
}
}
}, {
"id" : "9483915e-18cd-48a7-a804-ae0123da2931",
"pluginType" : "EXTRACTOR",
"name" : "tap-msaccess",
"namespace" : "tap_msaccess",
"variant" : "matatika",
"label" : "Microsoft Access",
"description" : "Database management system from Microsoft that combines the relational Access Database Engine with a graphical user interface and software-development tools",
"logoUrl" : "/assets/logos/extractors/msaccess.png",
"hidden" : false,
"docs" : "https://www.matatika.com/docs/instant-insights/tap-msaccess/",
"pipUrl" : "git+https://github.com/Matatika/[email protected]",
"repo" : "https://github.com/Matatika/tap-msaccess",
"capabilities" : [ "DISCOVER", "STREAM_MAPS", "CATALOG", "STATE", "SCHEMA_FLATTENING", "ABOUT", "BATCH" ],
"select" : [ ],
"update" : { },
"vars" : { },
"settings" : [ {
"name" : "database_file",
"aliases" : [ ],
"label" : "Database file",
"options" : [ ],
"kind" : "STRING",
"description" : "Local path or URL to a Microsoft Access database `.mdb` or `.accdb` file",
"hidden" : false,
"sensitive" : false,
"required" : "true",
"protected" : false
} ],
"variants" : [ ],
"commands" : { },
"matatikaHidden" : false,
"requires" : [ ],
"fullDescription" : "Database management system from Microsoft that combines the relational Access Database Engine with a graphical user interface and software-development tools\n\n## Settings\n\n\n### Database file\n\nLocal path or URL to a Microsoft Access database `.mdb` or `.accdb` file",
"_links" : {
"self" : {
"href" : "https://app.matatika.com/api/dataplugins/9483915e-18cd-48a7-a804-ae0123da2931"
}
}
}, {
"id" : "a9ba6541-32a3-47ab-bb96-8c4aef3c4ab4",
"pluginType" : "TRANSFORM",
"name" : "dbt-google-analytics",
"namespace" : "tap_google_analytics",
"variant" : "matatika",
"hidden" : false,
"pipUrl" : "https://github.com/Matatika/[email protected]",
"repo" : "https://github.com/Matatika/dbt-tap-google-analytics",
"capabilities" : [ ],
"select" : [ ],
"update" : { },
"vars" : {
"schema" : ""
},
"settings" : [ ],
"variants" : [ ],
"commands" : { },
"matatikaHidden" : false,
"requires" : [ {
"id" : "81ca6a43-b7bf-4e3d-b01f-7c9fff39b962",
"pluginType" : "TRANSFORMER",
"name" : "dbt",
"namespace" : "dbt",
"variant" : "dbt-labs",
"label" : "dbt",
"description" : " Power your project transformations with dbt™, a SQL-first transformation tool that enables analytics engineers to develop transformations with code.\n\n***Version Control and CI/CD***\n\nUse Matatika to deploy and promote changes between dev, UAT, and production environments.\n\n***Test and Document***\n\nUse Matatika to develop and test every model prior to production release, and share dynamically generated documentation with all stakeholders.\n\n***Develop***\n\nWrite modular data transformations in .sql – Matatika together with dbt handles the chore of dependency management. ",
"logoUrl" : "/assets/images/transformer/dbt.png",
"hidden" : false,
"docs" : "https://www.matatika.com/data-details/dbt/",
"pipUrl" : "dbt-core~=1.3.0 dbt-postgres~=1.3.0 dbt-snowflake~=1.3.0\n",
"repo" : "https://github.com/dbt-labs/dbt-core",
"capabilities" : [ ],
"select" : [ ],
"update" : { },
"vars" : { },
"settings" : [ {
"name" : "project_dir",
"aliases" : [ ],
"value" : "$MELTANO_PROJECT_ROOT/transform",
"options" : [ ],
"kind" : "STRING",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "profiles_dir",
"aliases" : [ ],
"value" : "$MELTANO_PROJECT_ROOT/transform/profile",
"options" : [ ],
"kind" : "STRING",
"hidden" : false,
"sensitive" : false,
"env" : "DBT_PROFILES_DIR",
"protected" : false
}, {
"name" : "target",
"aliases" : [ ],
"value" : "$MELTANO_LOAD__DIALECT",
"options" : [ ],
"kind" : "STRING",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "source_schema",
"aliases" : [ ],
"value" : "$MELTANO_LOAD__TARGET_SCHEMA",
"options" : [ ],
"kind" : "STRING",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "target_schema",
"aliases" : [ ],
"value" : "analytics",
"options" : [ ],
"kind" : "STRING",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "models",
"aliases" : [ ],
"value" : "$MELTANO_TRANSFORM__PACKAGE_NAME $MELTANO_EXTRACTOR_NAMESPACE my_meltano_project",
"options" : [ ],
"kind" : "STRING",
"hidden" : false,
"sensitive" : false,
"protected" : false
} ],
"variants" : [ ],
"commands" : {
"compile" : {
"args" : "compile",
"description" : "Generates executable SQL from source model, test, and analysis files. Compiled SQL files are written to the target/ directory."
},
"seed" : {
"args" : "seed",
"description" : "Load data from csv files into your data warehouse."
},
"test" : {
"args" : "test",
"description" : "Runs tests on data in deployed models."
},
"docs-generate" : {
"args" : "docs generate",
"description" : "Generate documentation artifacts for your project."
},
"deps" : {
"args" : "deps",
"description" : "Pull the most recent version of the dependencies listed in packages.yml"
},
"run" : {
"args" : "run",
"description" : "Compile SQL and execute against the current target database."
},
"clean" : {
"args" : "clean",
"description" : "Delete all folders in the clean-targets list (usually the dbt_modules and target directories.)"
},
"snapshot" : {
"args" : "snapshot",
"description" : "Execute snapshots defined in your project."
}
},
"matatikaHidden" : false,
"requires" : [ ],
"fullDescription" : " Power your project transformations with dbt™, a SQL-first transformation tool that enables analytics engineers to develop transformations with code.\n\n***Version Control and CI/CD***\n\nUse Matatika to deploy and promote changes between dev, UAT, and production environments.\n\n***Test and Document***\n\nUse Matatika to develop and test every model prior to production release, and share dynamically generated documentation with all stakeholders.\n\n***Develop***\n\nWrite modular data transformations in .sql – Matatika together with dbt handles the chore of dependency management. "
}, {
"id" : "51bcb7cb-13ab-4847-bda4-4db40bacf553",
"pluginType" : "EXTRACTOR",
"name" : "tap-google-analytics",
"namespace" : "tap_google_analytics",
"variant" : "matatika",
"label" : "Google Analytics",
"description" : "Google Analytics is a web analytics service that provides insights into website traffic and user behavior.\n\nGoogle Analytics allows website owners to track and analyze various metrics related to their website's performance, such as the number of visitors, pageviews, bounce rate, and average session duration. It also provides information on the demographics and interests of website visitors, as well as the sources of traffic, including organic search, paid search, social media, and referrals. This data can be used to optimize website content and marketing strategies, as well as to measure the effectiveness of advertising campaigns. Additionally, Google Analytics offers advanced features such as goal tracking, e-commerce tracking, and custom reporting, making it a powerful tool for businesses of all sizes.\n### Prerequisites\nTo obtain the OAuth identity provider authorization endpoint used to create and refresh tokens, you need to create a project in the Google API Console and enable the Google Analytics API. Then, you can create OAuth 2.0 credentials and configure the authorized redirect URIs. The authorization endpoint will be provided in the credentials.\n\nThe OAuth scopes you need to request access to depend on the specific data you want to access in Google Analytics. For example, if you want to read data from a specific view, you will need to request the \"https://www.googleapis.com/auth/analytics.readonly\" scope. You can find a list of available scopes in the Google Analytics API documentation.\n\nTo obtain the Access Token and OAuth Refresh Token, you need to authenticate the user and obtain their consent to access their Google Analytics data. This can be done using the Google Sign-In API or the OAuth 2.0 authorization flow. Once the user has granted access, you will receive an Access Token and a Refresh Token that you can use to make API requests.\n\nTo obtain the View ID, you need to log in to your Google Analytics account and navigate to the Admin section. From there, you can select the account, property, and view that you want to access and find the View ID in the View Settings.",
"logoUrl" : "/assets/images/datasource/tap-google-analytics.svg",
"hidden" : false,
"docs" : "https://www.matatika.com/docs/instant-insights/tap-google-analytics/",
"pipUrl" : "git+https://github.com/Matatika/[email protected]",
"repo" : "https://github.com/Matatika/tap-google-analytics",
"capabilities" : [ "DISCOVER", "CATALOG", "STATE" ],
"select" : [ ],
"update" : { },
"vars" : { },
"settings" : [ {
"name" : "oauth_credentials.authorization_url",
"aliases" : [ ],
"label" : "OAuth identity provider authorization endpoint used create and refresh tokens",
"value" : "https://oauth2.googleapis.com/token",
"options" : [ ],
"kind" : "STRING",
"description" : "The endpoint used to create and refresh OAuth tokens.",
"hidden" : true,
"sensitive" : false,
"required" : "true",
"protected" : false
}, {
"name" : "oauth_credentials.scope",
"aliases" : [ ],
"label" : "OAuth scopes we need to request access to",
"value" : "profile email https://www.googleapis.com/auth/analytics.readonly",
"options" : [ ],
"kind" : "STRING",
"description" : "The specific scopes we need to request access to in order to connect to Google Analytics.",
"hidden" : true,
"sensitive" : false,
"required" : "true",
"protected" : false
}, {
"name" : "oauth_credentials.access_token",
"aliases" : [ ],
"label" : "Access Token",
"options" : [ ],
"kind" : "STRING",
"description" : "The token used to authenticate and authorize API requests.",
"hidden" : true,
"sensitive" : false,
"required" : "true",
"protected" : false
}, {
"name" : "oauth_credentials.refresh_token",
"aliases" : [ ],
"label" : "OAuth Refresh Token",
"options" : [ ],
"kind" : "STRING",
"description" : "The token used to refresh the access token when it expires.",
"hidden" : true,
"sensitive" : false,
"required" : "true",
"protected" : false
}, {
"name" : "oauth_credentials.refresh_proxy_url",
"aliases" : [ ],
"label" : "Optional - will be called with 'oauth_credentials.refresh_token' to refresh the access token",
"options" : [ ],
"kind" : "STRING",
"description" : "An optional function that will be called to refresh the access token using the refresh token.",
"hidden" : true,
"sensitive" : false,
"protected" : false
}, {
"name" : "oauth_credentials.refresh_proxy_url_auth",
"aliases" : [ ],
"label" : "Optional - Sets Authorization header on 'oauth_credentials.refresh_url' request",
"options" : [ ],
"kind" : "STRING",
"description" : "An optional setting that sets the Authorization header on the refresh URL request.",
"hidden" : true,
"sensitive" : false,
"protected" : false
}, {
"name" : "oauth_credentials.client_id",
"aliases" : [ ],
"label" : "Optional - OAuth Client ID used if refresh_proxy_url not supplied",
"options" : [ ],
"kind" : "STRING",
"description" : "An optional OAuth Client ID used if the refresh proxy URL is not supplied.",
"hidden" : true,
"sensitive" : false,
"protected" : false
}, {
"name" : "oauth_credentials.client_secret",
"aliases" : [ ],
"label" : "Optional - OAuth Client Secret used if refresh_proxy_url not supplied",
"options" : [ ],
"kind" : "STRING",
"description" : "An optional OAuth Client Secret used if the refresh proxy URL is not supplied.",
"hidden" : true,
"sensitive" : false,
"protected" : false
}, {
"name" : "view_id",
"aliases" : [ ],
"label" : "View ID",
"options" : [ ],
"placeholder" : "Ex. 198343027",
"kind" : "STRING",
"description" : "The ID of the Google Analytics view to retrieve data from.",
"hidden" : false,
"sensitive" : false,
"required" : "true",
"protected" : false
}, {
"name" : "reports",
"aliases" : [ ],
"label" : "Reports",
"options" : [ ],
"placeholder" : "Ex. my_report_definition.json",
"kind" : "STRING",
"description" : "The specific reports to retrieve data from in the Google Analytics view.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "start_date",
"aliases" : [ ],
"label" : "Start date",
"options" : [ ],
"kind" : "DATE_ISO8601",
"description" : "The start date for the date range of data to retrieve.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "end_date",
"aliases" : [ ],
"label" : "End date",
"options" : [ ],
"kind" : "DATE_ISO8601",
"description" : "The end date for the date range of data to retrieve.",
"hidden" : false,
"sensitive" : false,
"protected" : false
} ],
"variants" : [ ],
"commands" : { },
"matatikaHidden" : false,
"requires" : [ ],
"fullDescription" : "Google Analytics is a web analytics service that provides insights into website traffic and user behavior.\n\nGoogle Analytics allows website owners to track and analyze various metrics related to their website's performance, such as the number of visitors, pageviews, bounce rate, and average session duration. It also provides information on the demographics and interests of website visitors, as well as the sources of traffic, including organic search, paid search, social media, and referrals. This data can be used to optimize website content and marketing strategies, as well as to measure the effectiveness of advertising campaigns. Additionally, Google Analytics offers advanced features such as goal tracking, e-commerce tracking, and custom reporting, making it a powerful tool for businesses of all sizes.\n### Prerequisites\nTo obtain the OAuth identity provider authorization endpoint used to create and refresh tokens, you need to create a project in the Google API Console and enable the Google Analytics API. Then, you can create OAuth 2.0 credentials and configure the authorized redirect URIs. The authorization endpoint will be provided in the credentials.\n\nThe OAuth scopes you need to request access to depend on the specific data you want to access in Google Analytics. For example, if you want to read data from a specific view, you will need to request the \"https://www.googleapis.com/auth/analytics.readonly\" scope. You can find a list of available scopes in the Google Analytics API documentation.\n\nTo obtain the Access Token and OAuth Refresh Token, you need to authenticate the user and obtain their consent to access their Google Analytics data. This can be done using the Google Sign-In API or the OAuth 2.0 authorization flow. Once the user has granted access, you will receive an Access Token and a Refresh Token that you can use to make API requests.\n\nTo obtain the View ID, you need to log in to your Google Analytics account and navigate to the Admin section. From there, you can select the account, property, and view that you want to access and find the View ID in the View Settings.\n\n## Settings\n\n\n### View ID\n\nThe ID of the Google Analytics view to retrieve data from.\n\n### Reports\n\nThe specific reports to retrieve data from in the Google Analytics view.\n\n### Start date\n\nThe start date for the date range of data to retrieve.\n\n### End date\n\nThe end date for the date range of data to retrieve."
} ],
"fullDescription" : "",
"_links" : {
"self" : {
"href" : "https://app.matatika.com/api/dataplugins/a9ba6541-32a3-47ab-bb96-8c4aef3c4ab4"
}
}
}, {
"id" : "bc91e7c0-6ade-43f3-987e-56083ce3f834",
"pluginType" : "EXTRACTOR",
"name" : "tap-anvil",
"namespace" : "tap_anvil",
"variant" : "svinstech",
"label" : "Anvil",
"description" : "Anvil is a web-based platform for building full-stack web apps with nothing but Python.\n\nAnvil allows users to build full-stack web applications using only Python code, without the need for front-end development skills or knowledge of HTML, CSS, or JavaScript. The platform provides a drag-and-drop interface for building user interfaces, as well as a built-in Python editor for writing server-side code. Anvil also includes a range of pre-built components and integrations, such as databases, authentication, and APIs, to help users build complex applications quickly and easily. With Anvil, developers can create web applications for a variety of use cases, from simple data entry forms to complex business applications.",
"logoUrl" : "/assets/logos/extractors/anvil.png",
"hidden" : false,
"docs" : "https://www.matatika.com/data-details/tap-anvil/",
"pipUrl" : "git+https://github.com/svinstech/tap-anvil.git",
"repo" : "https://github.com/svinstech/tap-anvil",
"capabilities" : [ "DISCOVER", "STREAM_MAPS", "CATALOG", "STATE", "SCHEMA_FLATTENING", "ABOUT" ],
"select" : [ ],
"update" : { },
"vars" : { },
"settings" : [ {
"name" : "api_key",
"aliases" : [ ],
"label" : "Api Key",
"options" : [ ],
"kind" : "STRING",
"description" : "A unique identifier used to authenticate and authorize API requests.",
"hidden" : false,
"sensitive" : true,
"protected" : false
}, {
"name" : "stream_maps",
"aliases" : [ ],
"label" : "Stream Maps",
"options" : [ ],
"kind" : "OBJECT",
"description" : "A mapping of input and output streams used to transform data.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "stream_map_config",
"aliases" : [ ],
"label" : "Stream Map Config",
"options" : [ ],
"kind" : "OBJECT",
"description" : "Configuration settings for the stream maps.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "flattening_enabled",
"aliases" : [ ],
"label" : "Flattening Enabled",
"options" : [ ],
"kind" : "BOOLEAN",
"description" : "A boolean value indicating whether or not to flatten nested data structures.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "flattening_max_depth",
"aliases" : [ ],
"label" : "Flattening Max Depth",
"options" : [ ],
"kind" : "INTEGER",
"description" : "The maximum depth of nested data structures to flatten.",
"hidden" : false,
"sensitive" : false,
"protected" : false
} ],
"variants" : [ ],
"commands" : { },
"matatikaHidden" : false,
"requires" : [ ],
"fullDescription" : "Anvil is a web-based platform for building full-stack web apps with nothing but Python.\n\nAnvil allows users to build full-stack web applications using only Python code, without the need for front-end development skills or knowledge of HTML, CSS, or JavaScript. The platform provides a drag-and-drop interface for building user interfaces, as well as a built-in Python editor for writing server-side code. Anvil also includes a range of pre-built components and integrations, such as databases, authentication, and APIs, to help users build complex applications quickly and easily. With Anvil, developers can create web applications for a variety of use cases, from simple data entry forms to complex business applications.\n\n## Settings\n\n\n### Api Key\n\nA unique identifier used to authenticate and authorize API requests.\n\n### Stream Maps\n\nA mapping of input and output streams used to transform data.\n\n### Stream Map Config\n\nConfiguration settings for the stream maps.\n\n### Flattening Enabled\n\nA boolean value indicating whether or not to flatten nested data structures.\n\n### Flattening Max Depth\n\nThe maximum depth of nested data structures to flatten.",
"_links" : {
"self" : {
"href" : "https://app.matatika.com/api/dataplugins/bc91e7c0-6ade-43f3-987e-56083ce3f834"
}
}
}, {
"id" : "53c122a6-8924-49d9-a9cb-11a2c0b11ebb",
"pluginType" : "TRANSFORM",
"name" : "dbt-googleads",
"namespace" : "tap_googleads",
"variant" : "matatika",
"description" : " Google Ads is an online advertising platform that allows businesses to create and display ads to potential customers.\nMatatika extract the data from the Google Ads API and creates out of the box insights including:\n\n***Google Ads campaigns***\n- Campaign Name\n- Campaign Status\n- Total Cost\n- Average CPC per Day\n- Total Clicks\n- Total Impressions\n- Popularity\n\n***Google Ads click locations***\nAnswer questions such as *What are my top 10 locations by clicks across all my campaigns?*\n\n***Google Ads campaign clicks***\nAnswer questions such as *How many clicks have all my campaigns had over the last 30 days?*\n\n",
"hidden" : false,
"pipUrl" : "https://github.com/Matatika/[email protected]",
"repo" : "https://github.com/Matatika/dbt-tap-googleads",
"capabilities" : [ ],
"select" : [ ],
"update" : { },
"vars" : {
"schema" : ""
},
"settings" : [ ],
"variants" : [ ],
"commands" : { },
"matatikaHidden" : false,
"requires" : [ {
"id" : "bdf19f6a-e898-49e6-bb59-8457b33907b1",
"pluginType" : "EXTRACTOR",
"name" : "tap-googleads",
"namespace" : "tap_googleads",
"variant" : "matatika",
"label" : "Google Ads",
"description" : "Google Ads is an online advertising platform that allows businesses to create and display ads to potential customers.\n\nGoogle Ads, formerly known as Google AdWords, is a pay-per-click (PPC) advertising platform that enables businesses to create and display ads to potential customers when they search for specific products or services on Google. Advertisers bid on specific keywords and pay for each click on their ads, with the cost per click (CPC) varying depending on the competition for the keyword. Google Ads also offers a range of targeting options, including location, demographics, and interests, allowing businesses to reach their ideal audience. Additionally, Google Ads provides detailed analytics and reporting, allowing advertisers to track the performance of their ads and make data-driven decisions to optimize their campaigns.\n### Prerequisites\nTo obtain the required settings for connecting to Google Ads, follow these steps:\n\n1. OAuth identity provider authorization endpoint used to create and refresh tokens: This endpoint is specific to the identity provider you are using. You can find this information in the documentation provided by the identity provider.\n\n2. OAuth scopes we need to request access to: The required OAuth scopes depend on the specific actions you want to perform in Google Ads. You can find a list of available scopes in the Google Ads API documentation.\n\n3. Access Token: To obtain an access token, you need to authenticate with Google using OAuth 2.0. Once you have authenticated, you will receive an access token that you can use to make API requests. You can find more information on how to obtain an access token in the Google Ads API documentation.\n\n4. OAuth Refresh Token: The refresh token is obtained during the initial authentication process and is used to obtain a new access token when the current one expires. You can find more information on how to obtain a refresh token in the Google Ads API documentation.\n\n5. Developer Token: The developer token is a unique identifier that is used to track API usage and ensure compliance with Google Ads policies. You can obtain a developer token by creating a Google Ads account and registering for the API.\n\n6. Customer Id: The customer ID is a unique identifier for each Google Ads account. You can find your customer ID in the Google Ads UI or by using the Google Ads API.",
"logoUrl" : "/assets/images/datasource/tap-googleads.svg",
"hidden" : false,
"docs" : "https://www.matatika.com/docs/instant-insights/tap-googleads/",
"pipUrl" : "git+https://github.com/Matatika/[email protected]",
"repo" : "https://github.com/Matatika/tap-googleads",
"capabilities" : [ "DISCOVER", "CATALOG", "STATE" ],
"select" : [ ],
"update" : { },
"vars" : { },
"settings" : [ {
"name" : "oauth_credentials.authorization_url",
"aliases" : [ ],
"label" : "OAuth identity provider authorization endpoint used create and refresh tokens",
"value" : "https://oauth2.googleapis.com/token",
"options" : [ ],
"kind" : "STRING",
"description" : "The endpoint used to create and refresh OAuth tokens.",
"hidden" : true,
"sensitive" : false,
"required" : "true",
"protected" : false
}, {
"name" : "oauth_credentials.scope",
"aliases" : [ ],
"label" : "OAuth scopes we need to request access to",
"value" : "https://www.googleapis.com/auth/adwords",
"options" : [ ],
"kind" : "STRING",
"description" : "The specific permissions we need to request access to in order to use the Google Ads API.",
"hidden" : true,
"sensitive" : false,
"required" : "true",
"protected" : false
}, {
"name" : "oauth_credentials.access_token",
"aliases" : [ ],
"label" : "Access Token",
"options" : [ ],
"kind" : "STRING",
"description" : "The token used to authenticate and authorize API requests.",
"hidden" : true,
"sensitive" : false,
"required" : "true",
"protected" : false
}, {
"name" : "oauth_credentials.refresh_token",
"aliases" : [ ],
"label" : "OAuth Refresh Token",
"options" : [ ],
"kind" : "STRING",
"description" : "The token used to refresh the access token when it expires.",
"hidden" : true,
"sensitive" : false,
"required" : "true",
"protected" : false
}, {
"name" : "oauth_credentials.refresh_proxy_url",
"aliases" : [ ],
"label" : "Optional - will be called with 'oauth_credentials.refresh_token' to refresh the access token",
"options" : [ ],
"kind" : "STRING",
"description" : "An optional function that will be called to refresh the access token using the refresh token.",
"hidden" : true,
"sensitive" : false,
"protected" : false
}, {
"name" : "oauth_credentials.refresh_proxy_url_auth",
"aliases" : [ ],
"label" : "Optional - Sets Authorization header on 'oauth_credentials.refresh_url' request",
"options" : [ ],
"kind" : "STRING",
"description" : "An optional setting that sets the Authorization header on the request to refresh the access token.",
"hidden" : true,
"sensitive" : false,
"protected" : false
}, {
"name" : "oauth_credentials.client_id",
"aliases" : [ ],
"label" : "Optional - OAuth Client ID used if refresh_proxy_url not supplied",
"options" : [ ],
"kind" : "STRING",
"description" : "An optional setting that specifies the OAuth Client ID to use if a refresh proxy URL is not supplied.",
"hidden" : true,
"sensitive" : false,
"protected" : false
}, {
"name" : "oauth_credentials.client_secret",
"aliases" : [ ],
"label" : "Optional - OAuth Client Secret used if refresh_proxy_url not supplied",
"options" : [ ],
"kind" : "STRING",
"description" : "An optional setting that specifies the OAuth Client Secret to use if a refresh proxy URL is not supplied.",
"hidden" : true,
"sensitive" : false,
"protected" : false
}, {
"name" : "start_date",
"aliases" : [ ],
"label" : "Start Date",
"options" : [ ],
"kind" : "DATE_ISO8601",
"description" : "The start date for the data range of the API request.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "end_date",
"aliases" : [ ],
"label" : "End Date",
"options" : [ ],
"kind" : "DATE_ISO8601",
"description" : "The end date for the data range of the API request.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "developer_token",
"aliases" : [ ],
"label" : "Developer Token",
"value" : "DYSuW0qdfU5-jti8Zdh1HQ",
"options" : [ ],
"kind" : "STRING",
"description" : "The token used to identify the developer making the API request.",
"hidden" : true,
"sensitive" : false,
"required" : "true",
"protected" : false
}, {
"name" : "customer_id",
"aliases" : [ ],
"label" : "Customer Id",
"options" : [ ],
"kind" : "STRING",
"description" : "The ID of the Google Ads account to make the API request on behalf of.",
"hidden" : false,
"sensitive" : false,
"required" : "true",
"protected" : false
} ],
"variants" : [ ],
"commands" : { },
"matatikaHidden" : false,
"requires" : [ ],
"fullDescription" : "Google Ads is an online advertising platform that allows businesses to create and display ads to potential customers.\n\nGoogle Ads, formerly known as Google AdWords, is a pay-per-click (PPC) advertising platform that enables businesses to create and display ads to potential customers when they search for specific products or services on Google. Advertisers bid on specific keywords and pay for each click on their ads, with the cost per click (CPC) varying depending on the competition for the keyword. Google Ads also offers a range of targeting options, including location, demographics, and interests, allowing businesses to reach their ideal audience. Additionally, Google Ads provides detailed analytics and reporting, allowing advertisers to track the performance of their ads and make data-driven decisions to optimize their campaigns.\n### Prerequisites\nTo obtain the required settings for connecting to Google Ads, follow these steps:\n\n1. OAuth identity provider authorization endpoint used to create and refresh tokens: This endpoint is specific to the identity provider you are using. You can find this information in the documentation provided by the identity provider.\n\n2. OAuth scopes we need to request access to: The required OAuth scopes depend on the specific actions you want to perform in Google Ads. You can find a list of available scopes in the Google Ads API documentation.\n\n3. Access Token: To obtain an access token, you need to authenticate with Google using OAuth 2.0. Once you have authenticated, you will receive an access token that you can use to make API requests. You can find more information on how to obtain an access token in the Google Ads API documentation.\n\n4. OAuth Refresh Token: The refresh token is obtained during the initial authentication process and is used to obtain a new access token when the current one expires. You can find more information on how to obtain a refresh token in the Google Ads API documentation.\n\n5. Developer Token: The developer token is a unique identifier that is used to track API usage and ensure compliance with Google Ads policies. You can obtain a developer token by creating a Google Ads account and registering for the API.\n\n6. Customer Id: The customer ID is a unique identifier for each Google Ads account. You can find your customer ID in the Google Ads UI or by using the Google Ads API.\n\n## Settings\n\n\n### Start Date\n\nThe start date for the data range of the API request.\n\n### End Date\n\nThe end date for the data range of the API request.\n\n### Customer Id\n\nThe ID of the Google Ads account to make the API request on behalf of."
}, {
"id" : "81ca6a43-b7bf-4e3d-b01f-7c9fff39b962",
"pluginType" : "TRANSFORMER",
"name" : "dbt",
"namespace" : "dbt",
"variant" : "dbt-labs",
"label" : "dbt",
"description" : " Power your project transformations with dbt™, a SQL-first transformation tool that enables analytics engineers to develop transformations with code.\n\n***Version Control and CI/CD***\n\nUse Matatika to deploy and promote changes between dev, UAT, and production environments.\n\n***Test and Document***\n\nUse Matatika to develop and test every model prior to production release, and share dynamically generated documentation with all stakeholders.\n\n***Develop***\n\nWrite modular data transformations in .sql – Matatika together with dbt handles the chore of dependency management. ",
"logoUrl" : "/assets/images/transformer/dbt.png",
"hidden" : false,
"docs" : "https://www.matatika.com/data-details/dbt/",
"pipUrl" : "dbt-core~=1.3.0 dbt-postgres~=1.3.0 dbt-snowflake~=1.3.0\n",
"repo" : "https://github.com/dbt-labs/dbt-core",
"capabilities" : [ ],
"select" : [ ],
"update" : { },
"vars" : { },
"settings" : [ {
"name" : "project_dir",
"aliases" : [ ],
"value" : "$MELTANO_PROJECT_ROOT/transform",
"options" : [ ],
"kind" : "STRING",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "profiles_dir",
"aliases" : [ ],
"value" : "$MELTANO_PROJECT_ROOT/transform/profile",
"options" : [ ],
"kind" : "STRING",
"hidden" : false,
"sensitive" : false,
"env" : "DBT_PROFILES_DIR",
"protected" : false
}, {
"name" : "target",
"aliases" : [ ],
"value" : "$MELTANO_LOAD__DIALECT",
"options" : [ ],
"kind" : "STRING",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "source_schema",
"aliases" : [ ],
"value" : "$MELTANO_LOAD__TARGET_SCHEMA",
"options" : [ ],
"kind" : "STRING",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "target_schema",
"aliases" : [ ],
"value" : "analytics",
"options" : [ ],
"kind" : "STRING",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "models",
"aliases" : [ ],
"value" : "$MELTANO_TRANSFORM__PACKAGE_NAME $MELTANO_EXTRACTOR_NAMESPACE my_meltano_project",
"options" : [ ],
"kind" : "STRING",
"hidden" : false,
"sensitive" : false,
"protected" : false
} ],
"variants" : [ ],
"commands" : {
"compile" : {
"args" : "compile",
"description" : "Generates executable SQL from source model, test, and analysis files. Compiled SQL files are written to the target/ directory."
},
"seed" : {
"args" : "seed",
"description" : "Load data from csv files into your data warehouse."
},
"test" : {
"args" : "test",
"description" : "Runs tests on data in deployed models."
},
"docs-generate" : {
"args" : "docs generate",
"description" : "Generate documentation artifacts for your project."
},
"deps" : {
"args" : "deps",
"description" : "Pull the most recent version of the dependencies listed in packages.yml"
},
"run" : {
"args" : "run",
"description" : "Compile SQL and execute against the current target database."
},
"clean" : {
"args" : "clean",
"description" : "Delete all folders in the clean-targets list (usually the dbt_modules and target directories.)"
},
"snapshot" : {
"args" : "snapshot",
"description" : "Execute snapshots defined in your project."
}
},
"matatikaHidden" : false,
"requires" : [ ],
"fullDescription" : " Power your project transformations with dbt™, a SQL-first transformation tool that enables analytics engineers to develop transformations with code.\n\n***Version Control and CI/CD***\n\nUse Matatika to deploy and promote changes between dev, UAT, and production environments.\n\n***Test and Document***\n\nUse Matatika to develop and test every model prior to production release, and share dynamically generated documentation with all stakeholders.\n\n***Develop***\n\nWrite modular data transformations in .sql – Matatika together with dbt handles the chore of dependency management. "
} ],
"fullDescription" : " Google Ads is an online advertising platform that allows businesses to create and display ads to potential customers.\nMatatika extract the data from the Google Ads API and creates out of the box insights including:\n\n***Google Ads campaigns***\n- Campaign Name\n- Campaign Status\n- Total Cost\n- Average CPC per Day\n- Total Clicks\n- Total Impressions\n- Popularity\n\n***Google Ads click locations***\nAnswer questions such as *What are my top 10 locations by clicks across all my campaigns?*\n\n***Google Ads campaign clicks***\nAnswer questions such as *How many clicks have all my campaigns had over the last 30 days?*\n\n",
"_links" : {
"self" : {
"href" : "https://app.matatika.com/api/dataplugins/53c122a6-8924-49d9-a9cb-11a2c0b11ebb"
}
}
} ]
},
"_links" : {
"first" : {
"href" : "https://app.matatika.com/api/dataplugins?page=0&size=20"
},
"self" : {
"href" : "https://app.matatika.com/api/dataplugins?page=0&size=20"
},
"next" : {
"href" : "https://app.matatika.com/api/dataplugins?page=1&size=20"
},
"last" : {
"href" : "https://app.matatika.com/api/dataplugins?page=26&size=20"
}
},
"page" : {
"size" : 20,
"totalElements" : 532,
"totalPages" : 27,
"number" : 0
}
}
View the Matatika discovery.yml
GET
/api/discovery.yml
Returns a Meltano discovery.yml containing all dataplugins supported by Matatika.
Request
Example Snippets
cURL
curl -H "Authorization: Bearer $ACCESS_TOKEN" 'https://app.matatika.com/api/discovery.yml' -i -X GET \
-H 'Accept: application/json, application/javascript, text/javascript, text/json' \
-H 'Content-Type: application/json'
Python (requests)
import requests
url = "https://app.matatika.com/api/discovery.yml"
headers = {
'Authorization': ACCESS_TOKEN
}
response = requests.request("GET", url, headers=headers)
print(response.text.encode('utf8'))
Response
200 OK
version: 20
extractors: []
loaders: []
transformers: []
files: []
utilities: []
View all workspace dataplugins
GET
/api/workspaces/{workspace-id}/dataplugins
Returns all dataplugins available to the workspace {workspace-id}.
Prerequisites
- Workspace
{workspace-id}must exist
Request
Example Snippets
cURL
curl -H "Authorization: Bearer $ACCESS_TOKEN" 'https://app.matatika.com/api/workspaces/8b97a7ce-165e-45d7-b3e9-e18777312f8f/dataplugins' -i -X GET \
-H 'Accept: application/json, application/javascript, text/javascript, text/json' \
-H 'Content-Type: application/json'
Python (requests)
import requests
url = "https://app.matatika.com/api/workspaces/8b97a7ce-165e-45d7-b3e9-e18777312f8f/dataplugins"
headers = {
'Authorization': ACCESS_TOKEN
}
response = requests.request("GET", url, headers=headers)
print(response.text.encode('utf8'))
Response
200 OK
Dataplugin collection with HAL links.
{
"_embedded" : {
"dataplugins" : [ {
"id" : "73b171be-27dc-4b4b-8cea-fce2492eb1ee",
"pluginType" : "LOADER",
"name" : "target-postgres",
"namespace" : "postgres_transferwise",
"variant" : "matatika",
"label" : "Postgres Warehouse",
"description" : "Postgres Warehouse is a data warehousing solution built on top of the Postgres database management system.\n\nPostgres Warehouse is designed to handle large volumes of data and complex queries, making it an ideal solution for businesses that need to store and analyze large amounts of data. It provides a number of features that are specifically tailored to data warehousing, such as columnar storage, parallel processing, and support for advanced analytics. Additionally, Postgres Warehouse is highly scalable, allowing businesses to easily add more resources as their data needs grow. Overall, Postgres Warehouse is a powerful and flexible data warehousing solution that can help businesses make better decisions by providing them with the insights they need to succeed.\n### Prerequisites\nThe process of obtaining the required settings for connecting to a Postgres Warehouse may vary depending on the specific setup and configuration of the database. However, here are some general ways to obtain each of the required settings:\n\n- User: The user is typically created when the database is set up. You can ask the database administrator or check the database documentation to find out the username.\n- Password: The password is also typically created when the database is set up. You can ask the database administrator or check the database documentation to find out the password.\n- Host: The host is the server where the database is located. You can ask the database administrator or check the database documentation to find out the host name or IP address.\n- Port: The port is the number that the database listens on for incoming connections. The default port for Postgres is 5432, but it may be different depending on the configuration. You can ask the database administrator or check the database documentation to find out the port number.\n- Database Name: The database name is the name of the specific database you want to connect to. You can ask the database administrator or check the database documentation to find out the database name.\n- Default Target Schema: The default target schema is the schema that you want to use as the default when connecting to the database. This may be set up by the database administrator or you may need to create it yourself. You can ask the database administrator or check the database documentation to find out the default target schema.",
"logoUrl" : "/assets/logos/loaders/postgres.png",
"hidden" : false,
"docs" : "https://www.matatika.com/data-details/target-postgres/",
"pipUrl" : "git+https://github.com/Matatika/[email protected]",
"repo" : "git+https://github.com/Matatika/[email protected]",
"capabilities" : [ ],
"select" : [ ],
"update" : { },
"vars" : { },
"settings" : [ {
"name" : "user",
"aliases" : [ "username" ],
"label" : "User",
"options" : [ ],
"kind" : "STRING",
"description" : "The username used to connect to the Postgres Warehouse.",
"hidden" : false,
"sensitive" : false,
"required" : "true",
"protected" : false
}, {
"name" : "password",
"aliases" : [ ],
"label" : "Password",
"options" : [ ],
"kind" : "STRING",
"description" : "The password used to authenticate the user.",
"hidden" : false,
"sensitive" : true,
"required" : "true",
"protected" : false
}, {
"name" : "host",
"aliases" : [ "address" ],
"label" : "Host",
"options" : [ ],
"kind" : "STRING",
"description" : "The hostname or IP address of the Postgres Warehouse server.",
"hidden" : false,
"sensitive" : false,
"required" : "true",
"protected" : false
}, {
"name" : "port",
"aliases" : [ ],
"label" : "Port",
"value" : "5432",
"options" : [ ],
"kind" : "INTEGER",
"description" : "The port number used to connect to the Postgres Warehouse server.",
"hidden" : false,
"sensitive" : false,
"required" : "true",
"protected" : false
}, {
"name" : "dbname",
"aliases" : [ "database" ],
"label" : "Database Name",
"options" : [ ],
"kind" : "STRING",
"description" : "The name of the database to connect to.",
"hidden" : false,
"sensitive" : false,
"required" : "true",
"protected" : false
}, {
"name" : "default_target_schema",
"aliases" : [ ],
"label" : "Default Target Schema",
"value" : "analytics",
"options" : [ ],
"kind" : "STRING",
"description" : "The default schema to use when writing data to the Postgres Warehouse.",
"hidden" : false,
"sensitive" : false,
"required" : "true",
"protected" : false
}, {
"name" : "ssl",
"aliases" : [ ],
"label" : "SSL",
"value" : "false",
"options" : [ ],
"kind" : "BOOLEAN",
"description" : "Whether or not to use SSL encryption when connecting to the Postgres Warehouse.",
"hidden" : true,
"sensitive" : false,
"protected" : false,
"value_post_processor" : "STRINGIFY"
}, {
"name" : "batch_size_rows",
"aliases" : [ ],
"label" : "Batch Size Rows",
"value" : "100000",
"options" : [ ],
"kind" : "INTEGER",
"description" : "The number of rows to write to the Postgres Warehouse in each batch.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "underscore_camel_case_fields",
"aliases" : [ ],
"label" : "Underscore Camel Case Fields",
"value" : "true",
"options" : [ ],
"kind" : "BOOLEAN",
"description" : "Whether or not to convert field names from camel case to underscore-separated format.",
"hidden" : true,
"sensitive" : false,
"protected" : false
}, {
"name" : "flush_all_streams",
"aliases" : [ ],
"label" : "Flush All Streams",
"value" : "false",
"options" : [ ],
"kind" : "BOOLEAN",
"description" : "Whether or not to flush all streams to the Postgres Warehouse before closing the connection.",
"hidden" : true,
"sensitive" : false,
"protected" : false
}, {
"name" : "parallelism",
"aliases" : [ ],
"label" : "Parallelism",
"value" : "0",
"options" : [ ],
"kind" : "INTEGER",
"description" : "The number of threads to use when writing data to the Postgres Warehouse.",
"hidden" : true,
"sensitive" : false,
"protected" : false
}, {
"name" : "parallelism_max",
"aliases" : [ ],
"label" : "Max Parallelism",
"value" : "16",
"options" : [ ],
"kind" : "INTEGER",
"description" : "The maximum number of threads to use when writing data to the Postgres Warehouse.",
"hidden" : true,
"sensitive" : false,
"protected" : false
}, {
"name" : "default_target_schema_select_permission",
"aliases" : [ ],
"label" : "Default Target Schema Select Permission",
"options" : [ ],
"kind" : "STRING",
"description" : "The permission level required to select data from the default target schema.",
"hidden" : true,
"sensitive" : false,
"protected" : false
}, {
"name" : "schema_mapping",
"aliases" : [ ],
"label" : "Schema Mapping",
"options" : [ ],
"kind" : "STRING",
"description" : "A mapping of source schema names to target schema names.",
"hidden" : true,
"sensitive" : false,
"protected" : false
}, {
"name" : "add_metadata_columns",
"aliases" : [ ],
"label" : "Add Metadata Columns",
"value" : "true",
"options" : [ ],
"kind" : "BOOLEAN",
"description" : "Whether or not to add metadata columns to the target table.",
"hidden" : true,
"sensitive" : false,
"protected" : false
}, {
"name" : "hard_delete",
"aliases" : [ ],
"label" : "Hard Delete",
"value" : "false",
"options" : [ ],
"kind" : "BOOLEAN",
"description" : "Whether or not to perform hard deletes when deleting data from the Postgres Warehouse.",
"hidden" : true,
"sensitive" : false,
"protected" : false
}, {
"name" : "data_flattening_max_level",
"aliases" : [ ],
"label" : "Data Flattening Max Level",
"value" : "10",
"options" : [ ],
"kind" : "INTEGER",
"description" : "The maximum level of nested data structures to flatten when writing data to the Postgres Warehouse.",
"hidden" : true,
"sensitive" : false,
"protected" : false
}, {
"name" : "primary_key_required",
"aliases" : [ ],
"label" : "Primary Key Required",
"value" : "false",
"options" : [ ],
"kind" : "BOOLEAN",
"description" : "Whether or not a primary key is required for the target table.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "validate_records",
"aliases" : [ ],
"label" : "Validate Records",
"value" : "false",
"options" : [ ],
"kind" : "BOOLEAN",
"description" : "Whether or not to validate records before writing them to the Postgres Warehouse.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "temp_dir",
"aliases" : [ ],
"label" : "Temporary Directory",
"options" : [ ],
"kind" : "STRING",
"description" : "The directory to use for temporary files when writing data to the Postgres Warehouse.",
"hidden" : true,
"sensitive" : false,
"protected" : false
} ],
"variants" : [ ],
"commands" : { },
"matatikaHidden" : false,
"requires" : [ ],
"fullDescription" : "Postgres Warehouse is a data warehousing solution built on top of the Postgres database management system.\n\nPostgres Warehouse is designed to handle large volumes of data and complex queries, making it an ideal solution for businesses that need to store and analyze large amounts of data. It provides a number of features that are specifically tailored to data warehousing, such as columnar storage, parallel processing, and support for advanced analytics. Additionally, Postgres Warehouse is highly scalable, allowing businesses to easily add more resources as their data needs grow. Overall, Postgres Warehouse is a powerful and flexible data warehousing solution that can help businesses make better decisions by providing them with the insights they need to succeed.\n### Prerequisites\nThe process of obtaining the required settings for connecting to a Postgres Warehouse may vary depending on the specific setup and configuration of the database. However, here are some general ways to obtain each of the required settings:\n\n- User: The user is typically created when the database is set up. You can ask the database administrator or check the database documentation to find out the username.\n- Password: The password is also typically created when the database is set up. You can ask the database administrator or check the database documentation to find out the password.\n- Host: The host is the server where the database is located. You can ask the database administrator or check the database documentation to find out the host name or IP address.\n- Port: The port is the number that the database listens on for incoming connections. The default port for Postgres is 5432, but it may be different depending on the configuration. You can ask the database administrator or check the database documentation to find out the port number.\n- Database Name: The database name is the name of the specific database you want to connect to. You can ask the database administrator or check the database documentation to find out the database name.\n- Default Target Schema: The default target schema is the schema that you want to use as the default when connecting to the database. This may be set up by the database administrator or you may need to create it yourself. You can ask the database administrator or check the database documentation to find out the default target schema.\n\n## Settings\n\n\n### User\n\nThe username used to connect to the Postgres Warehouse.\n\n### Password\n\nThe password used to authenticate the user.\n\n### Host\n\nThe hostname or IP address of the Postgres Warehouse server.\n\n### Port\n\nThe port number used to connect to the Postgres Warehouse server.\n\n### Database Name\n\nThe name of the database to connect to.\n\n### Default Target Schema\n\nThe default schema to use when writing data to the Postgres Warehouse.\n\n### Batch Size Rows\n\nThe number of rows to write to the Postgres Warehouse in each batch.\n\n### Primary Key Required\n\nWhether or not a primary key is required for the target table.\n\n### Validate Records\n\nWhether or not to validate records before writing them to the Postgres Warehouse.",
"_links" : {
"self" : {
"href" : "https://app.matatika.com/api/dataplugins/73b171be-27dc-4b4b-8cea-fce2492eb1ee"
},
"update dataplugin" : {
"href" : "https://app.matatika.com/api/workspaces/8b97a7ce-165e-45d7-b3e9-e18777312f8f/dataplugins/73b171be-27dc-4b4b-8cea-fce2492eb1ee",
"type" : "PUT"
}
}
}, {
"id" : "62ddd552-6cea-4924-b405-da6a71701b77",
"pluginType" : "TRANSFORMER",
"name" : "dbt",
"namespace" : "dbt",
"variant" : "dbt-labs",
"label" : "dbt",
"description" : " Power your project transformations with dbt™, a SQL-first transformation tool that enables analytics engineers to develop transformations with code.\n\n***Version Control and CI/CD***\n\nUse Matatika to deploy and promote changes between dev, UAT, and production environments.\n\n***Test and Document***\n\nUse Matatika to develop and test every model prior to production release, and share dynamically generated documentation with all stakeholders.\n\n***Develop***\n\nWrite modular data transformations in .sql – Matatika together with dbt handles the chore of dependency management. ",
"logoUrl" : "/assets/images/transformer/dbt.png",
"hidden" : false,
"docs" : "https://www.matatika.com/data-details/dbt/",
"pipUrl" : "dbt-core~=1.3.0 dbt-postgres~=1.3.0 dbt-snowflake~=1.3.0\n",
"repo" : "https://github.com/dbt-labs/dbt-core",
"capabilities" : [ ],
"select" : [ ],
"update" : { },
"vars" : { },
"settings" : [ {
"name" : "project_dir",
"aliases" : [ ],
"value" : "$MELTANO_PROJECT_ROOT/transform",
"options" : [ ],
"kind" : "STRING",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "profiles_dir",
"aliases" : [ ],
"value" : "$MELTANO_PROJECT_ROOT/transform/profile",
"options" : [ ],
"kind" : "STRING",
"hidden" : false,
"sensitive" : false,
"env" : "DBT_PROFILES_DIR",
"protected" : false
}, {
"name" : "target",
"aliases" : [ ],
"value" : "$MELTANO_LOAD__DIALECT",
"options" : [ ],
"kind" : "STRING",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "source_schema",
"aliases" : [ ],
"value" : "$MELTANO_LOAD__TARGET_SCHEMA",
"options" : [ ],
"kind" : "STRING",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "target_schema",
"aliases" : [ ],
"value" : "analytics",
"options" : [ ],
"kind" : "STRING",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "models",
"aliases" : [ ],
"value" : "$MELTANO_TRANSFORM__PACKAGE_NAME $MELTANO_EXTRACTOR_NAMESPACE my_meltano_project",
"options" : [ ],
"kind" : "STRING",
"hidden" : false,
"sensitive" : false,
"protected" : false
} ],
"variants" : [ ],
"commands" : {
"compile" : {
"args" : "compile",
"description" : "Generates executable SQL from source model, test, and analysis files. Compiled SQL files are written to the target/ directory."
},
"seed" : {
"args" : "seed",
"description" : "Load data from csv files into your data warehouse."
},
"test" : {
"args" : "test",
"description" : "Runs tests on data in deployed models."
},
"docs-generate" : {
"args" : "docs generate",
"description" : "Generate documentation artifacts for your project."
},
"deps" : {
"args" : "deps",
"description" : "Pull the most recent version of the dependencies listed in packages.yml"
},
"run" : {
"args" : "run",
"description" : "Compile SQL and execute against the current target database."
},
"clean" : {
"args" : "clean",
"description" : "Delete all folders in the clean-targets list (usually the dbt_modules and target directories.)"
},
"snapshot" : {
"args" : "snapshot",
"description" : "Execute snapshots defined in your project."
}
},
"matatikaHidden" : false,
"requires" : [ {
"id" : "e6c1ad3d-ebf5-4c4a-b129-f68156b47555",
"pluginType" : "FILE",
"name" : "files-dbt",
"namespace" : "dbt",
"variant" : "matatika",
"description" : " Files dbt is a file bundle that automatically configures your project to run transforms with dbt.\nThe bundle includes template project configuration:\n\n- transform/models (directory)\n- transform/profile/profiles.yml\n- transform/dbt_project.yml\n- transform/.gitignore\n- transform/macros/centralize_test_failures.sql\n",
"hidden" : false,
"pipUrl" : "git+https://github.com/Matatika/[email protected]",
"repo" : "https://github.com/Matatika/files-dbt",
"capabilities" : [ ],
"select" : [ ],
"update" : {
"transform/profile/profiles.yml" : "true"
},
"vars" : { },
"settings" : [ ],
"variants" : [ ],
"commands" : { },
"matatikaHidden" : false,
"requires" : [ ],
"fullDescription" : " Files dbt is a file bundle that automatically configures your project to run transforms with dbt.\nThe bundle includes template project configuration:\n\n- transform/models (directory)\n- transform/profile/profiles.yml\n- transform/dbt_project.yml\n- transform/.gitignore\n- transform/macros/centralize_test_failures.sql\n"
} ],
"fullDescription" : " Power your project transformations with dbt™, a SQL-first transformation tool that enables analytics engineers to develop transformations with code.\n\n***Version Control and CI/CD***\n\nUse Matatika to deploy and promote changes between dev, UAT, and production environments.\n\n***Test and Document***\n\nUse Matatika to develop and test every model prior to production release, and share dynamically generated documentation with all stakeholders.\n\n***Develop***\n\nWrite modular data transformations in .sql – Matatika together with dbt handles the chore of dependency management. ",
"_links" : {
"self" : {
"href" : "https://app.matatika.com/api/dataplugins/62ddd552-6cea-4924-b405-da6a71701b77"
},
"update dataplugin" : {
"href" : "https://app.matatika.com/api/workspaces/8b97a7ce-165e-45d7-b3e9-e18777312f8f/dataplugins/62ddd552-6cea-4924-b405-da6a71701b77",
"type" : "PUT"
}
}
}, {
"id" : "e4ca11eb-8d7e-41eb-b41a-97febf38c280",
"pluginType" : "EXTRACTOR",
"name" : "tap-custom-test",
"variant" : "sit",
"label" : "Tap Custom Test",
"description" : "A dataplugin created during an SIT run",
"hidden" : false,
"pipUrl" : "git+https://github.com/Matatika/example-repository",
"capabilities" : [ ],
"select" : [ ],
"update" : { },
"vars" : { },
"settings" : [ {
"name" : "username",
"aliases" : [ ],
"label" : "Username",
"options" : [ ],
"placeholder" : "username",
"kind" : "STRING",
"description" : "The username login credential.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "email",
"aliases" : [ ],
"label" : "Email",
"options" : [ ],
"placeholder" : "[email protected]",
"kind" : "EMAIL",
"description" : "The email login credential.",
"hidden" : false,
"sensitive" : false,
"protected" : false
}, {
"name" : "start_date",
"aliases" : [ ],
"label" : "Start Date",
"options" : [ ],
"placeholder" : "2020-01-01T00:00:00Z",
"kind" : "DATE_ISO8601",
"description" : "The data to begin extracting data from, in ISO 8601 format.",
"hidden" : false,
"sensitive" : false,
"protected" : false
} ],
"variants" : [ ],
"commands" : { },
"matatikaHidden" : false,
"requires" : [ ],
"fullDescription" : "A dataplugin created during an SIT run\n\n## Settings\n\n\n### Username\n\nThe username login credential.\n\n### Email\n\nThe email login credential.\n\n### Start Date\n\nThe data to begin extracting data from, in ISO 8601 format.",
"_links" : {
"self" : {
"href" : "https://app.matatika.com/api/dataplugins/e4ca11eb-8d7e-41eb-b41a-97febf38c280"
},
"update dataplugin" : {
"href" : "https://app.matatika.com/api/workspaces/8b97a7ce-165e-45d7-b3e9-e18777312f8f/dataplugins/e4ca11eb-8d7e-41eb-b41a-97febf38c280",
"type" : "PUT"
},
"delete dataplugin" : {
"href" : "https://app.matatika.com/api/dataplugins/e4ca11eb-8d7e-41eb-b41a-97febf38c280",
"type" : "DELETE"
}
}
}, {
"id" : "f1fd3b79-1a36-43ed-993e-9cfd18e2dfd6",
"pluginType" : "EXTRACTOR",
"name" : "tap-test",
"variant" : "sit",
"hidden" : false,
"capabilities" : [ ],
"select" : [ ],
"update" : { },
"vars" : { },
"settings" : [ ],
"variants" : [ ],
"commands" : { },
"matatikaHidden" : false,
"requires" : [ ],
"fullDescription" : "",
"_links" : {
"self" : {
"href" : "https://app.matatika.com/api/dataplugins/f1fd3b79-1a36-43ed-993e-9cfd18e2dfd6"
},
"update dataplugin" : {
"href" : "https://app.matatika.com/api/workspaces/8b97a7ce-165e-45d7-b3e9-e18777312f8f/dataplugins/f1fd3b79-1a36-43ed-993e-9cfd18e2dfd6",
"type" : "PUT"
},
"delete dataplugin" : {
"href" : "https://app.matatika.com/api/dataplugins/f1fd3b79-1a36-43ed-993e-9cfd18e2dfd6",
"type" : "DELETE"
}
}
}, {
"id" : "e48ea841-4cdd-4b2a-89d8-cb16b490fc5b",
"pluginType" : "LOADER",
"name" : "target-test",
"variant" : "sit",
"hidden" : false,
"capabilities" : [ ],
"select" : [ ],
"update" : { },
"vars" : { },
"settings" : [ ],
"variants" : [ ],
"commands" : { },
"matatikaHidden" : false,
"requires" : [ ],
"fullDescription" : "",
"_links" : {
"self" : {
"href" : "https://app.matatika.com/api/dataplugins/e48ea841-4cdd-4b2a-89d8-cb16b490fc5b"
},
"update dataplugin" : {
"href" : "https://app.matatika.com/api/workspaces/8b97a7ce-165e-45d7-b3e9-e18777312f8f/dataplugins/e48ea841-4cdd-4b2a-89d8-cb16b490fc5b",
"type" : "PUT"
},
"delete dataplugin" : {
"href" : "https://app.matatika.com/api/dataplugins/e48ea841-4cdd-4b2a-89d8-cb16b490fc5b",
"type" : "DELETE"
}
}
}, {
"id" : "8fa6d850-73f6-4c39-8128-2d0368fdd0da",
"pluginType" : "TRANSFORM",
"name" : "dbt-tap-test",
"variant" : "sit",
"hidden" : false,
"capabilities" : [ ],
"select" : [ ],
"update" : { },
"vars" : { },
"settings" : [ ],
"variants" : [ ],
"commands" : { },
"matatikaHidden" : false,
"requires" : [ ],
"fullDescription" : "",
"_links" : {
"self" : {
"href" : "https://app.matatika.com/api/dataplugins/8fa6d850-73f6-4c39-8128-2d0368fdd0da"
},
"update dataplugin" : {
"href" : "https://app.matatika.com/api/workspaces/8b97a7ce-165e-45d7-b3e9-e18777312f8f/dataplugins/8fa6d850-73f6-4c39-8128-2d0368fdd0da",
"type" : "PUT"
},
"delete dataplugin" : {
"href" : "https://app.matatika.com/api/dataplugins/8fa6d850-73f6-4c39-8128-2d0368fdd0da",
"type" : "DELETE"
}
}
}, {
"id" : "c7a0d505-28a4-4953-8930-226adf0067e4",
"pluginType" : "FILE",
"name" : "analyze-test",
"variant" : "sit",
"hidden" : false,
"capabilities" : [ ],
"select" : [ ],
"update" : { },
"vars" : { },
"settings" : [ ],
"variants" : [ ],
"commands" : { },
"matatikaHidden" : false,
"requires" : [ ],
"fullDescription" : "",
"_links" : {
"self" : {
"href" : "https://app.matatika.com/api/dataplugins/c7a0d505-28a4-4953-8930-226adf0067e4"
},
"update dataplugin" : {
"href" : "https://app.matatika.com/api/workspaces/8b97a7ce-165e-45d7-b3e9-e18777312f8f/dataplugins/c7a0d505-28a4-4953-8930-226adf0067e4",
"type" : "PUT"
},
"delete dataplugin" : {
"href" : "https://app.matatika.com/api/dataplugins/c7a0d505-28a4-4953-8930-226adf0067e4",
"type" : "DELETE"
}
}
} ]
},
"_links" : {
"self" : {
"href" : "https://app.matatika.com/api/workspaces/8b97a7ce-165e-45d7-b3e9-e18777312f8f/dataplugins?page=0&size=20"
}
},
"page" : {
"size" : 20,
"totalElements" : 7,
"totalPages" : 1,
"number" : 0
}
}
View a workspace discovery.yml
GET
/api/workspaces/{workspace-id}/discovery.yml
Returns a Meltano discovery.yml containing all dataplugins available to the workspace {workspace-id}.
Prerequisites
- Workspace
{workspace-id}must exist
Request
Example Snippets
cURL
curl -H "Authorization: Bearer $ACCESS_TOKEN" 'https://app.matatika.com/api/workspaces/8b97a7ce-165e-45d7-b3e9-e18777312f8f/discovery.yml' -i -X GET \
-H 'Accept: application/json, application/javascript, text/javascript, text/json' \
-H 'Content-Type: application/json'
Python (requests)
import requests
url = "https://app.matatika.com/api/workspaces/8b97a7ce-165e-45d7-b3e9-e18777312f8f/discovery.yml"
headers = {
'Authorization': ACCESS_TOKEN
}
response = requests.request("GET", url, headers=headers)
print(response.text.encode('utf8'))
Response
200 OK
version: 20
extractors:
- id: f1fd3b79-1a36-43ed-993e-9cfd18e2dfd6
name: tap-test
variant: sit
hidden: false
- id: e4ca11eb-8d7e-41eb-b41a-97febf38c280
name: tap-custom-test
variant: sit
label: Tap Custom Test
description: A dataplugin created during an SIT run
hidden: false
pip_url: git+https://github.com/Matatika/example-repository
settings:
- name: username
label: Username
placeholder: username
kind: string
description: The username login credential.
hidden: false
sensitive: false
protected: false
- name: email
label: Email
placeholder: [email protected]
kind: email
description: The email login credential.
hidden: false
sensitive: false
protected: false
- name: start_date
label: Start Date
placeholder: 2020-01-01T00:00:00Z
kind: date_iso8601
description: "The data to begin extracting data from, in ISO 8601 format."
hidden: false
sensitive: false
protected: false
full_description: |-
A dataplugin created during an SIT run
## Settings
### Username
The username login credential.
### Email
The email login credential.
### Start Date
The data to begin extracting data from, in ISO 8601 format.
- id: 92c7df8c-4eb6-4cc6-8f6b-e5d1a2acfdda
name: tap-thinkific
namespace: tap_thinkific
variant: birdiecare
label: Thinkific
description: |-
Thinkific is an online course creation platform.
Thinkific is a platform that allows individuals and businesses to create and sell online courses. It provides tools for course creation, customization, marketing, and delivery, as well as features for student engagement and progress tracking. Thinkific also offers integrations with other tools and services, such as payment gateways, email marketing platforms, and analytics tools. With Thinkific, users can create and sell courses on a variety of topics, from business and marketing to health and wellness, and reach a global audience.
logo_url: /assets/logos/extractors/thinkific.png
hidden: false
docs: https://www.matatika.com/data-details/tap-thinkific/
pip_url: git+https://github.com/birdiecare/tap-thinkific.git
repo: https://github.com/birdiecare/tap-thinkific
capabilities:
- discover
- stream_maps
- catalog
- state
- about
settings:
- name: api_key
label: API Key
kind: string
description: A unique identifier used to authenticate and authorize API requests
hidden: false
sensitive: true
protected: false
- name: subdomain
label: Subdomain
kind: string
description: The unique identifier for the Thinkific account being accessed
hidden: false
sensitive: false
protected: false
- name: start_date
label: Start Date
kind: string
description: The date from which data should be retrieved or processed
hidden: false
sensitive: false
protected: false
full_description: |-
Thinkific is an online course creation platform.
Thinkific is a platform that allows individuals and businesses to create and sell online courses. It provides tools for course creation, customization, marketing, and delivery, as well as features for student engagement and progress tracking. Thinkific also offers integrations with other tools and services, such as payment gateways, email marketing platforms, and analytics tools. With Thinkific, users can create and sell courses on a variety of topics, from business and marketing to health and wellness, and reach a global audience.
## Settings
### API Key
A unique identifier used to authenticate and authorize API requests
### Subdomain
The unique identifier for the Thinkific account being accessed
### Start Date
The date from which data should be retrieved or processed
- id: e9758d71-1bc8-4ab0-8e99-1e7575bce596
name: tap-redshift
namespace: tap_redshift
variant: monad-inc
label: Redshift
description: |-
Redshift is a cloud-based data warehousing service provided by Amazon Web Services (AWS).
Redshift allows users to store and analyze large amounts of data in a scalable and cost-effective manner. It uses columnar storage and parallel processing to enable fast querying of data using SQL. Redshift integrates with a variety of data sources and tools, including AWS services like S3 and EMR, as well as popular BI and ETL tools. It also offers features like automatic backups, encryption, and workload management to ensure data security and performance. Overall, Redshift is a powerful solution for businesses looking to manage and analyze their data in the cloud.
logo_url: /assets/logos/extractors/redshift.png
hidden: false
docs: https://www.matatika.com/data-details/tap-redshift/
pip_url: git+https://github.com/Monad-Inc/tap-redshift.git
repo: https://github.com/Monad-Inc/tap-redshift
capabilities:
- discover
- catalog
- state
settings:
- name: host
label: Host
kind: string
description: The URL or IP address of the Redshift cluster
hidden: false
sensitive: false
protected: false
- name: user
label: User
kind: string
description: The username used to authenticate with the Redshift cluster
hidden: false
sensitive: false
protected: false
- name: start_date
label: Start Date
kind: date_iso8601
description: The date from which data will be retrieved
hidden: false
sensitive: false
protected: false
- name: port
label: Port
kind: integer
description: The port number used to connect to the Redshift cluster
hidden: false
sensitive: false
protected: false
- name: dbname
label: Database Name
kind: string
description: The name of the database within the Redshift cluster
hidden: false
sensitive: false
protected: false
- name: password
label: Password
kind: string
description: The password used to authenticate with the Redshift cluster
hidden: false
sensitive: true
protected: false
- name: schema
label: Schema Name
kind: string
description: The name of the schema within the database
hidden: false
sensitive: false
protected: false
full_description: |-
Redshift is a cloud-based data warehousing service provided by Amazon Web Services (AWS).
Redshift allows users to store and analyze large amounts of data in a scalable and cost-effective manner. It uses columnar storage and parallel processing to enable fast querying of data using SQL. Redshift integrates with a variety of data sources and tools, including AWS services like S3 and EMR, as well as popular BI and ETL tools. It also offers features like automatic backups, encryption, and workload management to ensure data security and performance. Overall, Redshift is a powerful solution for businesses looking to manage and analyze their data in the cloud.
## Settings
### Host
The URL or IP address of the Redshift cluster
### User
The username used to authenticate with the Redshift cluster
### Start Date
The date from which data will be retrieved
### Port
The port number used to connect to the Redshift cluster
### Database Name
The name of the database within the Redshift cluster
### Password
The password used to authenticate with the Redshift cluster
### Schema Name
The name of the schema within the database
- id: 10adc98a-dae3-4e7d-854d-81ea9b0c575a
name: tap-facebook-reviews
namespace: tap_facebook_reviews
variant: packlane
label: Facebook Reviews
description: |-
Facebook Reviews: A tool for businesses to collect and display customer reviews on their Facebook page.
Facebook Reviews is a feature that allows businesses to collect and display customer reviews on their Facebook page. This tool helps businesses build credibility and trust with potential customers by showcasing positive feedback from previous customers. Businesses can also respond to reviews and engage with customers to address any concerns or issues. Facebook Reviews is a valuable tool for businesses looking to improve their online reputation and attract new customers.
logo_url: /assets/logos/extractors/facebook-reviews.png
hidden: false
docs: https://www.matatika.com/data-details/tap-facebook-reviews/
pip_url: git+https://github.com/Packlane/tap-facebook-reviews.git
repo: https://github.com/Packlane/tap-facebook-reviews
capabilities:
- discover
- catalog
full_description: |-
Facebook Reviews: A tool for businesses to collect and display customer reviews on their Facebook page.
Facebook Reviews is a feature that allows businesses to collect and display customer reviews on their Facebook page. This tool helps businesses build credibility and trust with potential customers by showcasing positive feedback from previous customers. Businesses can also respond to reviews and engage with customers to address any concerns or issues. Facebook Reviews is a valuable tool for businesses looking to improve their online reputation and attract new customers.
- id: 123f0342-634c-46c0-9213-8dfd197abe03
name: tap-criteo
namespace: tap_criteo
variant: edgarrmondragon
label: Criteo
description: |-
Criteo: A digital advertising platform.
Criteo is a digital advertising platform that uses machine learning algorithms to deliver personalized ads to consumers across various devices and channels. It helps advertisers reach their target audience by analyzing consumer behavior and purchasing patterns to deliver relevant ads at the right time. Criteo's platform also provides insights and analytics to help advertisers optimize their campaigns and measure their return on investment.
logo_url: /assets/logos/extractors/criteo.png
hidden: false
docs: https://www.matatika.com/data-details/tap-criteo/
pip_url: git+https://github.com/edgarrmondragon/tap-criteo.git
repo: https://github.com/edgarrmondragon/tap-criteo
executable: tap-criteo
capabilities:
- discover
- stream_maps
- catalog
- schema_flattening
- about
settings:
- name: advertiser_ids
label: Advertiser IDs
kind: array
description: The unique IDs assigned to each advertiser account within Criteo.
hidden: false
sensitive: false
protected: false
- name: client_id
label: Client ID
kind: string
description: The unique identifier for the client application connecting to the Criteo API.
hidden: false
sensitive: true
protected: false
- name: client_secret
label: Client Secret
kind: string
description: The secret key used to authenticate the client application.
hidden: false
sensitive: true
protected: false
- name: flattening_enabled
label: Flattening Enabled
kind: boolean
description: A boolean value indicating whether or not to flatten nested JSON objects in the API response.
hidden: false
sensitive: false
protected: false
- name: flattening_max_depth
label: Flattening Max Depth
kind: integer
description: The maximum depth to which nested JSON objects should be flattened.
hidden: false
sensitive: false
protected: false
- name: reports
label: Reports
kind: array
description: The type of report to retrieve from the Criteo API.
hidden: false
sensitive: false
protected: false
- name: start_date
label: Start Date
kind: date_iso8601
description: The date from which to start retrieving data for the specified report.
hidden: false
sensitive: false
protected: false
- name: stream_map_config
label: Stream Map Config
kind: object
description: The configuration settings for the stream map used to retrieve data from the Criteo API.
hidden: false
sensitive: false
protected: false
- name: stream_maps
label: Stream Maps
kind: object
description: The specific stream maps to use for retrieving data from the Criteo API.
hidden: false
sensitive: false
protected: false
full_description: |-
Criteo: A digital advertising platform.
Criteo is a digital advertising platform that uses machine learning algorithms to deliver personalized ads to consumers across various devices and channels. It helps advertisers reach their target audience by analyzing consumer behavior and purchasing patterns to deliver relevant ads at the right time. Criteo's platform also provides insights and analytics to help advertisers optimize their campaigns and measure their return on investment.
## Settings
### Advertiser IDs
The unique IDs assigned to each advertiser account within Criteo.
### Client ID
The unique identifier for the client application connecting to the Criteo API.
### Client Secret
The secret key used to authenticate the client application.
### Flattening Enabled
A boolean value indicating whether or not to flatten nested JSON objects in the API response.
### Flattening Max Depth
The maximum depth to which nested JSON objects should be flattened.
### Reports
The type of report to retrieve from the Criteo API.
### Start Date
The date from which to start retrieving data for the specified report.
### Stream Map Config
The configuration settings for the stream map used to retrieve data from the Criteo API.
### Stream Maps
The specific stream maps to use for retrieving data from the Criteo API.
- id: 5190f60e-1978-4e49-b6b9-57de5b260455
name: tap-amazon-sp
namespace: tap_amazon_seller
variant: hotgluexyz
label: Amazon Selling Partner (SP)
description: |-
Amazon Selling Partner (SP) is a platform that helps sellers manage their Amazon business.
Amazon Selling Partner (SP) is a comprehensive platform that provides sellers with tools to manage their Amazon business. It offers features such as inventory management, order fulfillment, advertising, and analytics. With SP, sellers can track their sales performance, manage their inventory, and optimize their product listings. The platform also provides access to Amazon's advertising tools, allowing sellers to create and manage campaigns to promote their products. Additionally, SP offers insights and analytics to help sellers make data-driven decisions to grow their business on Amazon.
logo_url: /assets/logos/extractors/amazon-sp.png
hidden: false
docs: https://www.matatika.com/data-details/tap-amazon-sp/
pip_url: git+https://gitlab.com/hotglue/tap-amazon-seller.git
repo: https://gitlab.com/hotglue/tap-amazon-seller
executable: tap-amazon-seller
capabilities:
- discover
- stream_maps
- catalog
- state
- schema_flattening
- about
settings:
- name: aws_access_key
label: AWS Access Key
kind: string
description: The access key ID for the AWS account.
hidden: false
sensitive: true
protected: false
- name: aws_secret_key
label: AWS Secret Key
kind: string
description: The secret access key for the AWS account.
hidden: false
sensitive: true
protected: false
- name: client_secret
label: Client Secret
kind: string
description: The client secret for the OAuth 2.0 client.
hidden: false
sensitive: true
protected: false
- name: flattening_enabled
label: Flattening Enabled
kind: boolean
description: A boolean value indicating whether or not to flatten the response data.
hidden: false
sensitive: false
protected: false
- name: flattening_max_depth
label: Flattening Max Depth
kind: integer
description: The maximum depth to which the response data should be flattened.
hidden: false
sensitive: false
protected: false
- name: lwa_client_id
label: Lwa Client ID
kind: string
description: The client ID for the Login with Amazon (LWA) client.
hidden: false
sensitive: true
protected: false
- name: marketplaces
label: Marketplaces
kind: array
description: The Amazon marketplaces for which the API requests will be made.
hidden: false
sensitive: false
protected: false
- name: processing_status
label: Processing Status
value: "[\"IN_QUEUE\",\"IN_PROGRESS\"]"
kind: array
description: The processing status of the API request.
hidden: false
sensitive: false
protected: false
- name: refresh_token
label: Refresh Token
kind: string
description: The refresh token for the OAuth 2.0 client.
hidden: false
sensitive: true
protected: false
- name: report_types
label: Report Types
value: "[\"GET_LEDGER_DETAIL_VIEW_DATA\",\"GET_MERCHANT_LISTINGS_ALL_DATA\"]"
kind: array
description: The types of reports that can be requested from the API.
hidden: false
sensitive: false
protected: false
- name: role_arn
label: Role Arn
kind: string
description: The Amazon Resource Name (ARN) of the role that the API will assume.
hidden: false
sensitive: false
protected: false
- name: sandbox
label: Sandbox
value: "false"
kind: boolean
description: A boolean value indicating whether or not to use the Amazon Selling Partner API sandbox environment.
hidden: false
sensitive: false
protected: false
- name: stream_map_config
label: Stream Map Config
kind: object
description: The configuration for the stream map.
hidden: false
sensitive: false
protected: false
- name: stream_maps
label: Stream Maps
kind: object
description: The stream maps for the API requests.
hidden: false
sensitive: false
protected: false
full_description: |-
Amazon Selling Partner (SP) is a platform that helps sellers manage their Amazon business.
Amazon Selling Partner (SP) is a comprehensive platform that provides sellers with tools to manage their Amazon business. It offers features such as inventory management, order fulfillment, advertising, and analytics. With SP, sellers can track their sales performance, manage their inventory, and optimize their product listings. The platform also provides access to Amazon's advertising tools, allowing sellers to create and manage campaigns to promote their products. Additionally, SP offers insights and analytics to help sellers make data-driven decisions to grow their business on Amazon.
## Settings
### AWS Access Key
The access key ID for the AWS account.
### AWS Secret Key
The secret access key for the AWS account.
### Client Secret
The client secret for the OAuth 2.0 client.
### Flattening Enabled
A boolean value indicating whether or not to flatten the response data.
### Flattening Max Depth
The maximum depth to which the response data should be flattened.
### Lwa Client ID
The client ID for the Login with Amazon (LWA) client.
### Marketplaces
The Amazon marketplaces for which the API requests will be made.
### Processing Status
The processing status of the API request.
### Refresh Token
The refresh token for the OAuth 2.0 client.
### Report Types
The types of reports that can be requested from the API.
### Role Arn
The Amazon Resource Name (ARN) of the role that the API will assume.
### Sandbox
A boolean value indicating whether or not to use the Amazon Selling Partner API sandbox environment.
### Stream Map Config
The configuration for the stream map.
### Stream Maps
The stream maps for the API requests.
- id: c6263b4c-090a-45f8-8669-9db5edc87ead
name: tap-fulfil
namespace: tap_fulfil
variant: fulfilio
label: Fulfil
description: |-
Fulfil is a cloud-based software for managing inventory, orders, and shipping.
Fulfil is an all-in-one solution for businesses to manage their inventory, orders, and shipping. With features such as real-time inventory tracking, order management, and shipping integrations, Fulfil helps businesses streamline their operations and improve their overall efficiency. The software also includes tools for managing customer relationships, generating reports, and automating tasks, making it a comprehensive solution for businesses of all sizes. Additionally, Fulfil offers integrations with popular e-commerce platforms such as Shopify, Magento, and WooCommerce, allowing businesses to easily sync their online stores with their inventory and order management systems.
logo_url: /assets/logos/extractors/fulfil.png
hidden: false
docs: https://www.matatika.com/data-details/tap-fulfil/
pip_url: git+https://github.com/fulfilio/tap-fulfil.git
repo: https://github.com/fulfilio/tap-fulfil
capabilities:
- discover
- catalog
full_description: |-
Fulfil is a cloud-based software for managing inventory, orders, and shipping.
Fulfil is an all-in-one solution for businesses to manage their inventory, orders, and shipping. With features such as real-time inventory tracking, order management, and shipping integrations, Fulfil helps businesses streamline their operations and improve their overall efficiency. The software also includes tools for managing customer relationships, generating reports, and automating tasks, making it a comprehensive solution for businesses of all sizes. Additionally, Fulfil offers integrations with popular e-commerce platforms such as Shopify, Magento, and WooCommerce, allowing businesses to easily sync their online stores with their inventory and order management systems.
- id: 7e2df860-abd3-4900-a771-c59f7305c77e
name: tap-clarabridge
namespace: tap_clarabridge
variant: pathlight
label: Clarabridge
description: |-
Clarabridge is a customer experience management software and service provider.
Clarabridge offers a suite of software and services that help businesses collect, analyze, and act on customer feedback across various channels such as social media, email, chat, and surveys. The platform uses natural language processing and machine learning to extract insights from customer feedback and provide actionable insights to improve customer experience, increase customer loyalty, and drive business growth. Clarabridge's solutions are used by leading brands across industries such as retail, hospitality, financial services, and healthcare.
logo_url: /assets/logos/extractors/clarabridge.png
hidden: false
docs: https://www.matatika.com/data-details/tap-clarabridge/
pip_url: git+https://github.com/Pathlight/tap-clarabridge.git
repo: https://github.com/Pathlight/tap-clarabridge
capabilities:
- discover
- catalog
full_description: |-
Clarabridge is a customer experience management software and service provider.
Clarabridge offers a suite of software and services that help businesses collect, analyze, and act on customer feedback across various channels such as social media, email, chat, and surveys. The platform uses natural language processing and machine learning to extract insights from customer feedback and provide actionable insights to improve customer experience, increase customer loyalty, and drive business growth. Clarabridge's solutions are used by leading brands across industries such as retail, hospitality, financial services, and healthcare.
- id: c4186ab8-7fbd-4857-8a2c-d004d2511823
name: tap-govuk-bank-holidays
namespace: tap_govuk_bank_holidays
variant: matatika
label: UK Bank Holidays
description: |-
UK Bank Holidays
If a bank holiday is on a weekend, a ‘substitute’ weekday becomes a bank holiday, normally the following Monday.
## Learn more
[GOV.UK Bank Holidays](https://www.gov.uk/bank-holidays)
logo_url: https://www.gov.uk/assets/static/govuk-opengraph-image-dade2dad5775023b0568381c4c074b86318194edb36d3d68df721eea7deeac4b.png
hidden: false
docs: https://www.matatika.com/data-details/tap-govuk-bank-holidays/
pip_url: git+https://github.com/Matatika/tap-spreadsheets-anywhere@v0.2.1
repo: https://github.com/Matatika/tap-spreadsheets-anywhere
executable: tap-spreadsheets-anywhere
capabilities:
- discover
- catalog
- state
settings:
- name: tables
label: Tables
value: |-
[{
"path":"https://www.gov.uk/",
"name":"uk_bank_holidays_england_and_wales",
"pattern":"bank-holidays.json",
"start_date":"2018-01-01T00:00:00Z",
"key_properties":["date"],
"json_path":"$.england-and-wales.events",
"format":"json"
}, {
"path":"https://www.gov.uk/",
"name":"uk_bank_holidays_scotland",
"pattern":"bank-holidays.json",
"start_date":"2018-01-01T00:00:00Z",
"key_properties":["date"],
"json_path":"$.scotland.events",
"format":"json"
}, {
"path":"https://www.gov.uk/",
"name":"uk_bank_holidays_northern_ireland",
"pattern":"bank-holidays.json",
"start_date":"2018-01-01T00:00:00Z",
"key_properties":["date"],
"json_path":"$.northern-ireland.events",
"format":"json"
}]
kind: array
description: A collection of related data organized in rows and columns.
hidden: false
sensitive: false
required: "false"
protected: false
full_description: |-
UK Bank Holidays
If a bank holiday is on a weekend, a ‘substitute’ weekday becomes a bank holiday, normally the following Monday.
## Learn more
[GOV.UK Bank Holidays](https://www.gov.uk/bank-holidays)
## Settings
### Tables
A collection of related data organized in rows and columns.
- id: 7d0af4b1-4b6c-4fc2-b850-370983fe6597
name: tap-monday
namespace: tap_monday
variant: gthesheep
label: Monday.com
description: "Monday.com is a team management and collaboration platform that helps teams plan, organize, and track their work in one central location. \n\nMonday.com is a cloud-based platform that allows teams to manage their projects, tasks, and workflows in a visual and intuitive way. It offers a variety of customizable templates and features, such as task assignments, deadlines, progress tracking, and communication tools, to help teams stay on top of their work and collaborate effectively. With Monday.com, teams can streamline their workflows, improve their productivity, and achieve their goals faster."
logo_url: /assets/logos/extractors/monday.png
hidden: false
docs: https://www.matatika.com/data-details/tap-monday/
pip_url: git+https://github.com/gthesheep/tap-monday.git
repo: https://github.com/gthesheep/tap-monday
capabilities:
- discover
- stream_maps
- catalog
- state
- about
settings:
- name: auth_token
label: API Token
kind: string
description: A unique identifier that grants access to the Monday.com API.
hidden: false
sensitive: true
protected: false
- name: board_limit
label: Board Limit
kind: string
description: The maximum number of boards that can be accessed through the API.
hidden: false
sensitive: false
protected: false
full_description: "Monday.com is a team management and collaboration platform that helps teams plan, organize, and track their work in one central location. \n\nMonday.com is a cloud-based platform that allows teams to manage their projects, tasks, and workflows in a visual and intuitive way. It offers a variety of customizable templates and features, such as task assignments, deadlines, progress tracking, and communication tools, to help teams stay on top of their work and collaborate effectively. With Monday.com, teams can streamline their workflows, improve their productivity, and achieve their goals faster.\n\n## Settings\n\n\n### API Token\n\nA unique identifier that grants access to the Monday.com API.\n\n### Board Limit\n\nThe maximum number of boards that can be accessed through the API."
- id: aee84aa6-17f1-4938-85b3-597e8bbeebc7
name: tap-dagster
namespace: tap_dagster
variant: voxmedia
label: Dagster
description: |-
Dagster is an open-source data orchestrator for machine learning, analytics, and ETL.
Dagster provides a unified framework for building data pipelines that allows developers to define the inputs, outputs, and dependencies of each step in the pipeline, making it easier to test, maintain, and scale complex data workflows. It also includes features such as data validation, error handling, and monitoring to ensure the reliability and quality of data processing. Dagster supports a variety of data sources and execution environments, including local development, cloud-based services, and containerized deployments.
logo_url: /assets/logos/extractors/dagster.png
hidden: false
docs: https://www.matatika.com/data-details/tap-dagster/
pip_url: git+https://github.com/voxmedia/tap-dagster.git
repo: https://github.com/voxmedia/tap-dagster
capabilities:
- discover
- stream_maps
- catalog
- state
- schema_flattening
- about
settings:
- name: auth_token
label: Auth Token
kind: string
description: A token used for authentication when connecting to the Dagster API.
hidden: false
sensitive: true
protected: false
- name: start_date
label: Start Date
kind: string
description: The date from which to start streaming data.
hidden: false
sensitive: false
protected: false
- name: api_url
label: Api Url
kind: string
description: The URL of the Dagster API.
hidden: false
sensitive: false
protected: false
- name: stream_maps
label: Stream Maps
kind: object
description: A list of stream maps to use when streaming data.
hidden: false
sensitive: false
protected: false
- name: stream_map_config
label: Stream Map Config
kind: object
description: Configuration settings for the stream maps.
hidden: false
sensitive: false
protected: false
- name: flattening_enabled
label: Flattening Enabled
kind: boolean
description: Whether or not to flatten the data when streaming.
hidden: false
sensitive: false
protected: false
- name: flattening_max_depth
label: Flattening Max Depth
kind: integer
description: The maximum depth to which the data should be flattened.
hidden: false
sensitive: false
protected: false
full_description: |-
Dagster is an open-source data orchestrator for machine learning, analytics, and ETL.
Dagster provides a unified framework for building data pipelines that allows developers to define the inputs, outputs, and dependencies of each step in the pipeline, making it easier to test, maintain, and scale complex data workflows. It also includes features such as data validation, error handling, and monitoring to ensure the reliability and quality of data processing. Dagster supports a variety of data sources and execution environments, including local development, cloud-based services, and containerized deployments.
## Settings
### Auth Token
A token used for authentication when connecting to the Dagster API.
### Start Date
The date from which to start streaming data.
### Api Url
The URL of the Dagster API.
### Stream Maps
A list of stream maps to use when streaming data.
### Stream Map Config
Configuration settings for the stream maps.
### Flattening Enabled
Whether or not to flatten the data when streaming.
### Flattening Max Depth
The maximum depth to which the data should be flattened.
- id: ab433553-3d8d-40e3-802f-53f8c9e025b5
name: tap-keap
namespace: tap_keap
variant: hotgluexyz
label: Keap
description: "Keap is a customer relationship management (CRM) software designed for small businesses to manage their sales, marketing, and customer service in one platform. \n\nKeap offers a range of features including contact management, appointment scheduling, lead capture and segmentation, email marketing, automation, and reporting. It allows businesses to streamline their processes and improve their customer relationships by providing a centralized platform for managing customer interactions. Keap also integrates with other tools such as QuickBooks, Gmail, and Outlook to provide a seamless experience for users. With Keap, small businesses can save time, increase efficiency, and grow their customer base."
logo_url: /assets/logos/extractors/keap.svg
hidden: false
docs: https://www.matatika.com/data-details/tap-keap/
pip_url: git+https://gitlab.com/hotglue/tap-keap.git
repo: https://gitlab.com/hotglue/tap-keap
executable: tap-keap
capabilities:
- discover
- stream_maps
- catalog
- state
- schema_flattening
- about
settings:
- name: access_token
label: Access Token
kind: string
description: A unique identifier that grants access to the Keap API.
hidden: false
sensitive: true
protected: false
- name: client_id
label: Client ID
kind: string
description: A unique identifier for the client application that is making the API request.
hidden: false
sensitive: true
protected: false
- name: client_secret
label: Client Secret
kind: string
description: A secret key that is used to authenticate the client application.
hidden: false
sensitive: true
protected: false
- name: expires_in
label: Expires In
kind: integer
description: The amount of time in seconds until the access token expires.
hidden: false
sensitive: false
protected: false
- name: flattening_enabled
label: Flattening Enabled
kind: boolean
description: A boolean value indicating whether or not to flatten nested objects in the API response.
hidden: false
sensitive: false
protected: false
- name: flattening_max_depth
label: Flattening Max Depth
kind: integer
description: The maximum depth to which nested objects will be flattened.
hidden: false
sensitive: false
protected: false
- name: start_date
label: Start Date
kind: date_iso8601
description: The date from which to start retrieving data from the API.
hidden: false
sensitive: false
protected: false
- name: stream_map_config
label: Stream Map Config
kind: object
description: A configuration file that maps API responses to a specific data model.
hidden: false
sensitive: false
protected: false
- name: stream_maps
label: Stream Maps
kind: object
description: A collection of stream maps that define how to transform API responses into a specific data model.
hidden: false
sensitive: false
protected: false
full_description: "Keap is a customer relationship management (CRM) software designed for small businesses to manage their sales, marketing, and customer service in one platform. \n\nKeap offers a range of features including contact management, appointment scheduling, lead capture and segmentation, email marketing, automation, and reporting. It allows businesses to streamline their processes and improve their customer relationships by providing a centralized platform for managing customer interactions. Keap also integrates with other tools such as QuickBooks, Gmail, and Outlook to provide a seamless experience for users. With Keap, small businesses can save time, increase efficiency, and grow their customer base.\n\n## Settings\n\n\n### Access Token\n\nA unique identifier that grants access to the Keap API.\n\n### Client ID\n\nA unique identifier for the client application that is making the API request.\n\n### Client Secret\n\nA secret key that is used to authenticate the client application.\n\n### Expires In\n\nThe amount of time in seconds until the access token expires.\n\n### Flattening Enabled\n\nA boolean value indicating whether or not to flatten nested objects in the API response.\n\n### Flattening Max Depth\n\nThe maximum depth to which nested objects will be flattened.\n\n### Start Date\n\nThe date from which to start retrieving data from the API.\n\n### Stream Map Config\n\nA configuration file that maps API responses to a specific data model.\n\n### Stream Maps\n\nA collection of stream maps that define how to transform API responses into a specific data model."
- id: b8428834-d995-4d66-9b31-105a83e80483
name: tap-mailchimp
namespace: tap_mailchimp
variant: singer-io
label: Mailchimp
description: |-
Mailchimp is an email marketing and automation platform.
Mailchimp is a cloud-based platform that allows businesses to create and send email campaigns, manage subscriber lists, and automate marketing tasks. It offers a variety of templates and design tools to create professional-looking emails, as well as analytics to track the success of campaigns. Mailchimp also integrates with other tools and platforms, such as social media and e-commerce sites, to help businesses reach their target audience and grow their customer base.
logo_url: /assets/logos/extractors/mailchimp.png
hidden: false
docs: https://www.matatika.com/data-details/tap-mailchimp/
pip_url: tap-mailchimp
repo: https://github.com/singer-io/tap-mailchimp
capabilities:
- discover
- catalog
- state
settings:
- name: request_timeout
label: Request Timeout
kind: integer
description: The maximum amount of time the client will wait for a response from the server before timing out.
hidden: false
sensitive: false
protected: false
- name: dc
label: Data Center
kind: string
description: The unique identifier for the Mailchimp data center that the API request will be sent to.
hidden: false
sensitive: false
protected: false
- name: page_size
label: Page Size
kind: integer
description: The number of results to return per page when making paginated API requests.
hidden: false
sensitive: false
protected: false
- name: user_agent
label: User Agent
kind: string
description: A string that identifies the client making the API request.
hidden: false
sensitive: false
protected: false
- name: start_date
label: Start Date
kind: date_iso8601
description: The date from which to start retrieving data when making API requests that return historical data.
hidden: false
sensitive: false
protected: false
- name: access_token
label: Access Token
kind: string
description: A unique identifier that grants access to a specific Mailchimp account and its associated data.
hidden: false
sensitive: true
protected: false
- name: api_key
label: API Key
kind: string
description: A unique identifier that grants access to the Mailchimp API and its associated functionality.
hidden: false
sensitive: true
protected: false
full_description: |-
Mailchimp is an email marketing and automation platform.
Mailchimp is a cloud-based platform that allows businesses to create and send email campaigns, manage subscriber lists, and automate marketing tasks. It offers a variety of templates and design tools to create professional-looking emails, as well as analytics to track the success of campaigns. Mailchimp also integrates with other tools and platforms, such as social media and e-commerce sites, to help businesses reach their target audience and grow their customer base.
## Settings
### Request Timeout
The maximum amount of time the client will wait for a response from the server before timing out.
### Data Center
The unique identifier for the Mailchimp data center that the API request will be sent to.
### Page Size
The number of results to return per page when making paginated API requests.
### User Agent
A string that identifies the client making the API request.
### Start Date
The date from which to start retrieving data when making API requests that return historical data.
### Access Token
A unique identifier that grants access to a specific Mailchimp account and its associated data.
### API Key
A unique identifier that grants access to the Mailchimp API and its associated functionality.
- id: 5a59fc3d-3e5a-4e77-a69a-2607160127a6
name: tap-rockgympro
namespace: tap_rockgympro
variant: cinchio
label: Rock Gym Pro
description: "Rock Gym Pro is a gym management software designed for rock climbing facilities. \n\nRock Gym Pro is a comprehensive software solution that helps rock climbing gyms manage their operations, from membership and billing to scheduling and inventory management. It offers features such as online registration, automated billing, and real-time reporting, as well as tools for managing classes, events, and competitions. The software also includes a mobile app for members, allowing them to check schedules, sign up for classes, and track their progress. With Rock Gym Pro, gym owners and managers can streamline their operations, improve customer experience, and grow their business."
logo_url: /assets/logos/extractors/rockgympro.png
hidden: false
docs: https://www.matatika.com/data-details/tap-rockgympro/
pip_url: git+https://github.com/cinchio/tap-rockgympro.git
repo: https://github.com/cinchio/tap-rockgympro
capabilities:
- discover
- catalog
full_description: "Rock Gym Pro is a gym management software designed for rock climbing facilities. \n\nRock Gym Pro is a comprehensive software solution that helps rock climbing gyms manage their operations, from membership and billing to scheduling and inventory management. It offers features such as online registration, automated billing, and real-time reporting, as well as tools for managing classes, events, and competitions. The software also includes a mobile app for members, allowing them to check schedules, sign up for classes, and track their progress. With Rock Gym Pro, gym owners and managers can streamline their operations, improve customer experience, and grow their business."
- id: bc91e7c0-6ade-43f3-987e-56083ce3f834
name: tap-anvil