Link Search Menu Expand Document

Dataplugins

Dataplugins simply define a source of data from a given repository. Matatika provides a number of pre-configured platform-wide dataplugins out-the-box, as well as the ability to create custom dataplugins through the API. From these, pipeline jobs can be run to inject data into a workspace.


Objects

Dataplugin

Path Type Format Description
id String Version 4 UUID The dataplugin ID
name String   The dataplugin name
description String   A description of the dataplugin
repositoryUrl String URL The dataplugin repository URL
settings Array of Setting   The dataplugin settings
{
  "id" : "549aab3e-1c5a-4b54-ba08-8491a27cecc1",
  "pluginType" : "TRANSFORMER",
  "name" : "dbt",
  "namespace" : "dbt",
  "variant" : "dbt",
  "logoUrl" : "/assets/images/transformer/dbt.png",
  "hidden" : false,
  "docs" : "https://meltano.com/docs/transforms.html",
  "pipUrl" : "dbt-core~=1.0.0 dbt-postgres~=1.0.0 dbt-redshift~=1.0.0 dbt-snowflake~=1.0.0 dbt-bigquery~=1.0.0\n",
  "repo" : "https://github.com/dbt-labs/dbt-core",
  "capabilities" : [ ],
  "select" : [ ],
  "update" : { },
  "vars" : { },
  "settings" : [ {
    "name" : "project_dir",
    "aliases" : [ ],
    "value" : "$MELTANO_PROJECT_ROOT/transform",
    "kind" : "STRING",
    "protected" : false
  }, {
    "name" : "profiles_dir",
    "aliases" : [ ],
    "value" : "$MELTANO_PROJECT_ROOT/transform/profile",
    "kind" : "STRING",
    "env" : "DBT_PROFILES_DIR",
    "protected" : false
  }, {
    "name" : "target",
    "aliases" : [ ],
    "value" : "$MELTANO_LOAD__DIALECT",
    "kind" : "STRING",
    "protected" : false
  }, {
    "name" : "source_schema",
    "aliases" : [ ],
    "value" : "$MELTANO_LOAD__TARGET_SCHEMA",
    "kind" : "STRING",
    "protected" : false
  }, {
    "name" : "target_schema",
    "aliases" : [ ],
    "value" : "analytics",
    "kind" : "STRING",
    "protected" : false
  }, {
    "name" : "models",
    "aliases" : [ ],
    "value" : "$MELTANO_TRANSFORM__PACKAGE_NAME $MELTANO_EXTRACTOR_NAMESPACE my_meltano_project",
    "kind" : "STRING",
    "protected" : false
  } ],
  "variants" : [ ],
  "commands" : {
    "compile" : {
      "args" : "compile",
      "description" : "Generates executable SQL from source model, test, and analysis files. Compiled SQL files are written to the target/ directory."
    },
    "seed" : {
      "args" : "seed",
      "description" : "Load data from csv files into your data warehouse."
    },
    "test" : {
      "args" : "test",
      "description" : "Runs tests on data in deployed models."
    },
    "deps" : {
      "args" : "deps",
      "description" : "Pull the most recent version of the dependencies listed in packages.yml"
    },
    "run" : {
      "args" : "run",
      "description" : "Compile SQL and execute against the current target database."
    },
    "clean" : {
      "args" : "clean",
      "description" : "Delete all folders in the clean-targets list (usually the dbt_modules and target directories.)"
    },
    "snapshot" : {
      "args" : "snapshot",
      "description" : "Execute snapshots defined in your project."
    }
  },
  "matatikaHidden" : false,
  "_links" : {
    "self" : {
      "href" : "https://catalog.matatika.com/api/dataplugins/549aab3e-1c5a-4b54-ba08-8491a27cecc1"
    },
    "update dataplugin" : {
      "href" : "https://catalog.matatika.com/api/workspaces/374ebea3-6784-403f-8722-db2002c8fd83/dataplugins/549aab3e-1c5a-4b54-ba08-8491a27cecc1",
      "type" : "PUT"
    }
  }
}

Setting

Path Type Format Description
name String   The setting name
value String   The setting default value
label String   The setting label
protected Boolean   The setting protection status
kind String Setting Kind The setting kind
description String   A description of the setting
placeholder String   The setting placeholder text
envAliases Array of String   Environment variable aliases for the setting
documentation String URL The setting documentation URL
oauth OAuth   The setting OAuth configuration
env String    

OAuth

Path Type Format Description
provider String   The OAuth provider

Formats

Setting Kind

String

Value Description
STRING String setting
INTEGER Integer setting
PASSWORD Password setting
HIDDEN Hidden setting
BOOLEAN Boolean setting
DATE_ISO8601 ISO 8601 date setting
EMAIL Email setting
OAUTH OAuth setting
FILE File setting
ARRAY Array setting

Requests


View all supported dataplugins

GET

/api/dataplugins

Returns all dataplugins supported by Matatika.

Request

Example Snippets

  • cURL
curl -H "Authorization: Bearer $ACCESS_TOKEN" 'https://catalog.matatika.com/api/dataplugins' -i -X GET \
    -H 'Accept: application/json, application/javascript, text/javascript, text/json' \
    -H 'Content-Type: application/json'
  • Python (requests)
import requests

url = "https://catalog.matatika.com/api/dataplugins"

headers = {
  'Authorization': ACCESS_TOKEN
}

response = requests.request("GET", url, headers=headers)

print(response.text.encode('utf8'))

Response

200 OK

Dataplugin collection with HAL links.

{
  "_embedded" : {
    "dataplugins" : [ {
      "id" : "720766bb-1eeb-4c36-8268-a35e00a1b75f",
      "pluginType" : "EXTRACTOR",
      "name" : "tap-auth0",
      "namespace" : "tap_auth0",
      "variant" : "meltano",
      "label" : "Auth0",
      "description" : "Auth0 is an identity and access management service, that allows you to manage the access of your users.\n\n## Prerequisites\n\nTo find out how to get the required credentials for this data source please go to our documentation: [Auth0 Prerequisites](https://www.matatika.com/docs/data-sources/tap-auth0/prerequisites)\n\n## Data source settings\n\n### Client ID\n\nYour Auth0 Machine-to-Machine application client id.\n\n### Client Secret\n\nYour Auth0 Machine-to-Machine application client secret.\n\n### Domain\n\nYour Auth0 tenant domain in the format `<TENANT_NAME>.<REGION_IDENTIFIER>.auth0.com``\n\n## Learn more\n\n[Matatika Auth0 Data Source Documentation](https://www.matatika.com/docs/data-sources/tap-auth0)\n\n[Meltano Hub Auth0 Documentation](https://hub.meltano.com/taps/auth0)",
      "logoUrl" : "/assets/images/datasource/tap-auth0.png",
      "hidden" : false,
      "pipUrl" : "git+https://github.com/Matatika/[email protected]",
      "repo" : "https://github.com/Matatika/tap-auth0",
      "capabilities" : [ "CATALOG", "DISCOVER", "STATE" ],
      "select" : [ ],
      "update" : { },
      "vars" : { },
      "settings" : [ {
        "name" : "client_id",
        "aliases" : [ ],
        "label" : "Client ID",
        "kind" : "PASSWORD",
        "protected" : false
      }, {
        "name" : "client_secret",
        "aliases" : [ ],
        "label" : "Client Secret",
        "kind" : "PASSWORD",
        "protected" : false
      }, {
        "name" : "domain",
        "aliases" : [ ],
        "label" : "Domain",
        "kind" : "STRING",
        "protected" : false
      }, {
        "name" : "job_poll_interval_ms",
        "aliases" : [ ],
        "value" : "2000",
        "kind" : "HIDDEN",
        "required" : "false",
        "protected" : false
      }, {
        "name" : "job_poll_max_count",
        "aliases" : [ ],
        "value" : "10",
        "kind" : "HIDDEN",
        "required" : "false",
        "protected" : false
      } ],
      "variants" : [ ],
      "commands" : { },
      "matatikaHidden" : false,
      "_links" : {
        "self" : {
          "href" : "https://catalog.matatika.com/api/dataplugins/720766bb-1eeb-4c36-8268-a35e00a1b75f"
        }
      }
    }, {
      "id" : "724fdc95-6395-4068-b74b-e6716c297887",
      "pluginType" : "TRANSFORM",
      "name" : "dbt-googleads",
      "namespace" : "tap_googleads",
      "variant" : "meltano",
      "hidden" : false,
      "pipUrl" : "https://github.com/Matatika/[email protected]",
      "repo" : "https://github.com/Matatika/dbt-tap-googleads",
      "capabilities" : [ ],
      "select" : [ ],
      "update" : { },
      "vars" : {
        "schema" : ""
      },
      "settings" : [ ],
      "variants" : [ ],
      "commands" : { },
      "matatikaHidden" : false,
      "_links" : {
        "self" : {
          "href" : "https://catalog.matatika.com/api/dataplugins/724fdc95-6395-4068-b74b-e6716c297887"
        }
      }
    }, {
      "id" : "1149bda6-c93f-4db6-a22c-f95afd60d575",
      "pluginType" : "FILE",
      "name" : "analyze-sit",
      "namespace" : "tap_matatika_sit",
      "variant" : "matatika",
      "hidden" : false,
      "pipUrl" : "git+https://github.com/Matatika/analyze-sit.git",
      "repo" : "https://github.com/Matatika/analyze-sit",
      "capabilities" : [ ],
      "select" : [ ],
      "update" : {
        "analyze/datasets/tap-matatika-sit/user-ages.yml" : "true",
        "analyze/datasets/tap-matatika-sit/user-genders.yml" : "true"
      },
      "vars" : { },
      "settings" : [ ],
      "variants" : [ ],
      "commands" : { },
      "matatikaHidden" : false,
      "_links" : {
        "self" : {
          "href" : "https://catalog.matatika.com/api/dataplugins/1149bda6-c93f-4db6-a22c-f95afd60d575"
        }
      }
    }, {
      "id" : "4011fed4-14f0-4ebb-a196-8f99c987948d",
      "pluginType" : "EXTRACTOR",
      "name" : "tap-google-analytics",
      "namespace" : "tap_google_analytics",
      "variant" : "meltano",
      "label" : "Google Analytics",
      "description" : "App and website analytics platform hosted by Google\n\n## Prerequisites\n\nBefore importing data from Google Analytics you or your webmaster will need to have performed some initial setup to grant\nyourself access to the Google Analytics for your website.\n\n[Google Analytics Prerequisites](https://www.matatika.com/docs/data-sources/tap-google-analytics/prerequisites)\n\n## Data source settings\n### View ID\n\nOnce you have connected with your google account, the `View ID` drop down will allow you to choose which view you want to get your data from.\n\n### Start Date\n\nProviding a start date allows you to set how much historical data you wish to import into you workspace.\n\nThis date has to be in the ISO-8601 date format, see the example below.\n\nFormat: `YYYY-MM-DDTHH:MM:SSZ`\n\nExample: `2020-10-01T13:12:11Z`\n\n### End Date\n\nProviding an end date allows you to limit how much data you wish to import into you workspace.\n\nThis date has to be in the ISO-8601 date format, see the example below.\n\nFormat: `YYYY-MM-DDTHH:MM:SSZ`\n\nExample: `2020-10-01T13:12:11Z`\n\n### Reports\n\nProviding the path to a reports json file allows you to supply the tap with a user defined definition of tables and information you want to sync from Google Analytics.\nThis does require you to commit a file to your workspace repository so that Google Analytics can find it during run time, and may also break the dbt transforms that are required to have our included datasets work.\nFor more information on how to define a custom reports file see the [Meltano Hub Documentation](https://hub.meltano.com/extractors/google-analytics#reports).\n\n## Learn more\n\n[Matatika Google Analytics Data Source Documentation](https://www.matatika.com/docs/data-sources/tap-google-analytics)\n\n[Meltano Hub Google Analytics Documentation](https://hub.meltano.com/extractors/google-analytics)",
      "logoUrl" : "/assets/images/datasource/tap-google-analytics.svg",
      "hidden" : false,
      "docs" : "https://www.matatika.com/docs/data-sources/tap-google-analytics/",
      "pipUrl" : "git+https://github.com/Matatika/[email protected]",
      "repo" : "https://github.com/Matatika/tap-google-analytics",
      "capabilities" : [ "CATALOG", "DISCOVER", "STATE" ],
      "select" : [ ],
      "update" : { },
      "vars" : { },
      "settings" : [ {
        "name" : "oauth_credentials.authorization_url",
        "aliases" : [ ],
        "label" : "OAuth identity provider authorization endpoint used create and refresh tokens",
        "value" : "https://oauth2.googleapis.com/token",
        "kind" : "HIDDEN",
        "protected" : false
      }, {
        "name" : "oauth_credentials.scope",
        "aliases" : [ ],
        "label" : "OAuth scopes we need to request access to",
        "value" : "profile email https://www.googleapis.com/auth/analytics.readonly",
        "kind" : "HIDDEN",
        "protected" : false
      }, {
        "name" : "oauth_credentials.access_token",
        "aliases" : [ ],
        "label" : "Access Token",
        "kind" : "HIDDEN",
        "protected" : false
      }, {
        "name" : "oauth_credentials.refresh_token",
        "aliases" : [ ],
        "label" : "OAuth Refresh Token",
        "kind" : "HIDDEN",
        "protected" : false
      }, {
        "name" : "oauth_credentials.refresh_proxy_url",
        "aliases" : [ ],
        "label" : "Optional - will be called with 'oauth_credentials.refresh_token' to refresh the access token",
        "kind" : "HIDDEN",
        "required" : "false",
        "protected" : false
      }, {
        "name" : "oauth_credentials.refresh_proxy_url_auth",
        "aliases" : [ ],
        "label" : "Optional - Sets Authorization header on 'oauth_credentials.refresh_url' request",
        "kind" : "HIDDEN",
        "required" : "false",
        "protected" : false
      }, {
        "name" : "oauth_credentials.client_id",
        "aliases" : [ ],
        "label" : "Optional - OAuth Client ID used if refresh_proxy_url not supplied",
        "kind" : "HIDDEN",
        "required" : "false",
        "protected" : false
      }, {
        "name" : "oauth_credentials.client_secret",
        "aliases" : [ ],
        "label" : "Optional - OAuth Client Secret used if refresh_proxy_url not supplied",
        "kind" : "HIDDEN",
        "required" : "false",
        "protected" : false
      }, {
        "name" : "view_id",
        "aliases" : [ ],
        "label" : "View ID",
        "placeholder" : "Ex. 198343027",
        "kind" : "STRING",
        "protected" : false
      }, {
        "name" : "reports",
        "aliases" : [ ],
        "label" : "Reports",
        "placeholder" : "Ex. my_report_definition.json",
        "kind" : "STRING",
        "required" : "false",
        "protected" : false
      }, {
        "name" : "start_date",
        "aliases" : [ ],
        "kind" : "DATE_ISO8601",
        "protected" : false
      }, {
        "name" : "end_date",
        "aliases" : [ ],
        "kind" : "DATE_ISO8601",
        "required" : "false",
        "protected" : false
      } ],
      "variants" : [ ],
      "commands" : { },
      "matatikaHidden" : false,
      "_links" : {
        "self" : {
          "href" : "https://catalog.matatika.com/api/dataplugins/4011fed4-14f0-4ebb-a196-8f99c987948d"
        }
      }
    }, {
      "id" : "7f87c991-733a-4068-9277-7e4b3030893a",
      "pluginType" : "TRANSFORM",
      "name" : "dbt-google-analytics",
      "namespace" : "tap_google_analytics",
      "variant" : "meltano",
      "hidden" : false,
      "pipUrl" : "https://github.com/Matatika/[email protected]",
      "repo" : "https://github.com/Matatika/dbt-tap-google-analytics",
      "capabilities" : [ ],
      "select" : [ ],
      "update" : { },
      "vars" : {
        "schema" : ""
      },
      "settings" : [ ],
      "variants" : [ ],
      "commands" : { },
      "matatikaHidden" : false,
      "_links" : {
        "self" : {
          "href" : "https://catalog.matatika.com/api/dataplugins/7f87c991-733a-4068-9277-7e4b3030893a"
        }
      }
    }, {
      "id" : "4f3acdb4-898b-4ddf-a70f-1141f7b73129",
      "pluginType" : "TRANSFORM",
      "name" : "dbt-solarvista",
      "namespace" : "tap_solarvista",
      "variant" : "matatika",
      "hidden" : false,
      "pipUrl" : "https://github.com/Matatika/[email protected]",
      "repo" : "https://github.com/Matatika/dbt-tap-solarvista",
      "capabilities" : [ ],
      "select" : [ ],
      "update" : { },
      "vars" : {
        "schema" : ""
      },
      "settings" : [ ],
      "variants" : [ ],
      "commands" : { },
      "matatikaHidden" : false,
      "_links" : {
        "self" : {
          "href" : "https://catalog.matatika.com/api/dataplugins/4f3acdb4-898b-4ddf-a70f-1141f7b73129"
        }
      }
    }, {
      "id" : "6472b907-3f72-4456-9ce3-dd97236ba84f",
      "pluginType" : "FILE",
      "name" : "analyze-google-analytics",
      "namespace" : "tap_google_analytics",
      "variant" : "matatika",
      "hidden" : false,
      "pipUrl" : "git+https://github.com/Matatika/[email protected]",
      "repo" : "https://github.com/Matatika/analyze-google-analytics",
      "capabilities" : [ ],
      "select" : [ ],
      "update" : {
        "analyze/datasets" : "true"
      },
      "vars" : { },
      "settings" : [ ],
      "variants" : [ ],
      "commands" : { },
      "matatikaHidden" : false,
      "_links" : {
        "self" : {
          "href" : "https://catalog.matatika.com/api/dataplugins/6472b907-3f72-4456-9ce3-dd97236ba84f"
        }
      }
    }, {
      "id" : "76424447-36fb-49c6-b594-6cd441cb5964",
      "pluginType" : "FILE",
      "name" : "files-dbt",
      "namespace" : "dbt",
      "variant" : "dbt",
      "logoUrl" : "/assets/images/transformer/dbt.png",
      "hidden" : false,
      "pipUrl" : "git+https://github.com/Matatika/[email protected]",
      "repo" : "https://github.com/Matatika/files-dbt",
      "capabilities" : [ ],
      "select" : [ ],
      "update" : {
        "transform/profile/profiles.yml" : "true"
      },
      "vars" : { },
      "settings" : [ ],
      "variants" : [ ],
      "commands" : { },
      "matatikaHidden" : false,
      "_links" : {
        "self" : {
          "href" : "https://catalog.matatika.com/api/dataplugins/76424447-36fb-49c6-b594-6cd441cb5964"
        }
      }
    }, {
      "id" : "14518e68-ecda-48c9-9c93-155453d89ef2",
      "pluginType" : "FILE",
      "name" : "analyze-auth0",
      "namespace" : "tap_auth0",
      "variant" : "matatika",
      "hidden" : false,
      "pipUrl" : "git+https://github.com/Matatika/[email protected]",
      "capabilities" : [ ],
      "select" : [ ],
      "update" : {
        "analyze/datasets/tap-auth0" : "true"
      },
      "vars" : { },
      "settings" : [ ],
      "variants" : [ ],
      "commands" : { },
      "matatikaHidden" : false,
      "_links" : {
        "self" : {
          "href" : "https://catalog.matatika.com/api/dataplugins/14518e68-ecda-48c9-9c93-155453d89ef2"
        }
      }
    }, {
      "id" : "dbf87b80-6eb6-483a-90bb-b7a8c094fb3a",
      "pluginType" : "FILE",
      "name" : "analyze-solarvista",
      "namespace" : "tap_solarvista",
      "variant" : "matatika",
      "hidden" : false,
      "pipUrl" : "git+https://github.com/Matatika/[email protected]",
      "repo" : "https://github.com/Matatika/analyze-solarvista",
      "capabilities" : [ ],
      "select" : [ ],
      "update" : {
        "analyze/datasets" : "true"
      },
      "vars" : { },
      "settings" : [ ],
      "variants" : [ ],
      "commands" : { },
      "matatikaHidden" : false,
      "_links" : {
        "self" : {
          "href" : "https://catalog.matatika.com/api/dataplugins/dbf87b80-6eb6-483a-90bb-b7a8c094fb3a"
        }
      }
    }, {
      "id" : "ffd26c88-aa25-4e04-913c-8dd0b22762d1",
      "pluginType" : "FILE",
      "name" : "analyze-trello",
      "namespace" : "tap_trello",
      "variant" : "matatika",
      "hidden" : false,
      "pipUrl" : "git+https://github.com/Matatika/[email protected]",
      "capabilities" : [ ],
      "select" : [ ],
      "update" : {
        "analyze/datasets/tap-trello" : "true"
      },
      "vars" : { },
      "settings" : [ ],
      "variants" : [ ],
      "commands" : { },
      "matatikaHidden" : false,
      "_links" : {
        "self" : {
          "href" : "https://catalog.matatika.com/api/dataplugins/ffd26c88-aa25-4e04-913c-8dd0b22762d1"
        }
      }
    }, {
      "id" : "512c097b-df0e-4437-ba9a-3374557a30d9",
      "pluginType" : "TRANSFORM",
      "name" : "dbt-tap-trello",
      "namespace" : "tap_trello",
      "variant" : "matatika",
      "hidden" : false,
      "pipUrl" : "https://github.com/Matatika/[email protected]",
      "capabilities" : [ ],
      "select" : [ ],
      "update" : { },
      "vars" : {
        "schema" : ""
      },
      "settings" : [ ],
      "variants" : [ ],
      "commands" : { },
      "matatikaHidden" : false,
      "_links" : {
        "self" : {
          "href" : "https://catalog.matatika.com/api/dataplugins/512c097b-df0e-4437-ba9a-3374557a30d9"
        }
      }
    }, {
      "id" : "bdf19f6a-e898-49e6-bb59-8457b33907b1",
      "pluginType" : "EXTRACTOR",
      "name" : "tap-googleads",
      "namespace" : "tap_googleads",
      "variant" : "matatika",
      "label" : "Google Ads",
      "description" : "App and website ads platform hosted by Google.\n\n## Prerequisites\n\n[Google Ads Prerequisites](https://www.matatika.com/docs/data-sources/tap-googleads/prerequisites)\n\n## Data source settings\n\n### Customer Id\n\nCustomer Id is the account id of your Google Ads account with the ads campaigns, and therefore data, that will get synced.\n\nYour Customer Id is usually in the format '111-111-1111' but to make requests to the api we have to remove the dashes.\n\n### Start Date\n\nProviding a start date allows you to set how much historical data you wish to import into you workspace.\n\nIf you do not provide a start date the tap will set a date of one year ago.\n\nThis date has to be in the ISO-8601 date format, see the example below.\n\nFormat: `YYYY-MM-DDTHH:MM:SSZ`\n\nExample: `2020-10-01T13:12:11Z`\n\n### End Date\n\nProviding a end date allows you to set how much historical data you wish to import into you workspace.\n\nIf you do not provide a end date the tap will sync all data up to the current date.\n\nThis date has to be in the ISO-8601 date format, see the example below.\n\nFormat: `YYYY-MM-DDTHH:MM:SSZ`\n\nExample: `2020-10-01T13:12:11Z`\n\n## Learn more\n\n[Matatika Google Ads Data Source Documentation](https://www.matatika.com/docs/data-sources/tap-googleads)\n\n[Meltano Hub Google Ads Documentation](https://hub.meltano.com/taps/googleads)",
      "logoUrl" : "/assets/images/datasource/tap-googleads.svg",
      "hidden" : false,
      "docs" : "https://www.matatika.com/docs/data-sources/tap-googleads/",
      "pipUrl" : "git+https://github.com/Matatika/[email protected]",
      "repo" : "https://github.com/Matatika/tap-googleads",
      "capabilities" : [ "CATALOG", "DISCOVER", "STATE" ],
      "select" : [ ],
      "update" : { },
      "vars" : { },
      "settings" : [ {
        "name" : "oauth_credentials.authorization_url",
        "aliases" : [ ],
        "label" : "OAuth identity provider authorization endpoint used create and refresh tokens",
        "value" : "https://oauth2.googleapis.com/token",
        "kind" : "HIDDEN",
        "protected" : false
      }, {
        "name" : "oauth_credentials.scope",
        "aliases" : [ ],
        "label" : "OAuth scopes we need to request access to",
        "value" : "https://www.googleapis.com/auth/adwords",
        "kind" : "HIDDEN",
        "protected" : false
      }, {
        "name" : "oauth_credentials.access_token",
        "aliases" : [ ],
        "label" : "Access Token",
        "kind" : "HIDDEN",
        "protected" : false
      }, {
        "name" : "oauth_credentials.refresh_token",
        "aliases" : [ ],
        "label" : "OAuth Refresh Token",
        "kind" : "HIDDEN",
        "protected" : false
      }, {
        "name" : "oauth_credentials.refresh_proxy_url",
        "aliases" : [ ],
        "label" : "Will be called with 'oauth_credentials.refresh_token' to refresh the access token",
        "kind" : "HIDDEN",
        "required" : "false",
        "protected" : false
      }, {
        "name" : "oauth_credentials.refresh_proxy_url_auth",
        "aliases" : [ ],
        "label" : "Sets Authorization header on 'oauth_credentials.refresh_url' request",
        "kind" : "HIDDEN",
        "required" : "false",
        "protected" : false
      }, {
        "name" : "oauth_credentials.client_id",
        "aliases" : [ ],
        "label" : "OAuth Client ID",
        "kind" : "HIDDEN",
        "required" : "false",
        "protected" : false
      }, {
        "name" : "oauth_credentials.client_secret",
        "aliases" : [ ],
        "label" : "OAuth Client Secret",
        "kind" : "HIDDEN",
        "required" : "false",
        "protected" : false
      }, {
        "name" : "start_date",
        "aliases" : [ ],
        "label" : "Start Date",
        "kind" : "DATE_ISO8601",
        "required" : "false",
        "protected" : false
      }, {
        "name" : "end_date",
        "aliases" : [ ],
        "label" : "End Date",
        "kind" : "DATE_ISO8601",
        "required" : "false",
        "protected" : false
      }, {
        "name" : "developer_token",
        "aliases" : [ ],
        "label" : "Developer Token",
        "value" : "DYSuW0qdfU5-jti8Zdh1HQ",
        "kind" : "HIDDEN",
        "protected" : false
      }, {
        "name" : "customer_id",
        "aliases" : [ ],
        "label" : "Customer Id",
        "kind" : "STRING",
        "protected" : false
      } ],
      "variants" : [ ],
      "commands" : { },
      "matatikaHidden" : false,
      "_links" : {
        "self" : {
          "href" : "https://catalog.matatika.com/api/dataplugins/bdf19f6a-e898-49e6-bb59-8457b33907b1"
        }
      }
    }, {
      "id" : "931124c6-882f-4f0d-b0ca-6db09f1e1948",
      "pluginType" : "EXTRACTOR",
      "name" : "tap-matatika-sit",
      "namespace" : "tap_matatika_sit",
      "variant" : "matatika",
      "label" : "Matatika SIT",
      "description" : "Test extractor based on tap-spreadsheets-anywhere used during Matatika SIT runs",
      "logoUrl" : "/assets/images/datasource/tap-matatika-sit.svg",
      "hidden" : false,
      "docs" : "https://meltano.com/plugins/extractors/spreadsheets-anywhere.html",
      "pipUrl" : "git+https://github.com/ets/tap-spreadsheets-anywhere.git",
      "repo" : "https://github.com/ets/tap-spreadsheets-anywhere",
      "executable" : "tap-spreadsheets-anywhere",
      "capabilities" : [ "CATALOG", "DISCOVER", "STATE" ],
      "select" : [ ],
      "update" : { },
      "vars" : { },
      "settings" : [ {
        "name" : "tables",
        "aliases" : [ ],
        "value" : "[{\"path\":\"https://raw.githubusercontent.com/Matatika/matatika-examples/master/example_data\",\"name\":\"gitflixusers\",\"pattern\":\"GitFlixUsers.csv\",\"start_date\":\"2021-01-01T00:00:00Z\",\"key_properties\":[\"id\"],\"format\":\"csv\"}]",
        "kind" : "ARRAY",
        "description" : "An array holding json objects that each describe a set of targeted source files. See docs for details.",
        "protected" : false
      } ],
      "variants" : [ ],
      "commands" : { },
      "matatikaHidden" : false,
      "_links" : {
        "self" : {
          "href" : "https://catalog.matatika.com/api/dataplugins/931124c6-882f-4f0d-b0ca-6db09f1e1948"
        }
      }
    }, {
      "id" : "58c0c3c2-94a0-46d9-bfda-3b8905ddb3ab",
      "pluginType" : "EXTRACTOR",
      "name" : "tap-shopify",
      "namespace" : "tap_shopify",
      "variant" : "matatika",
      "label" : "Shopify",
      "description" : "E-commerce platform for online stores and retail point-of-sale systems.\n\n## Prerequisites\n\nBefore importing data from Shopify, you or your store admin may need to perform some\ninitial setup to grant read access.\n\n[Prerequisites](https://www.matatika.com/docs/data-sources/tap-shopify/prerequisites)\n\n## Settings\n\n### Access Token\n\nThe access token to authenticate with the Shopify API.  See Prerequisites.\n\n\n### Store\n\nShopify store id, use the prefix of your admin url.\n\ne.g. https://<your store id>.myshopify.com/admin\n\n\n### Start Date\n\nProvide a start date to control the amount of historical data.\n\nFormat: `YYYY-MM-DDTHH:MM:SSZ`\nExample: `2020-10-01T13:12:11Z`\n\n\n## Learn more\n\nLearn more about the Shopify data source\n\n[Matatika Shopify Data Source Documentation](https://www.matatika.com/docs/data-sources/tap-shopify)",
      "logoUrl" : "/assets/images/datasource/tap-shopify.png",
      "hidden" : false,
      "docs" : "https://www.matatika.com/docs/data-sources/tap-shopify/",
      "pipUrl" : "git+https://github.com/Matatika/[email protected]",
      "repo" : "https://github.com/Matatika/tap-shopify",
      "capabilities" : [ "STATE" ],
      "select" : [ ],
      "update" : { },
      "vars" : { },
      "settings" : [ {
        "name" : "access_token",
        "aliases" : [ ],
        "label" : "Access Token",
        "kind" : "PASSWORD",
        "protected" : false
      }, {
        "name" : "store",
        "aliases" : [ ],
        "label" : "Store",
        "kind" : "STRING",
        "protected" : false
      }, {
        "name" : "start_date",
        "aliases" : [ ],
        "label" : "Start Date",
        "kind" : "DATE_ISO8601",
        "required" : "false",
        "protected" : false
      } ],
      "variants" : [ ],
      "commands" : { },
      "matatikaHidden" : false,
      "_links" : {
        "self" : {
          "href" : "https://catalog.matatika.com/api/dataplugins/58c0c3c2-94a0-46d9-bfda-3b8905ddb3ab"
        }
      }
    }, {
      "id" : "c439b543-9822-4294-b183-d20a13490fb2",
      "pluginType" : "EXTRACTOR",
      "name" : "tap-meltano",
      "namespace" : "tap_meltano",
      "variant" : "matatika",
      "label" : "Meltano",
      "description" : "Meltano is a data pipeline and modular open source data ops platform.\n\nWhen you run a Meltano job and you have a `MELTANO_DATABASE_URI` set, data about that job run is stored in that database.\n\nBy using this data import, and providing an existing `MELTANO_DATABASE_URI`, you will get insights on Meltano job runs.\n\n## Prerequisites\n\n[Meltano Prerequisites](https://www.matatika.com/docs/data-sources/tap-meltano/prerequisites)\n\n## Data source settings\n\n### Meltano Database URI\n\nYour target Meltano project's database URI. By default, (if left blank), this value will be set to your workspace's Meltano database URI.\n\nExample database URI: `postgresql://<username>:<password>@<host>:<port>/<database>?options=-csearch_path%3D<schema>`\n\n## Learn more\n\n[Matatika Meltano Data Source Documentation](https://www.matatika.com/docs/data-sources/tap-meltano)\n\n[Meltano Hub Meltano Documentation](https://hub.meltano.com/taps/meltano)",
      "logoUrl" : "/assets/images/datasource/tap-meltano.png",
      "hidden" : false,
      "docs" : "https://www.matatika.com/docs/data-sources/tap-meltano/",
      "pipUrl" : "git+https://github.com/Matatika/[email protected]",
      "repo" : "https://github.com/Matatika/tap-meltano",
      "capabilities" : [ "CATALOG", "DISCOVER", "STATE" ],
      "select" : [ ],
      "update" : { },
      "vars" : { },
      "settings" : [ {
        "name" : "meltano_database_uri",
        "aliases" : [ ],
        "value" : "$MELTANO_DATABASE_URI",
        "kind" : "PASSWORD",
        "required" : "true",
        "protected" : false
      } ],
      "variants" : [ ],
      "commands" : { },
      "matatikaHidden" : false,
      "_links" : {
        "self" : {
          "href" : "https://catalog.matatika.com/api/dataplugins/c439b543-9822-4294-b183-d20a13490fb2"
        }
      }
    }, {
      "id" : "c59e646a-3306-4da0-83d8-4979d65dba6d",
      "pluginType" : "EXTRACTOR",
      "name" : "tap-spreadsheets-anywhere",
      "namespace" : "tap_spreadsheets_anywhere",
      "variant" : "ets",
      "label" : "Spreadsheets Anywhere",
      "description" : "Spreadsheets anywhere allows you to read data from spreadsheets (CSVs, Excel, JSON) from cloud or local storage.\n\n## Data source settings\n\n### Files\n\nFiles takes JSON objects inside an Array to describe one or more target files.\n\nExample\n\n```json\n[{\n  \"path\":\"https://meltano.com/files\",\n  \"name\":\"gitflixusers\",\n  \"pattern\":\"GitFlixUsers.csv\",\n  \"start_date\":\"2021-01-01T00:00:00Z\",\n  \"key_properties\":[\"id\"],\n  \"format\":\"csv\"\n}]\n```\n\nNote no trailing `/` on the path, or prefixed `/` on the pattern.\n\nSpreadsheets Anywhere will build the URL back up correctly, so in this case it will look for a file at `https://meltano.com/files/GitFlixUsers.csv`\n\nPassing the above array containing JSON to our files field will sync the data of the file `GitFlixUsers.csv` into your datastore under the table name `gitflixusers`.\n\n## Learn more\n\nThese documentation links contain an overview of how Spreadsheets Anywhere works, its settings, capabilities, a list of all supported formats and some more examples.\n\n[Meltano Hub Spreadsheets Anywhere Documentation](https://hub.meltano.com/extractors/spreadsheets-anywhere)\n\n[Spreadsheets Anywhere GitHub Documentation](https://github.com/ets/tap-spreadsheets-anywhere/blob/master/README.md)",
      "logoUrl" : "/assets/images/datasource/tap-spreadsheets-anywhere.png",
      "hidden" : false,
      "docs" : "https://meltano.com/plugins/extractors/spreadsheets-anywhere.html",
      "pipUrl" : "git+https://github.com/ets/tap-spreadsheets-anywhere.git",
      "repo" : "https://github.com/ets/tap-spreadsheets-anywhere",
      "capabilities" : [ "CATALOG", "DISCOVER", "STATE" ],
      "select" : [ ],
      "update" : { },
      "vars" : { },
      "settings" : [ {
        "name" : "tables",
        "aliases" : [ ],
        "kind" : "ARRAY",
        "description" : "An array holding json objects that each describe a set of targeted source files. See docs for details.",
        "protected" : false
      } ],
      "variants" : [ ],
      "commands" : { },
      "matatikaHidden" : false,
      "_links" : {
        "self" : {
          "href" : "https://catalog.matatika.com/api/dataplugins/c59e646a-3306-4da0-83d8-4979d65dba6d"
        }
      }
    }, {
      "id" : "2798a08b-dec4-4f73-bac6-cfaf567302e5",
      "pluginType" : "EXTRACTOR",
      "name" : "tap-spotify",
      "namespace" : "tap_spotify",
      "variant" : "matatika",
      "label" : "Spotify",
      "description" : "Spotify is a digital music streaming service.\n\nBy using this data import you will see insights into your music trends.\n\n## Prerequisites\n\nTo get you Spotify credentials you can follow the steps listed in the [Spotify Prerequisites](https://www.matatika.com/docs/data-sources/tap-spotify/prerequisites).\n\n## Data source settings\n\n### Client ID\n\nYour Spotify Client ID.\n\n### Client Secret\n\nYour Spotify Client Secret.\n\n### Refresh Token\n\nYour Spotify Client Refresh Token.\n\n## Learn more\n\n[Matatika Spotify Data Source Documentation](https://www.matatika.com/docs/data-sources/tap-spotify)\n\n[Meltano Hub Spotify Documentation](https://hub.meltano.com/taps/spotify)",
      "logoUrl" : "/assets/images/datasource/tap-spotify.png",
      "hidden" : false,
      "docs" : "https://www.matatika.com/docs/data-sources/tap-spotify/",
      "pipUrl" : "git+https://github.com/Matatika/[email protected]",
      "repo" : "https://github.com/Matatika/tap-spotify",
      "capabilities" : [ "CATALOG", "DISCOVER", "STATE" ],
      "select" : [ ],
      "update" : { },
      "vars" : { },
      "settings" : [ {
        "name" : "client_id",
        "aliases" : [ ],
        "kind" : "PASSWORD",
        "required" : "true",
        "protected" : false
      }, {
        "name" : "client_secret",
        "aliases" : [ ],
        "kind" : "PASSWORD",
        "required" : "true",
        "protected" : false
      }, {
        "name" : "refresh_token",
        "aliases" : [ ],
        "kind" : "PASSWORD",
        "required" : "true",
        "protected" : false
      } ],
      "variants" : [ ],
      "commands" : { },
      "matatikaHidden" : false,
      "_links" : {
        "self" : {
          "href" : "https://catalog.matatika.com/api/dataplugins/2798a08b-dec4-4f73-bac6-cfaf567302e5"
        }
      }
    }, {
      "id" : "88e9837c-26b9-473f-aef0-0eb3e47acaf0",
      "pluginType" : "EXTRACTOR",
      "name" : "tap-trello",
      "namespace" : "tap_trello",
      "variant" : "meltano",
      "label" : "Trello",
      "description" : "Trello is a collaboration tool that organizes your projects into boards.\n\n## Prerequisites\n\n[Trello Prerequisites](https://www.matatika.com/docs/data-sources/tap-trello/prerequisites)\n\n## Data source settings\n\n### Developer API Key\n\nTo get your Developer API Key:\n\n1. Go to this link: [https://trello.com/app-key](https://trello.com/app-key) and log in if needed.\n2. The Developer API Key will be displayed below the title of the page.\n\n### Access Token\n\nTo get your Access Token:\n\n1. Go to this link: [https://trello.com/app-key](https://trello.com/app-key) and log in if needed.\n2. Below the Developer API Key, click the generate a Token link.\n3. You will now be show a screen about what permissions you are granting to your API key. Click allow at the bottom of the page.\n4. You will then be redirected to a page with a token, this is your Access Token.\n\n## Learn more\n\n[Matatika Trello Data Source Documentation](https://www.matatika.com/docs/data-sources/tap-trello)\n\n[Meltano Hub Trello Documentation](https://hub.meltano.com/taps/trello--matatika)",
      "logoUrl" : "/assets/images/datasource/tap-trello.png",
      "hidden" : false,
      "pipUrl" : "git+https://github.com/Matatika/[email protected]",
      "repo" : "https://github.com/Matatika/tap-trello",
      "capabilities" : [ "CATALOG", "DISCOVER", "STATE" ],
      "select" : [ ],
      "update" : { },
      "vars" : { },
      "settings" : [ {
        "name" : "developer_api_key",
        "aliases" : [ ],
        "label" : "Developer API Key",
        "kind" : "PASSWORD",
        "protected" : false
      }, {
        "name" : "access_token",
        "aliases" : [ ],
        "label" : "Access Token",
        "kind" : "PASSWORD",
        "protected" : false
      }, {
        "name" : "start_date",
        "aliases" : [ ],
        "kind" : "STRING",
        "required" : "False",
        "protected" : false
      } ],
      "variants" : [ ],
      "commands" : { },
      "matatikaHidden" : false,
      "_links" : {
        "self" : {
          "href" : "https://catalog.matatika.com/api/dataplugins/88e9837c-26b9-473f-aef0-0eb3e47acaf0"
        }
      }
    }, {
      "id" : "aec1159a-2110-4a51-b7a7-8e40e98b865f",
      "pluginType" : "LOADER",
      "name" : "target-postgres",
      "namespace" : "postgres_transferwise",
      "variant" : "transferwise",
      "label" : "PostgreSQL",
      "description" : "PostgreSQL Data Warehouse\n\nThe World's Most Advanced Open Source Relational Database\n\n## Settings\n\n### User\nPostgres user\n\n### Password\nPostgres password\n\n### Host\nPostgres host\nExample: `test-host.db.elephantsql.com`\n\n### Port\nPostgres port\nExample: `5432`\n\n### Database Name\nPostgres database name\n\n### Default Target Schema\nName of the schema where tables will be created (no database prefix)",
      "logoUrl" : "/assets/images/datastore/target-postgres.png",
      "hidden" : false,
      "docs" : "https://hub.meltano.com/loaders/postgres.html",
      "pipUrl" : "git+https://github.com/Matatika/[email protected]",
      "repo" : "https://github.com/transferwise/pipelinewise-target-postgres",
      "executable" : "target-postgres",
      "capabilities" : [ ],
      "select" : [ ],
      "update" : { },
      "vars" : { },
      "settings" : [ {
        "name" : "user",
        "aliases" : [ "username" ],
        "label" : "User",
        "kind" : "STRING",
        "description" : "PostgreSQL user",
        "protected" : false
      }, {
        "name" : "password",
        "aliases" : [ ],
        "label" : "Password",
        "kind" : "PASSWORD",
        "description" : "PostgreSQL password",
        "protected" : false
      }, {
        "name" : "host",
        "aliases" : [ "address" ],
        "label" : "Host",
        "kind" : "STRING",
        "description" : "PostgreSQL host",
        "protected" : false
      }, {
        "name" : "port",
        "aliases" : [ ],
        "label" : "Port",
        "value" : "5432",
        "kind" : "INTEGER",
        "description" : "PostgreSQL port",
        "protected" : false
      }, {
        "name" : "dbname",
        "aliases" : [ "database" ],
        "label" : "Database Name",
        "kind" : "STRING",
        "description" : "PostgreSQL database name",
        "protected" : false
      }, {
        "name" : "default_target_schema",
        "aliases" : [ ],
        "label" : "Default Target Schema",
        "value" : "analytics",
        "kind" : "STRING",
        "description" : "Name of the schema where the tables will be created.",
        "protected" : false
      }, {
        "name" : "ssl",
        "aliases" : [ ],
        "label" : "SSL",
        "value" : "false",
        "kind" : "HIDDEN",
        "required" : "false",
        "protected" : false,
        "value_post_processor" : "STRINGIFY"
      }, {
        "name" : "batch_size_rows",
        "aliases" : [ ],
        "label" : "Batch Size Rows",
        "value" : "100000",
        "kind" : "INTEGER",
        "description" : "Maximum number of rows in each batch. At the end of each batch, the rows in the batch are loaded into Postgres.",
        "required" : "false",
        "protected" : false
      }, {
        "name" : "underscore_camel_case_fields",
        "aliases" : [ ],
        "label" : "Underscore Camel Case Fields",
        "value" : "true",
        "kind" : "HIDDEN",
        "description" : "Enable underscoring camel case field names, make the resulting postgres column names the same as from the Meltano variant of target-postgres.",
        "required" : "false",
        "protected" : false
      }, {
        "name" : "flush_all_streams",
        "aliases" : [ ],
        "label" : "Flush All Streams",
        "value" : "false",
        "kind" : "HIDDEN",
        "description" : "Flush and load every stream into Postgres when one batch is full. Warning: This may trigger the COPY command to use files with low number of records.",
        "required" : "false",
        "protected" : false
      }, {
        "name" : "parallelism",
        "aliases" : [ ],
        "label" : "Parallelism",
        "value" : "0",
        "kind" : "HIDDEN",
        "description" : "The number of threads used to flush tables. 0 will create a thread for each stream, up to parallelism_max. -1 will create a thread for each CPU core. Any other positive number will create that number of threads, up to parallelism_max.",
        "required" : "false",
        "protected" : false
      }, {
        "name" : "parallelism_max",
        "aliases" : [ ],
        "label" : "Max Parallelism",
        "value" : "16",
        "kind" : "HIDDEN",
        "description" : "Max number of parallel threads to use when flushing tables.",
        "required" : "false",
        "protected" : false
      }, {
        "name" : "default_target_schema_select_permission",
        "aliases" : [ ],
        "label" : "Default Target Schema Select Permission",
        "kind" : "HIDDEN",
        "description" : "Grant USAGE privilege on newly created schemas and grant SELECT privilege on newly created tables to a specific role or a list of roles. If `schema_mapping` is not defined then every stream sent by the tap is granted accordingly.",
        "required" : "false",
        "protected" : false
      }, {
        "name" : "schema_mapping",
        "aliases" : [ ],
        "label" : "Schema Mapping",
        "kind" : "HIDDEN",
        "description" : "Useful if you want to load multiple streams from one tap to multiple Postgres schemas.\n\nIf the tap sends the `stream_id` in `<schema_name>-<table_name>` format then this option overwrites the `default_target_schema` value.\nNote, that using `schema_mapping` you can overwrite the `default_target_schema_select_permission` value to grant SELECT permissions to different groups per schemas or optionally you can create indices automatically for the replicated tables.\n",
        "required" : "false",
        "protected" : false
      }, {
        "name" : "add_metadata_columns",
        "aliases" : [ ],
        "label" : "Add Metadata Columns",
        "value" : "true",
        "kind" : "HIDDEN",
        "description" : "Metadata columns add extra row level information about data ingestions, (i.e. when was the row read in source, when was inserted or deleted in postgres etc.) Metadata columns are creating automatically by adding extra columns to the tables with a column prefix `_SDC_`. The column names are following the stitch naming conventions documented at https://www.stitchdata.com/docs/data-structure/integration-schemas#sdc-columns. Enabling metadata columns will flag the deleted rows by setting the `_SDC_DELETED_AT` metadata column. Without the `add_metadata_columns` option the deleted rows from singer taps will not be recongisable in Postgres.",
        "required" : "false",
        "protected" : false
      }, {
        "name" : "hard_delete",
        "aliases" : [ ],
        "label" : "Hard Delete",
        "value" : "false",
        "kind" : "HIDDEN",
        "description" : "When `hard_delete` option is true then DELETE SQL commands will be performed in Postgres to delete rows in tables. It's achieved by continuously checking the `_SDC_DELETED_AT` metadata column sent by the singer tap. Due to deleting rows requires metadata columns, `hard_delete` option automatically enables the `add_metadata_columns` option as well.",
        "required" : "false",
        "protected" : false
      }, {
        "name" : "data_flattening_max_level",
        "aliases" : [ ],
        "label" : "Data Flattening Max Level",
        "value" : "10",
        "kind" : "HIDDEN",
        "description" : "Object type RECORD items from taps can be transformed to flattened columns by creating columns automatically. When value is 0 (default) then flattening functionality is turned off.",
        "required" : "false",
        "protected" : false
      }, {
        "name" : "primary_key_required",
        "aliases" : [ ],
        "label" : "Primary Key Required",
        "value" : "false",
        "kind" : "BOOLEAN",
        "description" : "Log based and Incremental replications on tables with no Primary Key cause duplicates when merging UPDATE events. When set to true, stop loading data if no Primary Key is defined.",
        "required" : "false",
        "protected" : false
      }, {
        "name" : "validate_records",
        "aliases" : [ ],
        "label" : "Validate Records",
        "value" : "false",
        "kind" : "BOOLEAN",
        "description" : "Validate every single record message to the corresponding JSON schema. This option is disabled by default and invalid RECORD messages will fail only at load time by Postgres. Enabling this option will detect invalid records earlier but could cause performance degradation.",
        "required" : "false",
        "protected" : false
      }, {
        "name" : "temp_dir",
        "aliases" : [ ],
        "label" : "Temporary Directory",
        "kind" : "HIDDEN",
        "description" : "(Default: platform-dependent) Directory of temporary CSV files with RECORD messages.",
        "required" : "false",
        "protected" : false
      } ],
      "variants" : [ ],
      "commands" : { },
      "matatikaHidden" : false,
      "_links" : {
        "self" : {
          "href" : "https://catalog.matatika.com/api/dataplugins/aec1159a-2110-4a51-b7a7-8e40e98b865f"
        }
      }
    } ]
  },
  "_links" : {
    "first" : {
      "href" : "https://catalog.matatika.com/api/dataplugins?page=0&size=20"
    },
    "self" : {
      "href" : "https://catalog.matatika.com/api/dataplugins?page=0&size=20"
    },
    "next" : {
      "href" : "https://catalog.matatika.com/api/dataplugins?page=1&size=20"
    },
    "last" : {
      "href" : "https://catalog.matatika.com/api/dataplugins?page=1&size=20"
    }
  },
  "page" : {
    "size" : 20,
    "totalElements" : 35,
    "totalPages" : 2,
    "number" : 0
  }
}

View the Matatika discovery.yml

GET

/api/discovery.yml

Returns a Meltano discovery.yml containing all dataplugins supported by Matatika.

Request

Example Snippets

  • cURL
curl -H "Authorization: Bearer $ACCESS_TOKEN" 'https://catalog.matatika.com/api/discovery.yml' -i -X GET \
    -H 'Accept: application/json, application/javascript, text/javascript, text/json' \
    -H 'Content-Type: application/json'
  • Python (requests)
import requests

url = "https://catalog.matatika.com/api/discovery.yml"

headers = {
  'Authorization': ACCESS_TOKEN
}

response = requests.request("GET", url, headers=headers)

print(response.text.encode('utf8'))

Response

200 OK

Meltano discovery.yml.

version: 20
transformers:
  - name: dbt
    namespace: dbt
    variant: dbt
    logo_url: /assets/images/transformer/dbt.png
    docs: https://meltano.com/docs/transforms.html
    repo: https://github.com/dbt-labs/dbt-core
    pip_url: >
      dbt-core~=1.0.0
      dbt-postgres~=1.0.0
      dbt-redshift~=1.0.0
      dbt-snowflake~=1.0.0
      dbt-bigquery~=1.0.0
    settings:
      - name: project_dir
        value: $MELTANO_PROJECT_ROOT/transform
      - name: profiles_dir
        env: DBT_PROFILES_DIR
        value: $MELTANO_PROJECT_ROOT/transform/profile
      - name: target
        value: $MELTANO_LOAD__DIALECT
      - name: source_schema
        value: $MELTANO_LOAD__TARGET_SCHEMA
      - name: target_schema
        value: analytics
      - name: models
        value: $MELTANO_TRANSFORM__PACKAGE_NAME $MELTANO_EXTRACTOR_NAMESPACE my_meltano_project
    commands:
      clean:
        args: clean
        description: Delete all folders in the clean-targets list (usually the dbt_modules and target directories.)
      compile:
        args: compile
        description: Generates executable SQL from source model, test, and analysis files. Compiled SQL files are written to the target/ directory.
      deps:
        args: deps
        description: Pull the most recent version of the dependencies listed in packages.yml
      run:
        args: run
        description: Compile SQL and execute against the current target database.
      seed:
        args: seed
        description: Load data from csv files into your data warehouse.
      snapshot:
        args: snapshot
        description: Execute snapshots defined in your project.
      test:
        args: test
        description: Runs tests on data in deployed models.
files:
  - name: files-dbt
    namespace: dbt
    logo_url: /assets/images/transformer/dbt.png
    update:
      transform/profile/profiles.yml: true
    variant: dbt
    repo: https://github.com/Matatika/files-dbt
    pip_url: git+https://github.com/Matatika/[email protected]1.0.x.1
utilities:
  - name: notebook
    namespace: notebook
    label: Notebook
    logo_url: /assets/images/utility/notebook.png
    variant: notebook
    description: |-
      Utility for running and converting Jupyter notebooks.

      ## Utility settings
      ### Path

      This setting is the location of the Jupyter notebook or directory of notebooks you want to run and or convert.

      ### Format

      If you want to convert your Jupyter notebooks, you can provide a format.

      Currently supported: pdf

      ### Config

      When converting your Jupyter notebooks you can provide additional config to customize the output.

      We have one custom config: no-code. This exclude any code output from executed files and makes the output pdf generally neater.

      You can pass completely custom configs into this setting, for more information around those go to the documentation link in the Learn more section below.

      ## Learn more

      For more information and detail about the above settings, use the links below.

      [Notebook Utility Documentation](https://github.com/Matatika/utility-notebook/blob/master/README.md)
    pip_url: git+https://github.com/Matatika/utility-notebook.git{:target="_blank"}
    hidden: false
    commands:
      run:
        args: run $NOTEBOOK_PATH
        description: Run one/many notebooks, saving their executed state.
      convert:
        args: convert $NOTEBOOK_PATH -f $NOTEBOOK_FORMAT
        description: Convert one/many notebooks to a selected format.
      run-convert:
        args: run $NOTEBOOK_PATH -f $NOTEBOOK_FORMAT
        description: Run and convert one/many notebooks to a selected format.
      run-convert-config:
        args: run $NOTEBOOK_PATH -f $NOTEBOOK_FORMAT -c $NOTEBOOK_CONFIG
        description: Run and convert one/many notebooks to a selected format with a config.
    settings:
      - name: path
        label: Path
      - name: format
        label: Format
        required: false
      - name: config
        label: Config
        required: false
  - name: matatika
    namespace: matatika-utils
    label: Matatika
    logo_url: /assets/images/utility/matatika.png
    variant: matatika
    pip_url: git+https://github.com/Matatika/utility-matatika.git
    hidden: false
    commands:
      schedules:
        args: schedules
        description: Convert Meltano jobs and schedules into Matatika pipeline yamls.
  - name: sendgrid
    namespace: sendgrid
    label: Sendgrid
    logo_url: /assets/images/utility/sendgrid.png
    variant: sendgrid
    description: |-
      Utility for sending emails using sendgrid.

      ## Utility settings
      ### API Key

      Your sendgrid API key.

      ### To Address(es)

      One or more email addresses to send your email to.

      Example: `[email protected]` or `[email protected], support2@matatika.com`

      ### From Address

      The email address you are sending from. This email address needs to be verified in your sendgrid Sender Authentication.

      ### Title

      The title of the email you are sending.

      ### Body

      The body of the email you are sending.

      ### Attachments

      One ore more attachments you want to send with your email.

      Example: `path/to/file.pdf` or `path/to/file.pdf, file_two.pdf`

    pip_url: git+https://github.com/Matatika/utility-sendgrid.git
    hidden: false
    commands:
      send:
        args: " "
        description: Sends sendgrid email.
    settings:
      - name: to_addresses
        label: To Address(es)
      - name: from_address
        label: From Address
      - name: title
        label: Title
      - name: body
        label: Body
      - name: attachments
        label: Attachments
        required: false
      - name: api_key
        label: API Key
        kind: password
  - name: gdrive
    namespace: utility_gdrive
    label: Google Drive
    variant: matatika
    description: |-
      Utility for downloading files from Google Drive. We then recommend using an extractor to move the local files to your destination.

      Please note that these files are note commit to your workspace's repository, so you need to fully process the files you want in a single pipeline run.

      ## Utility settings
      ### GDrive File or Folder ID

      The file or folder ID that you want to sync from Google Drive. If you select a folder it will get all files in the top level of that folder, skipping sub folders.

      You can search for files by typing in this setting field, and the results list updates accordingly.

      You will need to select the file or folder from the list after you have found the one you are looking for, as the results in the list contain additional information that the utility needs.

      ### Output Path

      Optionally provide an output path for the download files to go to.

      ## Learn more

      For more information and detail about the above settings, use the links below.

      [Google Drive Utility Documentation](https://github.com/Matatika/utility-gdrive/blob/master/README.md)
    pip_url: git+https://github.com/Matatika/[email protected]0.2.0
    logo_url: /assets/images/utility/gdrive.png
    hidden: false
    commands:
      run:
        args: $GDRIVE_FILE_ID
        description: Run utility-gdrive to download file or folder of files.
    settings:
      - name: file_id
        label: GDrive File or Folder ID
      - name: output_path
        label: Output Path
        required: false
        env: GDRIVE_OUTPUT_PATH
      - name: oauth_credentials.authorization_url
        kind: hidden
        label: OAuth identity provider authorization endpoint used create and refresh tokens
        value: https://oauth2.googleapis.com/token
      - name: oauth_credentials.scope
        kind: hidden
        label: OAuth scopes we need to request access to
        value: https://www.googleapis.com/auth/drive.readonly
      - name: oauth_credentials.refresh_proxy_url
        kind: hidden
        label: Optional - will be called with 'oauth_credentials.refresh_token' to refresh the access token
        required: false
      - name: oauth_credentials.refresh_proxy_url_auth
        kind: hidden
        label: Optional - Sets Authorization header on 'oauth_credentials.refresh_url' request
        required: false
      - name: oauth_credentials.client_id
        kind: hidden
        label: Optional - OAuth Client ID used if refresh_proxy_url not supplied
        env: GDRIVE_CLIENT_ID
        required: false
      - name: oauth_credentials.client_secret
        kind: hidden
        label: Optional - OAuth Client Secret used if refresh_proxy_url not supplied
        env: GDRIVE_CLIENT_SECRET
        required: false
      - name: oauth_credentials.refresh_token
        kind: hidden
        label: OAuth Refresh Token
        env: GDRIVE_REFRESH_TOKEN
        required: false
      - name: oauth_credentials.access_token
        kind: hidden
        label: Access Token
        env: GDRIVE_ACCESS_TOKEN


View all workspace dataplugins

GET

/api/workspaces/{workspace-id}/dataplugins

Returns all dataplugins available to the workspace {workspace-id}.

Prerequisites

  • Workspace {workspace-id} must exist

Request

Example Snippets

  • cURL
curl -H "Authorization: Bearer $ACCESS_TOKEN" 'https://catalog.matatika.com/api/workspaces/374ebea3-6784-403f-8722-db2002c8fd83/dataplugins' -i -X GET \
    -H 'Accept: application/json, application/javascript, text/javascript, text/json' \
    -H 'Content-Type: application/json'
  • Python (requests)
import requests

url = "https://catalog.matatika.com/api/workspaces/374ebea3-6784-403f-8722-db2002c8fd83/dataplugins"

headers = {
  'Authorization': ACCESS_TOKEN
}

response = requests.request("GET", url, headers=headers)

print(response.text.encode('utf8'))

Response

200 OK

Dataplugin collection with HAL links.

{
  "_embedded" : {
    "dataplugins" : [ {
      "id" : "549aab3e-1c5a-4b54-ba08-8491a27cecc1",
      "pluginType" : "TRANSFORMER",
      "name" : "dbt",
      "namespace" : "dbt",
      "variant" : "dbt",
      "logoUrl" : "/assets/images/transformer/dbt.png",
      "hidden" : false,
      "docs" : "https://meltano.com/docs/transforms.html",
      "pipUrl" : "dbt-core~=1.0.0 dbt-postgres~=1.0.0 dbt-redshift~=1.0.0 dbt-snowflake~=1.0.0 dbt-bigquery~=1.0.0\n",
      "repo" : "https://github.com/dbt-labs/dbt-core",
      "capabilities" : [ ],
      "select" : [ ],
      "update" : { },
      "vars" : { },
      "settings" : [ {
        "name" : "project_dir",
        "aliases" : [ ],
        "value" : "$MELTANO_PROJECT_ROOT/transform",
        "kind" : "STRING",
        "protected" : false
      }, {
        "name" : "profiles_dir",
        "aliases" : [ ],
        "value" : "$MELTANO_PROJECT_ROOT/transform/profile",
        "kind" : "STRING",
        "env" : "DBT_PROFILES_DIR",
        "protected" : false
      }, {
        "name" : "target",
        "aliases" : [ ],
        "value" : "$MELTANO_LOAD__DIALECT",
        "kind" : "STRING",
        "protected" : false
      }, {
        "name" : "source_schema",
        "aliases" : [ ],
        "value" : "$MELTANO_LOAD__TARGET_SCHEMA",
        "kind" : "STRING",
        "protected" : false
      }, {
        "name" : "target_schema",
        "aliases" : [ ],
        "value" : "analytics",
        "kind" : "STRING",
        "protected" : false
      }, {
        "name" : "models",
        "aliases" : [ ],
        "value" : "$MELTANO_TRANSFORM__PACKAGE_NAME $MELTANO_EXTRACTOR_NAMESPACE my_meltano_project",
        "kind" : "STRING",
        "protected" : false
      } ],
      "variants" : [ ],
      "commands" : {
        "compile" : {
          "args" : "compile",
          "description" : "Generates executable SQL from source model, test, and analysis files. Compiled SQL files are written to the target/ directory."
        },
        "seed" : {
          "args" : "seed",
          "description" : "Load data from csv files into your data warehouse."
        },
        "test" : {
          "args" : "test",
          "description" : "Runs tests on data in deployed models."
        },
        "deps" : {
          "args" : "deps",
          "description" : "Pull the most recent version of the dependencies listed in packages.yml"
        },
        "run" : {
          "args" : "run",
          "description" : "Compile SQL and execute against the current target database."
        },
        "clean" : {
          "args" : "clean",
          "description" : "Delete all folders in the clean-targets list (usually the dbt_modules and target directories.)"
        },
        "snapshot" : {
          "args" : "snapshot",
          "description" : "Execute snapshots defined in your project."
        }
      },
      "matatikaHidden" : false,
      "_links" : {
        "self" : {
          "href" : "https://catalog.matatika.com/api/dataplugins/549aab3e-1c5a-4b54-ba08-8491a27cecc1"
        },
        "update dataplugin" : {
          "href" : "https://catalog.matatika.com/api/workspaces/374ebea3-6784-403f-8722-db2002c8fd83/dataplugins/549aab3e-1c5a-4b54-ba08-8491a27cecc1",
          "type" : "PUT"
        }
      }
    }, {
      "id" : "23ee8e46-7278-4271-82fc-699b0e94b357",
      "pluginType" : "LOADER",
      "name" : "target-postgres",
      "namespace" : "postgres_transferwise",
      "variant" : "transferwise",
      "label" : "PostgreSQL",
      "description" : "PostgreSQL Data Warehouse\n\nThe World's Most Advanced Open Source Relational Database\n\n## Settings\n\n### User\nPostgres user\n\n### Password\nPostgres password\n\n### Host\nPostgres host\nExample: `test-host.db.elephantsql.com`\n\n### Port\nPostgres port\nExample: `5432`\n\n### Database Name\nPostgres database name\n\n### Default Target Schema\nName of the schema where tables will be created (no database prefix)",
      "logoUrl" : "/assets/images/datastore/target-postgres.png",
      "hidden" : false,
      "docs" : "https://hub.meltano.com/loaders/postgres.html",
      "pipUrl" : "git+https://github.com/Matatika/[email protected]",
      "repo" : "https://github.com/transferwise/pipelinewise-target-postgres",
      "executable" : "target-postgres",
      "capabilities" : [ ],
      "select" : [ ],
      "update" : { },
      "vars" : { },
      "settings" : [ {
        "name" : "user",
        "aliases" : [ "username" ],
        "label" : "User",
        "kind" : "STRING",
        "description" : "PostgreSQL user",
        "protected" : false
      }, {
        "name" : "password",
        "aliases" : [ ],
        "label" : "Password",
        "kind" : "PASSWORD",
        "description" : "PostgreSQL password",
        "protected" : false
      }, {
        "name" : "host",
        "aliases" : [ "address" ],
        "label" : "Host",
        "kind" : "STRING",
        "description" : "PostgreSQL host",
        "protected" : false
      }, {
        "name" : "port",
        "aliases" : [ ],
        "label" : "Port",
        "value" : "5432",
        "kind" : "INTEGER",
        "description" : "PostgreSQL port",
        "protected" : false
      }, {
        "name" : "dbname",
        "aliases" : [ "database" ],
        "label" : "Database Name",
        "kind" : "STRING",
        "description" : "PostgreSQL database name",
        "protected" : false
      }, {
        "name" : "default_target_schema",
        "aliases" : [ ],
        "label" : "Default Target Schema",
        "value" : "analytics",
        "kind" : "STRING",
        "description" : "Name of the schema where the tables will be created.",
        "protected" : false
      }, {
        "name" : "ssl",
        "aliases" : [ ],
        "label" : "SSL",
        "value" : "false",
        "kind" : "HIDDEN",
        "required" : "false",
        "protected" : false,
        "value_post_processor" : "STRINGIFY"
      }, {
        "name" : "batch_size_rows",
        "aliases" : [ ],
        "label" : "Batch Size Rows",
        "value" : "100000",
        "kind" : "INTEGER",
        "description" : "Maximum number of rows in each batch. At the end of each batch, the rows in the batch are loaded into Postgres.",
        "required" : "false",
        "protected" : false
      }, {
        "name" : "underscore_camel_case_fields",
        "aliases" : [ ],
        "label" : "Underscore Camel Case Fields",
        "value" : "true",
        "kind" : "HIDDEN",
        "description" : "Enable underscoring camel case field names, make the resulting postgres column names the same as from the Meltano variant of target-postgres.",
        "required" : "false",
        "protected" : false
      }, {
        "name" : "flush_all_streams",
        "aliases" : [ ],
        "label" : "Flush All Streams",
        "value" : "false",
        "kind" : "HIDDEN",
        "description" : "Flush and load every stream into Postgres when one batch is full. Warning: This may trigger the COPY command to use files with low number of records.",
        "required" : "false",
        "protected" : false
      }, {
        "name" : "parallelism",
        "aliases" : [ ],
        "label" : "Parallelism",
        "value" : "0",
        "kind" : "HIDDEN",
        "description" : "The number of threads used to flush tables. 0 will create a thread for each stream, up to parallelism_max. -1 will create a thread for each CPU core. Any other positive number will create that number of threads, up to parallelism_max.",
        "required" : "false",
        "protected" : false
      }, {
        "name" : "parallelism_max",
        "aliases" : [ ],
        "label" : "Max Parallelism",
        "value" : "16",
        "kind" : "HIDDEN",
        "description" : "Max number of parallel threads to use when flushing tables.",
        "required" : "false",
        "protected" : false
      }, {
        "name" : "default_target_schema_select_permission",
        "aliases" : [ ],
        "label" : "Default Target Schema Select Permission",
        "kind" : "HIDDEN",
        "description" : "Grant USAGE privilege on newly created schemas and grant SELECT privilege on newly created tables to a specific role or a list of roles. If `schema_mapping` is not defined then every stream sent by the tap is granted accordingly.",
        "required" : "false",
        "protected" : false
      }, {
        "name" : "schema_mapping",
        "aliases" : [ ],
        "label" : "Schema Mapping",
        "kind" : "HIDDEN",
        "description" : "Useful if you want to load multiple streams from one tap to multiple Postgres schemas.\n\nIf the tap sends the `stream_id` in `<schema_name>-<table_name>` format then this option overwrites the `default_target_schema` value.\nNote, that using `schema_mapping` you can overwrite the `default_target_schema_select_permission` value to grant SELECT permissions to different groups per schemas or optionally you can create indices automatically for the replicated tables.\n",
        "required" : "false",
        "protected" : false
      }, {
        "name" : "add_metadata_columns",
        "aliases" : [ ],
        "label" : "Add Metadata Columns",
        "value" : "true",
        "kind" : "HIDDEN",
        "description" : "Metadata columns add extra row level information about data ingestions, (i.e. when was the row read in source, when was inserted or deleted in postgres etc.) Metadata columns are creating automatically by adding extra columns to the tables with a column prefix `_SDC_`. The column names are following the stitch naming conventions documented at https://www.stitchdata.com/docs/data-structure/integration-schemas#sdc-columns. Enabling metadata columns will flag the deleted rows by setting the `_SDC_DELETED_AT` metadata column. Without the `add_metadata_columns` option the deleted rows from singer taps will not be recongisable in Postgres.",
        "required" : "false",
        "protected" : false
      }, {
        "name" : "hard_delete",
        "aliases" : [ ],
        "label" : "Hard Delete",
        "value" : "false",
        "kind" : "HIDDEN",
        "description" : "When `hard_delete` option is true then DELETE SQL commands will be performed in Postgres to delete rows in tables. It's achieved by continuously checking the `_SDC_DELETED_AT` metadata column sent by the singer tap. Due to deleting rows requires metadata columns, `hard_delete` option automatically enables the `add_metadata_columns` option as well.",
        "required" : "false",
        "protected" : false
      }, {
        "name" : "data_flattening_max_level",
        "aliases" : [ ],
        "label" : "Data Flattening Max Level",
        "value" : "10",
        "kind" : "HIDDEN",
        "description" : "Object type RECORD items from taps can be transformed to flattened columns by creating columns automatically. When value is 0 (default) then flattening functionality is turned off.",
        "required" : "false",
        "protected" : false
      }, {
        "name" : "primary_key_required",
        "aliases" : [ ],
        "label" : "Primary Key Required",
        "value" : "false",
        "kind" : "BOOLEAN",
        "description" : "Log based and Incremental replications on tables with no Primary Key cause duplicates when merging UPDATE events. When set to true, stop loading data if no Primary Key is defined.",
        "required" : "false",
        "protected" : false
      }, {
        "name" : "validate_records",
        "aliases" : [ ],
        "label" : "Validate Records",
        "value" : "false",
        "kind" : "BOOLEAN",
        "description" : "Validate every single record message to the corresponding JSON schema. This option is disabled by default and invalid RECORD messages will fail only at load time by Postgres. Enabling this option will detect invalid records earlier but could cause performance degradation.",
        "required" : "false",
        "protected" : false
      }, {
        "name" : "temp_dir",
        "aliases" : [ ],
        "label" : "Temporary Directory",
        "kind" : "HIDDEN",
        "description" : "(Default: platform-dependent) Directory of temporary CSV files with RECORD messages.",
        "required" : "false",
        "protected" : false
      } ],
      "variants" : [ ],
      "commands" : { },
      "matatikaHidden" : false,
      "_links" : {
        "self" : {
          "href" : "https://catalog.matatika.com/api/dataplugins/23ee8e46-7278-4271-82fc-699b0e94b357"
        },
        "update dataplugin" : {
          "href" : "https://catalog.matatika.com/api/workspaces/374ebea3-6784-403f-8722-db2002c8fd83/dataplugins/23ee8e46-7278-4271-82fc-699b0e94b357",
          "type" : "PUT"
        }
      }
    }, {
      "id" : "ca779797-f25f-457a-95cf-801bedfd65c2",
      "pluginType" : "EXTRACTOR",
      "name" : "tap-custom-test",
      "variant" : "sit",
      "label" : "Tap Custom Test",
      "description" : "A dataplugin created during an SIT run",
      "hidden" : false,
      "pipUrl" : "git+https://github.com/Matatika/example-repository",
      "capabilities" : [ ],
      "select" : [ ],
      "update" : { },
      "vars" : { },
      "settings" : [ {
        "name" : "username",
        "aliases" : [ ],
        "label" : "Username",
        "placeholder" : "username",
        "kind" : "STRING",
        "description" : "The username login credential.",
        "protected" : false
      }, {
        "name" : "email",
        "aliases" : [ ],
        "label" : "Email",
        "placeholder" : "[email protected]",
        "kind" : "EMAIL",
        "description" : "The email login credential.",
        "protected" : false
      }, {
        "name" : "start_date",
        "aliases" : [ ],
        "label" : "Start Date",
        "placeholder" : "2020-01-01T00:00:00Z",
        "kind" : "DATE_ISO8601",
        "description" : "The data to begin extracting data from, in ISO 8601 format.",
        "protected" : false
      } ],
      "variants" : [ ],
      "commands" : { },
      "matatikaHidden" : false,
      "_links" : {
        "self" : {
          "href" : "https://catalog.matatika.com/api/dataplugins/ca779797-f25f-457a-95cf-801bedfd65c2"
        },
        "update dataplugin" : {
          "href" : "https://catalog.matatika.com/api/workspaces/374ebea3-6784-403f-8722-db2002c8fd83/dataplugins/ca779797-f25f-457a-95cf-801bedfd65c2",
          "type" : "PUT"
        },
        "delete dataplugin" : {
          "href" : "https://catalog.matatika.com/api/dataplugins/ca779797-f25f-457a-95cf-801bedfd65c2",
          "type" : "DELETE"
        }
      }
    }, {
      "id" : "82592e17-4191-4a0d-bcb7-f5d1b7487ec3",
      "pluginType" : "FILE",
      "name" : "analyze-test",
      "variant" : "sit",
      "hidden" : false,
      "capabilities" : [ ],
      "select" : [ ],
      "update" : { },
      "vars" : { },
      "settings" : [ ],
      "variants" : [ ],
      "commands" : { },
      "matatikaHidden" : false,
      "_links" : {
        "self" : {
          "href" : "https://catalog.matatika.com/api/dataplugins/82592e17-4191-4a0d-bcb7-f5d1b7487ec3"
        },
        "update dataplugin" : {
          "href" : "https://catalog.matatika.com/api/workspaces/374ebea3-6784-403f-8722-db2002c8fd83/dataplugins/82592e17-4191-4a0d-bcb7-f5d1b7487ec3",
          "type" : "PUT"
        },
        "delete dataplugin" : {
          "href" : "https://catalog.matatika.com/api/dataplugins/82592e17-4191-4a0d-bcb7-f5d1b7487ec3",
          "type" : "DELETE"
        }
      }
    }, {
      "id" : "27fcfbb9-213a-4222-b689-6cfa4ed1f838",
      "pluginType" : "EXTRACTOR",
      "name" : "tap-test",
      "variant" : "sit",
      "hidden" : false,
      "capabilities" : [ ],
      "select" : [ ],
      "update" : { },
      "vars" : { },
      "settings" : [ ],
      "variants" : [ ],
      "commands" : { },
      "matatikaHidden" : false,
      "_links" : {
        "self" : {
          "href" : "https://catalog.matatika.com/api/dataplugins/27fcfbb9-213a-4222-b689-6cfa4ed1f838"
        },
        "update dataplugin" : {
          "href" : "https://catalog.matatika.com/api/workspaces/374ebea3-6784-403f-8722-db2002c8fd83/dataplugins/27fcfbb9-213a-4222-b689-6cfa4ed1f838",
          "type" : "PUT"
        },
        "delete dataplugin" : {
          "href" : "https://catalog.matatika.com/api/dataplugins/27fcfbb9-213a-4222-b689-6cfa4ed1f838",
          "type" : "DELETE"
        }
      }
    }, {
      "id" : "6476b752-45a1-4e2e-81e8-608f906660d2",
      "pluginType" : "LOADER",
      "name" : "target-test",
      "variant" : "sit",
      "hidden" : false,
      "capabilities" : [ ],
      "select" : [ ],
      "update" : { },
      "vars" : { },
      "settings" : [ ],
      "variants" : [ ],
      "commands" : { },
      "matatikaHidden" : false,
      "_links" : {
        "self" : {
          "href" : "https://catalog.matatika.com/api/dataplugins/6476b752-45a1-4e2e-81e8-608f906660d2"
        },
        "update dataplugin" : {
          "href" : "https://catalog.matatika.com/api/workspaces/374ebea3-6784-403f-8722-db2002c8fd83/dataplugins/6476b752-45a1-4e2e-81e8-608f906660d2",
          "type" : "PUT"
        },
        "delete dataplugin" : {
          "href" : "https://catalog.matatika.com/api/dataplugins/6476b752-45a1-4e2e-81e8-608f906660d2",
          "type" : "DELETE"
        }
      }
    }, {
      "id" : "21f9362a-aea3-4f6b-be5f-88e3ca47d468",
      "pluginType" : "TRANSFORM",
      "name" : "dbt-tap-test",
      "variant" : "sit",
      "hidden" : false,
      "capabilities" : [ ],
      "select" : [ ],
      "update" : { },
      "vars" : { },
      "settings" : [ ],
      "variants" : [ ],
      "commands" : { },
      "matatikaHidden" : false,
      "_links" : {
        "self" : {
          "href" : "https://catalog.matatika.com/api/dataplugins/21f9362a-aea3-4f6b-be5f-88e3ca47d468"
        },
        "update dataplugin" : {
          "href" : "https://catalog.matatika.com/api/workspaces/374ebea3-6784-403f-8722-db2002c8fd83/dataplugins/21f9362a-aea3-4f6b-be5f-88e3ca47d468",
          "type" : "PUT"
        },
        "delete dataplugin" : {
          "href" : "https://catalog.matatika.com/api/dataplugins/21f9362a-aea3-4f6b-be5f-88e3ca47d468",
          "type" : "DELETE"
        }
      }
    } ]
  },
  "_links" : {
    "self" : {
      "href" : "https://catalog.matatika.com/api/workspaces/374ebea3-6784-403f-8722-db2002c8fd83/dataplugins?page=0&size=20"
    }
  },
  "page" : {
    "size" : 20,
    "totalElements" : 7,
    "totalPages" : 1,
    "number" : 0
  }
}

View a workspace discovery.yml

GET

/api/workspaces/{workspace-id}/discovery.yml

Returns a Meltano discovery.yml containing all dataplugins available to the workspace {workspace-id}.

Prerequisites

  • Workspace {workspace-id} must exist

Request

Example Snippets

  • cURL
curl -H "Authorization: Bearer $ACCESS_TOKEN" 'https://catalog.matatika.com/api/workspaces/374ebea3-6784-403f-8722-db2002c8fd83/discovery.yml' -i -X GET \
    -H 'Accept: application/json, application/javascript, text/javascript, text/json' \
    -H 'Content-Type: application/json'
  • Python (requests)
import requests

url = "https://catalog.matatika.com/api/workspaces/374ebea3-6784-403f-8722-db2002c8fd83/discovery.yml"

headers = {
  'Authorization': ACCESS_TOKEN
}

response = requests.request("GET", url, headers=headers)

print(response.text.encode('utf8'))

Response

200 OK

Meltano discovery.yml.

version: 20
transformers:
- id: 5a866da0-6a08-4382-a31a-ee870fe4f9ca
  name: dbt
  namespace: dbt
  variant: dbt
  logo_url: /assets/images/transformer/dbt.png
  hidden: false
  docs: https://meltano.com/docs/transforms.html
  pip_url: |
    dbt-core~=1.0.0 dbt-postgres~=1.0.0 dbt-redshift~=1.0.0 dbt-snowflake~=1.0.0 dbt-bigquery~=1.0.0
  repo: https://github.com/dbt-labs/dbt-core
  settings:
  - name: project_dir
    value: $MELTANO_PROJECT_ROOT/transform
    kind: string
    protected: false
  - name: profiles_dir
    value: $MELTANO_PROJECT_ROOT/transform/profile
    kind: string
    env: DBT_PROFILES_DIR
    protected: false
  - name: target
    value: $MELTANO_LOAD__DIALECT
    kind: string
    protected: false
  - name: source_schema
    value: $MELTANO_LOAD__TARGET_SCHEMA
    kind: string
    protected: false
  - name: target_schema
    value: analytics
    kind: string
    protected: false
  - name: models
    value: $MELTANO_TRANSFORM__PACKAGE_NAME $MELTANO_EXTRACTOR_NAMESPACE my_meltano_project
    kind: string
    protected: false
  commands:
    compile:
      args: compile
      description: "Generates executable SQL from source model, test, and analysis files. Compiled SQL files are written to the target/ directory."
    seed:
      args: seed
      description: Load data from csv files into your data warehouse.
    test:
      args: test
      description: Runs tests on data in deployed models.
    deps:
      args: deps
      description: Pull the most recent version of the dependencies listed in packages.yml
    run:
      args: run
      description: Compile SQL and execute against the current target database.
    clean:
      args: clean
      description: Delete all folders in the clean-targets list (usually the dbt_modules and target directories.)
    snapshot:
      args: snapshot
      description: Execute snapshots defined in your project.
files:
- id: 82592e17-4191-4a0d-bcb7-f5d1b7487ec3
  name: analyze-test
  variant: sit
  hidden: false
- id: ffd26c88-aa25-4e04-913c-8dd0b22762d1
  name: analyze-trello
  namespace: tap_trello
  variant: matatika
  hidden: false
  pip_url: git+https://github.com/Matatika/[email protected]0.5.0
  update:
    analyze/datasets/tap-trello: "true"
- id: ea5edd4d-9f41-4b62-a504-a477e0866a2e
  name: analyze-spotify
  namespace: tap_spotify
  variant: matatika
  hidden: false
  pip_url: git+https://github.com/Matatika/[email protected]0.4.1
  repo: https://github.com/Matatika/analyze-spotify
  update:
    analyze/datasets/tap-spotify/export_global_viral_tracks_daily.yml: "true"
    analyze/datasets/tap-spotify/user_top_10_artists_long_term.yml: "true"
    analyze/datasets/tap-spotify/user_top_10_artists_medium_term.yml: "true"
    analyze/datasets/tap-spotify/count_of_all_genres_of_user_artists_long_term.yml: "true"
    analyze/datasets/tap-spotify/global_top_10_tracks_daily.yml: "true"
    analyze/datasets/tap-spotify/export_user_tracks_short_term.yml: "true"
    analyze/datasets/tap-spotify/count_of_artists_by_tracks_played_short_term.yml: "true"
    analyze/datasets/tap-spotify/global_top_10_viral_tracks_daily.yml: "true"
    analyze/datasets/tap-spotify/user_top_10_tracks_short_term.yml: "true"
    analyze/datasets/tap-spotify/export_global_tracks_daily.yml: "true"
    analyze/datasets/tap-spotify/export_global_tracks_weekly.yml: "true"
    analyze/datasets/tap-spotify/user_top_10_tracks_short_term_popularity.yml: "true"
    analyze/datasets/tap-spotify/global_top_10_tracks_weekly.yml: "true"
    analyze/datasets/tap-spotify/populatiry_of_top_25_user_tracks_compared_to_top_25_average_medium_term.yml: "true"
    analyze/datasets/tap-spotify/export_user_artists_short_term.yml: "true"
    analyze/datasets/tap-spotify/user_top_10_artists_short_term.yml: "true"
- id: 36431677-4afa-43a8-83f0-c6d54ff20635
  name: analyze-shopify
  namespace: tap_shopify
  variant: meltano
  hidden: false
  pip_url: git+https://github.com/Matatika/[email protected]0.1.0
  repo: https://github.com/Matatika/analyze-shopify
- id: f3b55dc0-a64c-4b6d-ac05-a50688dbc3ab
  name: analyze-shopify
  namespace: tap_shopify
  variant: matatika
  hidden: false
  pip_url: git+https://github.com/Matatika/[email protected]0.1.0
  repo: https://github.com/Matatika/analyze-shopify
- id: 14518e68-ecda-48c9-9c93-155453d89ef2
  name: analyze-auth0
  namespace: tap_auth0
  variant: matatika
  hidden: false
  pip_url: git+https://github.com/Matatika/analyze-auth0@v0.5.0
  update:
    analyze/datasets/tap-auth0: "true"
- id: 0879ca90-e5ba-49b9-8435-c68676133ac7
  name: analyze-meltano
  namespace: tap_meltano
  variant: matatika
  hidden: false
  pip_url: git+https://github.com/Matatika/[email protected]0.4.0
  repo: https://github.com/Matatika/analyze-meltano
  update:
    analyze/datasets/tap-meltano: "true"
- id: 6472b907-3f72-4456-9ce3-dd97236ba84f
  name: analyze-google-analytics
  namespace: tap_google_analytics
  variant: matatika
  hidden: false
  pip_url: git+https://github.com/Matatika/[email protected]0.4.0
  repo: https://github.com/Matatika/analyze-google-analytics
  update:
    analyze/datasets: "true"
- id: 673509fd-7b5a-40d5-a63b-649e6bbd3c82
  name: analyze-googleads
  namespace: tap_googleads
  variant: meltano
  hidden: false
  pip_url: git+https://github.com/Matatika/[email protected]0.5.0
  repo: https://github.com/Matatika/analyze-googleads
  update:
    analyze/datasets/tap-googleads: "true"
- id: dbf87b80-6eb6-483a-90bb-b7a8c094fb3a
  name: analyze-solarvista
  namespace: tap_solarvista
  variant: matatika
  hidden: false
  pip_url: git+https://github.com/Matatika/[email protected]0.5.0
  repo: https://github.com/Matatika/analyze-solarvista
  update:
    analyze/datasets: "true"
- id: 1149bda6-c93f-4db6-a22c-f95afd60d575
  name: analyze-sit
  namespace: tap_matatika_sit
  variant: matatika
  hidden: false
  pip_url: git+https://github.com/Matatika/analyze-sit.git
  repo: https://github.com/Matatika/analyze-sit
  update:
    analyze/datasets/tap-matatika-sit/user-ages.yml: "true"
    analyze/datasets/tap-matatika-sit/user-genders.yml: "true"
- id: 76424447-36fb-49c6-b594-6cd441cb5964
  name: files-dbt
  namespace: dbt
  variant: dbt
  logo_url: /assets/images/transformer/dbt.png
  hidden: false
  pip_url: git+https://github.com/Matatika/[email protected]1.0.x.1
  repo: https://github.com/Matatika/files-dbt
  update:
    transform/profile/profiles.yml: "true"
utilities:
- id: 8fd16389-2b08-46d1-ad4f-e93d2b57dcf0
  name: sendgrid
  namespace: sendgrid
  variant: sendgrid
  label: Sendgrid
  description: |-
    Utility for sending emails using sendgrid.

    ## Utility settings
    ### API Key

    Your sendgrid API key.

    ### To Address(es)

    One or more email addresses to send your email to.

    Example: `[email protected]` or `[email protected], support2@matatika.com`

    ### From Address

    The email address you are sending from. This email address needs to be verified in your sendgrid Sender Authentication.

    ### Title

    The title of the email you are sending.

    ### Body

    The body of the email you are sending.

    ### Attachments

    One ore more attachments you want to send with your email.

    Example: `path/to/file.pdf` or `path/to/file.pdf, file_two.pdf`
  logo_url: /assets/images/utility/sendgrid.png
  hidden: false
  pip_url: git+https://github.com/Matatika/utility-sendgrid.git
  settings:
  - name: to_addresses
    label: To Address(es)
    kind: string
    protected: false
  - name: from_address
    label: From Address
    kind: string
    protected: false
  - name: title
    label: Title
    kind: string
    protected: false
  - name: body
    label: Body
    kind: string
    protected: false
  - name: attachments
    label: Attachments
    kind: string
    required: "false"
    protected: false
  - name: api_key
    label: API Key
    kind: password
    protected: false
  commands:
    send:
      args: ' '
      description: Sends sendgrid email.
- id: e8484f4d-d247-43e3-8b71-aa3e867533e2
  name: gdrive
  namespace: utility_gdrive
  variant: matatika
  label: Google Drive
  description: |-
    Utility for downloading files from Google Drive. We then recommend using an extractor to move the local files to your destination.

    Please note that these files are note commit to your workspace's repository, so you need to fully process the files you want in a single pipeline run.

    ## Utility settings
    ### GDrive File or Folder ID

    The file or folder ID that you want to sync from Google Drive. If you select a folder it will get all files in the top level of that folder, skipping sub folders.

    You can search for files by typing in this setting field, and the results list updates accordingly.

    You will need to select the file or folder from the list after you have found the one you are looking for, as the results in the list contain additional information that the utility needs.

    ### Output Path

    Optionally provide an output path for the download files to go to.

    ## Learn more

    For more information and detail about the above settings, use the links below.

    [Google Drive Utility Documentation](https://github.com/Matatika/utility-gdrive/blob/master/README.md)
  logo_url: /assets/images/utility/gdrive.png
  hidden: false
  pip_url: git+https://github.com/Matatika/[email protected]0.2.0
  settings:
  - name: file_id
    label: GDrive File or Folder ID
    kind: string
    protected: false
  - name: output_path
    label: Output Path
    kind: string
    env: GDRIVE_OUTPUT_PATH
    required: "false"
    protected: false
  - name: oauth_credentials.authorization_url
    label: OAuth identity provider authorization endpoint used create and refresh tokens
    value: https://oauth2.googleapis.com/token
    kind: hidden
    protected: false
  - name: oauth_credentials.scope
    label: OAuth scopes we need to request access to
    value: https://www.googleapis.com/auth/drive.readonly
    kind: hidden
    protected: false
  - name: oauth_credentials.refresh_proxy_url
    label: Optional - will be called with 'oauth_credentials.refresh_token' to refresh the access token
    kind: hidden
    required: "false"
    protected: false
  - name: oauth_credentials.refresh_proxy_url_auth
    label: Optional - Sets Authorization header on 'oauth_credentials.refresh_url' request
    kind: hidden
    required: "false"
    protected: false
  - name: oauth_credentials.client_id
    label: Optional - OAuth Client ID used if refresh_proxy_url not supplied
    kind: hidden
    env: GDRIVE_CLIENT_ID
    required: "false"
    protected: false
  - name: oauth_credentials.client_secret
    label: Optional - OAuth Client Secret used if refresh_proxy_url not supplied
    kind: hidden
    env: GDRIVE_CLIENT_SECRET
    required: "false"
    protected: false
  - name: oauth_credentials.refresh_token
    label: OAuth Refresh Token
    kind: hidden
    env: GDRIVE_REFRESH_TOKEN
    required: "false"
    protected: false
  - name: oauth_credentials.access_token
    label: Access Token
    kind: hidden
    env: GDRIVE_ACCESS_TOKEN
    protected: false
  commands:
    run:
      args: $GDRIVE_FILE_ID
      description: Run utility-gdrive to download file or folder of files.
- id: 22440cdb-6931-4bc1-b528-3243bc7613bc
  name: matatika
  namespace: matatika-utils
  variant: matatika
  label: Matatika
  logo_url: /assets/images/utility/matatika.png
  hidden: false
  pip_url: git+https://github.com/Matatika/utility-matatika.git
  commands:
    schedules:
      args: schedules
      description: Convert Meltano jobs and schedules into Matatika pipeline yamls.
- id: 85d35bec-b228-4f18-b8d7-b625a07609a9
  name: notebook
  namespace: notebook
  variant: notebook
  label: Notebook
  description: |-
    Utility for running and converting Jupyter notebooks.

    ## Utility settings
    ### Path

    This setting is the location of the Jupyter notebook or directory of notebooks you want to run and or convert.

    ### Format

    If you want to convert your Jupyter notebooks, you can provide a format.

    Currently supported: pdf

    ### Config

    When converting your Jupyter notebooks you can provide additional config to customize the output.

    We have one custom config: no-code. This exclude any code output from executed files and makes the output pdf generally neater.

    You can pass completely custom configs into this setting, for more information around those go to the documentation link in the Learn more section below.

    ## Learn more

    For more information and detail about the above settings, use the links below.

    [Notebook Utility Documentation](https://github.com/Matatika/utility-notebook/blob/master/README.md)
  logo_url: /assets/images/utility/notebook.png
  hidden: false
  pip_url: "git+https://github.com/Matatika/utility-notebook.git{:target=\"_blank\"}"
  settings:
  - name: path
    label: Path
    kind: string
    protected: false
  - name: format
    label: Format
    kind: string
    required: "false"
    protected: false
  - name: config
    label: Config
    kind: string
    required: "false"
    protected: false
  commands:
    run-convert-config:
      args: run $NOTEBOOK_PATH -f $NOTEBOOK_FORMAT -c $NOTEBOOK_CONFIG
      description: Run and convert one/many notebooks to a selected format with a config.
    test:
      args: meltano schedule list --format=json > text.json
      description: "Run one/many notebooks, saving their executed state."
    run:
      args: run $NOTEBOOK_PATH
      description: "Run one/many notebooks, saving their executed state."
    convert:
      args: convert $NOTEBOOK_PATH -f $NOTEBOOK_FORMAT
      description: Convert one/many notebooks to a selected format.
    run-convert:
      args: run $NOTEBOOK_PATH -f $NOTEBOOK_FORMAT
      description: Run and convert one/many notebooks to a selected format.
extractors:
- id: 27fcfbb9-213a-4222-b689-6cfa4ed1f838
  name: tap-test
  variant: sit
  hidden: false
- id: ca779797-f25f-457a-95cf-801bedfd65c2
  name: tap-custom-test
  variant: sit
  label: Tap Custom Test
  description: A dataplugin created during an SIT run
  hidden: false
  pip_url: git+https://github.com/Matatika/example-repository
  settings:
  - name: username
    label: Username
    placeholder: username
    kind: string
    description: The username login credential.
    protected: false
  - name: email
    label: Email
    placeholder: [email protected]
    kind: email
    description: The email login credential.
    protected: false
  - name: start_date
    label: Start Date
    placeholder: 2020-01-01T00:00:00Z
    kind: date_iso8601
    description: "The data to begin extracting data from, in ISO 8601 format."
    protected: false
- id: bdf19f6a-e898-49e6-bb59-8457b33907b1
  name: tap-googleads
  namespace: tap_googleads
  variant: matatika
  label: Google Ads
  description: |-
    App and website ads platform hosted by Google.

    ## Prerequisites

    [Google Ads Prerequisites](https://www.matatika.com/docs/data-sources/tap-googleads/prerequisites)

    ## Data source settings

    ### Customer Id

    Customer Id is the account id of your Google Ads account with the ads campaigns, and therefore data, that will get synced.

    Your Customer Id is usually in the format '111-111-1111' but to make requests to the api we have to remove the dashes.

    ### Start Date

    Providing a start date allows you to set how much historical data you wish to import into you workspace.

    If you do not provide a start date the tap will set a date of one year ago.

    This date has to be in the ISO-8601 date format, see the example below.

    Format: `YYYY-MM-DDTHH:MM:SSZ`

    Example: `2020-10-01T13:12:11Z`

    ### End Date

    Providing a end date allows you to set how much historical data you wish to import into you workspace.

    If you do not provide a end date the tap will sync all data up to the current date.

    This date has to be in the ISO-8601 date format, see the example below.

    Format: `YYYY-MM-DDTHH:MM:SSZ`

    Example: `2020-10-01T13:12:11Z`

    ## Learn more

    [Matatika Google Ads Data Source Documentation](https://www.matatika.com/docs/data-sources/tap-googleads)

    [Meltano Hub Google Ads Documentation](https://hub.meltano.com/taps/googleads)
  logo_url: /assets/images/datasource/tap-googleads.svg
  hidden: false
  docs: https://www.matatika.com/docs/data-sources/tap-googleads/
  pip_url: git+https://github.com/Matatika/[email protected]0.3.1
  repo: https://github.com/Matatika/tap-googleads
  capabilities:
  - catalog
  - discover
  - state
  settings:
  - name: oauth_credentials.authorization_url
    label: OAuth identity provider authorization endpoint used create and refresh tokens
    value: https://oauth2.googleapis.com/token
    kind: hidden
    protected: false
  - name: oauth_credentials.scope
    label: OAuth scopes we need to request access to
    value: https://www.googleapis.com/auth/adwords
    kind: hidden
    protected: false
  - name: oauth_credentials.access_token
    label: Access Token
    kind: hidden
    protected: false
  - name: oauth_credentials.refresh_token
    label: OAuth Refresh Token
    kind: hidden
    protected: false
  - name: oauth_credentials.refresh_proxy_url
    label: Will be called with 'oauth_credentials.refresh_token' to refresh the access token
    kind: hidden
    required: "false"
    protected: false
  - name: oauth_credentials.refresh_proxy_url_auth
    label: Sets Authorization header on 'oauth_credentials.refresh_url' request
    kind: hidden
    required: "false"
    protected: false
  - name: oauth_credentials.client_id
    label: OAuth Client ID
    kind: hidden
    required: "false"
    protected: false
  - name: oauth_credentials.client_secret
    label: OAuth Client Secret
    kind: hidden
    required: "false"
    protected: false
  - name: start_date
    label: Start Date
    kind: date_iso8601
    required: "false"
    protected: false
  - name: end_date
    label: End Date
    kind: date_iso8601
    required: "false"
    protected: false
  - name: developer_token
    label: Developer Token
    value: DYSuW0qdfU5-jti8Zdh1HQ
    kind: hidden
    protected: false
  - name: customer_id
    label: Customer Id
    kind: string
    protected: false
- id: 4011fed4-14f0-4ebb-a196-8f99c987948d
  name: tap-google-analytics
  namespace: tap_google_analytics
  variant: meltano
  label: Google Analytics
  description: |-
    App and website analytics platform hosted by Google

    ## Prerequisites

    Before importing data from Google Analytics you or your webmaster will need to have performed some initial setup to grant
    yourself access to the Google Analytics for your website.

    [Google Analytics Prerequisites](https://www.matatika.com/docs/data-sources/tap-google-analytics/prerequisites)

    ## Data source settings
    ### View ID

    Once you have connected with your google account, the `View ID` drop down will allow you to choose which view you want to get your data from.

    ### Start Date

    Providing a start date allows you to set how much historical data you wish to import into you workspace.

    This date has to be in the ISO-8601 date format, see the example below.

    Format: `YYYY-MM-DDTHH:MM:SSZ`

    Example: `2020-10-01T13:12:11Z`

    ### End Date

    Providing an end date allows you to limit how much data you wish to import into you workspace.

    This date has to be in the ISO-8601 date format, see the example below.

    Format: `YYYY-MM-DDTHH:MM:SSZ`

    Example: `2020-10-01T13:12:11Z`

    ### Reports

    Providing the path to a reports json file allows you to supply the tap with a user defined definition of tables and information you want to sync from Google Analytics.
    This does require you to commit a file to your workspace repository so that Google Analytics can find it during run time, and may also break the dbt transforms that are required to have our included datasets work.
    For more information on how to define a custom reports file see the [Meltano Hub Documentation](https://hub.meltano.com/extractors/google-analytics#reports).

    ## Learn more

    [Matatika Google Analytics Data Source Documentation](https://www.matatika.com/docs/data-sources/tap-google-analytics)

    [Meltano Hub Google Analytics Documentation](https://hub.meltano.com/extractors/google-analytics)
  logo_url: /assets/images/datasource/tap-google-analytics.svg
  hidden: false
  docs: https://www.matatika.com/docs/data-sources/tap-google-analytics/
  pip_url: git+https://github.com/Matatika/[email protected]0.3.0
  repo: https://github.com/Matatika/tap-google-analytics
  capabilities:
  - catalog
  - discover
  - state
  settings:
  - name: oauth_credentials.authorization_url
    label: OAuth identity provider authorization endpoint used create and refresh tokens
    value: https://oauth2.googleapis.com/token
    kind: hidden
    protected: false
  - name: oauth_credentials.scope
    label: OAuth scopes we need to request access to
    value: profile email https://www.googleapis.com/auth/analytics.readonly
    kind: hidden
    protected: false
  - name: oauth_credentials.access_token
    label: Access Token
    kind: hidden
    protected: false
  - name: oauth_credentials.refresh_token
    label: OAuth Refresh Token
    kind: hidden
    protected: false
  - name: oauth_credentials.refresh_proxy_url
    label: Optional - will be called with 'oauth_credentials.refresh_token' to refresh the access token
    kind: hidden
    required: "false"
    protected: false
  - name: oauth_credentials.refresh_proxy_url_auth
    label: Optional - Sets Authorization header on 'oauth_credentials.refresh_url' request
    kind: hidden
    required: "false"
    protected: false
  - name: oauth_credentials.client_id
    label: Optional - OAuth Client ID used if refresh_proxy_url not supplied
    kind: hidden
    required: "false"
    protected: false
  - name: oauth_credentials.client_secret
    label: Optional - OAuth Client Secret used if refresh_proxy_url not supplied
    kind: hidden
    required: "false"
    protected: false
  - name: view_id
    label: View ID
    placeholder: Ex. 198343027
    kind: string
    protected: false
  - name: reports
    label: Reports
    placeholder: Ex. my_report_definition.json
    kind: string
    required: "false"
    protected: false
  - name: start_date
    kind: date_iso8601
    protected: false
  - name: end_date
    kind: date_iso8601
    required: "false"
    protected: false
- id: 58c0c3c2-94a0-46d9-bfda-3b8905ddb3ab
  name: tap-shopify
  namespace: tap_shopify
  variant: matatika
  label: Shopify
  description: |-
    E-commerce platform for online stores and retail point-of-sale systems.

    ## Prerequisites

    Before importing data from Shopify, you or your store admin may need to perform some
    initial setup to grant read access.

    [Prerequisites](https://www.matatika.com/docs/data-sources/tap-shopify/prerequisites)

    ## Settings

    ### Access Token

    The access token to authenticate with the Shopify API.  See Prerequisites.


    ### Store

    Shopify store id, use the prefix of your admin url.

    e.g. https://<your store id>.myshopify.com/admin


    ### Start Date

    Provide a start date to control the amount of historical data.

    Format: `YYYY-MM-DDTHH:MM:SSZ`
    Example: `2020-10-01T13:12:11Z`


    ## Learn more

    Learn more about the Shopify data source

    [Matatika Shopify Data Source Documentation](https://www.matatika.com/docs/data-sources/tap-shopify)
  logo_url: /assets/images/datasource/tap-shopify.png
  hidden: false
  docs: https://www.matatika.com/docs/data-sources/tap-shopify/
  pip_url: git+https://github.com/Matatika/[email protected]0.1.0
  repo: https://github.com/Matatika/tap-shopify
  capabilities:
  - state
  settings:
  - name: access_token
    label: Access Token
    kind: password
    protected: false
  - name: store
    label: Store
    kind: string
    protected: false
  - name: start_date
    label: Start Date
    kind: date_iso8601
    required: "false"
    protected: false
- id: 720766bb-1eeb-4c36-8268-a35e00a1b75f
  name: tap-auth0
  namespace: tap_auth0
  variant: meltano
  label: Auth0
  description: |-
    Auth0 is an identity and access management service, that allows you to manage the access of your users.

    ## Prerequisites

    To find out how to get the required credentials for this data source please go to our documentation: [Auth0 Prerequisites](https://www.matatika.com/docs/data-sources/tap-auth0/prerequisites)

    ## Data source settings

    ### Client ID

    Your Auth0 Machine-to-Machine application client id.

    ### Client Secret

    Your Auth0 Machine-to-Machine application client secret.

    ### Domain

    Your Auth0 tenant domain in the format `<TENANT_NAME>.<REGION_IDENTIFIER>.auth0.com``

    ## Learn more

    [Matatika Auth0 Data Source Documentation](https://www.matatika.com/docs/data-sources/tap-auth0)

    [Meltano Hub Auth0 Documentation](https://hub.meltano.com/taps/auth0)
  logo_url: /assets/images/datasource/tap-auth0.png
  hidden: false
  pip_url: git+https://github.com/Matatika/tap-auth0@v0.3.1
  repo: https://github.com/Matatika/tap-auth0
  capabilities:
  - catalog
  - discover
  - state
  settings:
  - name: client_id
    label: Client ID
    kind: password
    protected: false
  - name: client_secret
    label: Client Secret
    kind: password
    protected: false
  - name: domain
    label: Domain
    kind: string
    protected: false
  - name: job_poll_interval_ms
    value: 2000
    kind: hidden
    required: "false"
    protected: false
  - name: job_poll_max_count
    value: 10
    kind: hidden
    required: "false"
    protected: false
- id: 33444aa0-a5e9-4edb-927a-d0c15707baa0
  name: tap-solarvista
  namespace: tap_solarvista
  variant: matatika
  label: Solarvista Live
  description: |-
    Solarvista Live is a Field Service Management platform to manage, plan, and mobilise your engineers.

    The Matatika Solarvista Live import will extract Solarvista data into your Matatika data store, giving you instant insights into revenue, projects, work items, and engineer performance.

    ## Prerequisites

    Before importing data from Solarvista Live you will need to have performed the initial setup in Solarvista.

    [Solarvista Live Prerequisites](https://www.matatika.com/docs/data-sources/tap-solarvista/prerequisites)


    ## Data source settings

    ### Account

    Solarvista API `account`

    To get your `account` id you will need to follow the steps that are linked in our [Prerequisites](https://www.matatika.com/docs/data-sources/tap-solarvista/prerequisites) section.

    ### ClientId

    Solarvista API `clientId`

    To get your `clientId` you will need to follow the steps that are linked in our [Prerequisites](https://www.matatika.com/docs/data-sources/tap-solarvista/prerequisites) section.

    ### Code

    Solarvista API `code`

    To get your `code` you will need to follow the steps that are linked in our [Prerequisites](https://www.matatika.com/docs/data-sources/tap-solarvista/prerequisites) section.

    ### Datasources

    Leave this field blank to accept the default Solarvista datasources or customise the type of data to sync, we support:
    - `customer`
    - `site`
    - `equipment`
    - `work-item`
    - `work-item-history`
    - `activity`
    - `users`
    - `territory`
    - `project`
    - `appointment`
    - `skill`


    ### Start Date

    Providing a start date allows you to set how much historical data you wish to import into your workspace.

    This date has to be in the ISO-8601 date format, see the example below.

    Format: `YYYY-MM-DDTHH:MM:SSZ`
    Example: `2020-10-01T13:12:11Z`


    ## Learn more

    [Matatika Solavista Live Data Source Documentation](https://www.matatika.com/docs/data-sources/tap-solarvista)

    [Meltano Hub Solarvista Live Documentation](https://hub.meltano.com/taps/solarvista)
  logo_url: /assets/images/datasource/tap-solarvista.png
  hidden: false
  docs: https://www.matatika.com/docs/data-sources/tap-solarvista/
  pip_url: git+https://github.com/Matatika/[email protected]0.3.1
  repo: https://github.com/Matatika/tap-solarvista
  capabilities:
  - state
  settings:
  - name: datasources
    label: Datasources
    kind: string
    protected: false
  - name: account
    label: Account
    kind: string
    protected: false
  - name: clientId
    label: Client ID
    kind: string
    protected: false
  - name: code
    label: Code
    kind: password
    protected: false
  - name: start_date
    label: Start Date
    kind: date_iso8601
    protected: false
- id: c59e646a-3306-4da0-83d8-4979d65dba6d
  name: tap-spreadsheets-anywhere
  namespace: tap_spreadsheets_anywhere
  variant: ets
  label: Spreadsheets Anywhere
  description: |-
    Spreadsheets anywhere allows you to read data from spreadsheets (CSVs, Excel, JSON) from cloud or local storage.

    ## Data source settings

    ### Files

    Files takes JSON objects inside an Array to describe one or more target files.

    Example

    ```json
    [{
      "path":"https://meltano.com/files",
      "name":"gitflixusers",
      "pattern":"GitFlixUsers.csv",
      "start_date":"2021-01-01T00:00:00Z",
      "key_properties":["id"],
      "format":"csv"
    }]
    ```

    Note no trailing `/` on the path, or prefixed `/` on the pattern.

    Spreadsheets Anywhere will build the URL back up correctly, so in this case it will look for a file at `https://meltano.com/files/GitFlixUsers.csv`

    Passing the above array containing JSON to our files field will sync the data of the file `GitFlixUsers.csv` into your datastore under the table name `gitflixusers`.

    ## Learn more

    These documentation links contain an overview of how Spreadsheets Anywhere works, its settings, capabilities, a list of all supported formats and some more examples.

    [Meltano Hub Spreadsheets Anywhere Documentation](https://hub.meltano.com/extractors/spreadsheets-anywhere)

    [Spreadsheets Anywhere GitHub Documentation](https://github.com/ets/tap-spreadsheets-anywhere/blob/master/README.md)
  logo_url: /assets/images/datasource/tap-spreadsheets-anywhere.png
  hidden: false
  docs: https://meltano.com/plugins/extractors/spreadsheets-anywhere.html
  pip_url: git+https://github.com/ets/tap-spreadsheets-anywhere.git
  repo: https://github.com/ets/tap-spreadsheets-anywhere
  capabilities:
  - catalog
  - discover
  - state
  settings:
  - name: tables
    kind: array
    description: An array holding json objects that each describe a set of targeted source files. See docs for details.
    protected: false
- id: 931124c6-882f-4f0d-b0ca-6db09f1e1948
  name: tap-matatika-sit
  namespace: tap_matatika_sit
  variant: matatika
  label: Matatika SIT
  description: Test extractor based on tap-spreadsheets-anywhere used during Matatika SIT runs
  logo_url: /assets/images/datasource/tap-matatika-sit.svg
  hidden: false
  docs: https://meltano.com/plugins/extractors/spreadsheets-anywhere.html
  pip_url: git+https://github.com/ets/tap-spreadsheets-anywhere.git
  repo: https://github.com/ets/tap-spreadsheets-anywhere
  executable: tap-spreadsheets-anywhere
  capabilities:
  - catalog
  - discover
  - state
  settings:
  - name: tables
    value: "[{\"path\":\"https://raw.githubusercontent.com/Matatika/matatika-examples/master/example_data\",\"name\":\"gitflixusers\",\"pattern\":\"GitFlixUsers.csv\",\"start_date\":\"2021-01-01T00:00:00Z\",\"key_properties\":[\"id\"],\"format\":\"csv\"}]"
    kind: array
    description: An array holding json objects that each describe a set of targeted source files. See docs for details.
    protected: false
- id: 88e9837c-26b9-473f-aef0-0eb3e47acaf0
  name: tap-trello
  namespace: tap_trello
  variant: meltano
  label: Trello
  description: |-
    Trello is a collaboration tool that organizes your projects into boards.

    ## Prerequisites

    [Trello Prerequisites](https://www.matatika.com/docs/data-sources/tap-trello/prerequisites)

    ## Data source settings

    ### Developer API Key

    To get your Developer API Key:

    1. Go to this link: [https://trello.com/app-key](https://trello.com/app-key) and log in if needed.
    2. The Developer API Key will be displayed below the title of the page.

    ### Access Token

    To get your Access Token:

    1. Go to this link: [https://trello.com/app-key](https://trello.com/app-key) and log in if needed.
    2. Below the Developer API Key, click the generate a Token link.
    3. You will now be show a screen about what permissions you are granting to your API key. Click allow at the bottom of the page.
    4. You will then be redirected to a page with a token, this is your Access Token.

    ## Learn more

    [Matatika Trello Data Source Documentation](https://www.matatika.com/docs/data-sources/tap-trello)

    [Meltano Hub Trello Documentation](https://hub.meltano.com/taps/trello--matatika)
  logo_url: /assets/images/datasource/tap-trello.png
  hidden: false
  pip_url: git+https://github.com/Matatika/[email protected]0.3.0
  repo: https://github.com/Matatika/tap-trello
  capabilities:
  - catalog
  - discover
  - state
  settings:
  - name: developer_api_key
    label: Developer API Key
    kind: password
    protected: false
  - name: access_token
    label: Access Token
    kind: password
    protected: false
  - name: start_date
    kind: string
    required: "False"
    protected: false
- id: c439b543-9822-4294-b183-d20a13490fb2
  name: tap-meltano
  namespace: tap_meltano
  variant: matatika
  label: Meltano
  description: |-
    Meltano is a data pipeline and modular open source data ops platform.

    When you run a Meltano job and you have a `MELTANO_DATABASE_URI` set, data about that job run is stored in that database.

    By using this data import, and providing an existing `MELTANO_DATABASE_URI`, you will get insights on Meltano job runs.

    ## Prerequisites

    [Meltano Prerequisites](https://www.matatika.com/docs/data-sources/tap-meltano/prerequisites)

    ## Data source settings

    ### Meltano Database URI

    Your target Meltano project's database URI. By default, (if left blank), this value will be set to your workspace's Meltano database URI.

    Example database URI: `postgresql://<username>:<password>@<host>:<port>/<database>?options=-csearch_path%3D<schema>`

    ## Learn more

    [Matatika Meltano Data Source Documentation](https://www.matatika.com/docs/data-sources/tap-meltano)

    [Meltano Hub Meltano Documentation](https://hub.meltano.com/taps/meltano)
  logo_url: /assets/images/datasource/tap-meltano.png
  hidden: false
  docs: https://www.matatika.com/docs/data-sources/tap-meltano/
  pip_url: git+https://github.com/Matatika/[email protected]0.3.0
  repo: https://github.com/Matatika/tap-meltano
  capabilities:
  - catalog
  - discover
  - state
  settings:
  - name: meltano_database_uri
    value: $MELTANO_DATABASE_URI
    kind: password
    required: "true"
    protected: false
- id: 2798a08b-dec4-4f73-bac6-cfaf567302e5
  name: tap-spotify
  namespace: tap_spotify
  variant: matatika
  label: Spotify
  description: |-
    Spotify is a digital music streaming service.

    By using this data import you will see insights into your music trends.

    ## Prerequisites

    To get you Spotify credentials you can follow the steps listed in the [Spotify Prerequisites](https://www.matatika.com/docs/data-sources/tap-spotify/prerequisites).

    ## Data source settings

    ### Client ID

    Your Spotify Client ID.

    ### Client Secret

    Your Spotify Client Secret.

    ### Refresh Token

    Your Spotify Client Refresh Token.

    ## Learn more

    [Matatika Spotify Data Source Documentation](https://www.matatika.com/docs/data-sources/tap-spotify)

    [Meltano Hub Spotify Documentation](https://hub.meltano.com/taps/spotify)
  logo_url: /assets/images/datasource/tap-spotify.png
  hidden: false
  docs: https://www.matatika.com/docs/data-sources/tap-spotify/
  pip_url: git+https://github.com/Matatika/[email protected]0.3.0
  repo: https://github.com/Matatika/tap-spotify
  capabilities:
  - catalog
  - discover
  - state
  settings:
  - name: client_id
    kind: password
    required: "true"
    protected: false
  - name: client_secret
    kind: password
    required: "true"
    protected: false
  - name: refresh_token
    kind: password
    required: "true"
    protected: false
- id: 3ce81fb1-f96b-420a-b368-442975d006e7
  name: tap-google-sheets
  namespace: tap_google_sheets
  variant: meltano
  label: Google Sheets
  description: |-
    Google Sheets is a web-based application that enables users to create, update and modify spreadsheets and share the data online in real time.

    ## Data source settings
    ### Sheet ID - `Required`

    Once you have connected with your google account, the `Sheet ID` drop down will allow you to choose which sheet you want to get your data from.

    ### Stream Name - `Optional`

    Set the stream name to control the table or file name output by the target. By default this will be the file name of your Google Sheet.

    ### Child Sheet Name - `Optional`

    Pick a different sheet within your Google Sheet to sync data from. By default we sync data from the first visible sheet in your Google Sheet.

    ## Learn more

    [Matatika Google Sheets Data Source Documentation](https://www.matatika.com/docs/data-sources/tap-google-sheets)
  logo_url: /assets/images/datasource/tap-google-sheets.png
  hidden: false
  docs: https://www.matatika.com/docs/data-sources/tap-google-sheets/
  pip_url: git+https://github.com/Matatika/[email protected]0.2.0
  repo: https://github.com/Matatika/tap-google-sheets
  capabilities:
  - catalog
  - discover
  - state
  settings:
  - name: oauth_credentials.authorization_url
    label: OAuth identity provider authorization endpoint used create and refresh tokens
    value: https://oauth2.googleapis.com/token
    kind: hidden
    protected: false
  - name: oauth_credentials.scope
    label: OAuth scopes we need to request access to
    value: https://www.googleapis.com/auth/spreadsheets.readonly https://www.googleapis.com/auth/drive.readonly
    kind: hidden
    protected: false
  - name: oauth_credentials.access_token
    label: Access Token
    kind: hidden
    protected: false
  - name: oauth_credentials.refresh_token
    label: OAuth Refresh Token
    kind: hidden
    protected: false
  - name: oauth_credentials.refresh_proxy_url
    label: Optional - will be called with 'oauth_credentials.refresh_token' to refresh the access token
    kind: hidden
    required: "false"
    protected: false
  - name: oauth_credentials.refresh_proxy_url_auth
    label: Optional - Sets Authorization header on 'oauth_credentials.refresh_url' request
    kind: hidden
    required: "false"
    protected: false
  - name: oauth_credentials.client_id
    label: Optional - OAuth Client ID used if refresh_proxy_url not supplied
    kind: hidden
    required: "false"
    protected: false
  - name: oauth_credentials.client_secret
    label: Optional - OAuth Client Secret used if refresh_proxy_url not supplied
    kind: hidden
    required: "false"
    protected: false
  - name: sheet_id
    label: Sheet ID
    kind: string
    protected: false
  - name: stream_name
    label: Stream Name (Optional)
    kind: string
    required: "false"
    protected: false
  - name: child_sheet_name
    label: Child Sheet Name (Optional)
    kind: string
    required: "false"
    protected: false
loaders:
- id: 6476b752-45a1-4e2e-81e8-608f906660d2
  name: target-test
  variant: sit
  hidden: false
- id: aec1159a-2110-4a51-b7a7-8e40e98b865f
  name: target-postgres
  namespace: postgres_transferwise
  variant: transferwise
  label: PostgreSQL
  description: |-
    PostgreSQL Data Warehouse

    The World's Most Advanced Open Source Relational Database

    ## Settings

    ### User
    Postgres user

    ### Password
    Postgres password

    ### Host
    Postgres host
    Example: `test-host.db.elephantsql.com`

    ### Port
    Postgres port
    Example: `5432`

    ### Database Name
    Postgres database name

    ### Default Target Schema
    Name of the schema where tables will be created (no database prefix)
  logo_url: /assets/images/datastore/target-postgres.png
  hidden: false
  docs: https://hub.meltano.com/loaders/postgres.html
  pip_url: git+https://github.com/Matatika/[email protected]0.1.0
  repo: https://github.com/transferwise/pipelinewise-target-postgres
  executable: target-postgres
  settings:
  - name: user
    aliases:
    - username
    label: User
    kind: string
    description: PostgreSQL user
    protected: false
  - name: password
    label: Password
    kind: password
    description: PostgreSQL password
    protected: false
  - name: host
    aliases:
    - address
    label: Host
    kind: string
    description: PostgreSQL host
    protected: false
  - name: port
    label: Port
    value: 5432
    kind: integer
    description: PostgreSQL port
    protected: false
  - name: dbname
    aliases:
    - database
    label: Database Name
    kind: string
    description: PostgreSQL database name
    protected: false
  - name: default_target_schema
    label: Default Target Schema
    value: analytics
    kind: string
    description: Name of the schema where the tables will be created.
    protected: false
  - name: ssl
    label: SSL
    value: "false"
    kind: hidden
    required: "false"
    protected: false
    value_post_processor: stringify
  - name: batch_size_rows
    label: Batch Size Rows
    value: 100000
    kind: integer
    description: "Maximum number of rows in each batch. At the end of each batch, the rows in the batch are loaded into Postgres."
    required: "false"
    protected: false
  - name: underscore_camel_case_fields
    label: Underscore Camel Case Fields
    value: "true"
    kind: hidden
    description: "Enable underscoring camel case field names, make the resulting postgres column names the same as from the Meltano variant of target-postgres."
    required: "false"
    protected: false
  - name: flush_all_streams
    label: Flush All Streams
    value: "false"
    kind: hidden
    description: "Flush and load every stream into Postgres when one batch is full. Warning: This may trigger the COPY command to use files with low number of records."
    required: "false"
    protected: false
  - name: parallelism
    label: Parallelism
    value: 0
    kind: hidden
    description: "The number of threads used to flush tables. 0 will create a thread for each stream, up to parallelism_max. -1 will create a thread for each CPU core. Any other positive number will create that number of threads, up to parallelism_max."
    required: "false"
    protected: false
  - name: parallelism_max
    label: Max Parallelism
    value: 16
    kind: hidden
    description: Max number of parallel threads to use when flushing tables.
    required: "false"
    protected: false
  - name: default_target_schema_select_permission
    label: Default Target Schema Select Permission
    kind: hidden
    description: Grant USAGE privilege on newly created schemas and grant SELECT privilege on newly created tables to a specific role or a list of roles. If `schema_mapping` is not defined then every stream sent by the tap is granted accordingly.
    required: "false"
    protected: false
  - name: schema_mapping
    label: Schema Mapping
    kind: hidden
    description: |
      Useful if you want to load multiple streams from one tap to multiple Postgres schemas.

      If the tap sends the `stream_id` in `<schema_name>-<table_name>` format then this option overwrites the `default_target_schema` value.
      Note, that using `schema_mapping` you can overwrite the `default_target_schema_select_permission` value to grant SELECT permissions to different groups per schemas or optionally you can create indices automatically for the replicated tables.
    required: "false"
    protected: false
  - name: add_metadata_columns
    label: Add Metadata Columns
    value: "true"
    kind: hidden
    description: "Metadata columns add extra row level information about data ingestions, (i.e. when was the row read in source, when was inserted or deleted in postgres etc.) Metadata columns are creating automatically by adding extra columns to the tables with a column prefix `_SDC_`. The column names are following the stitch naming conventions documented at https://www.stitchdata.com/docs/data-structure/integration-schemas#sdc-columns. Enabling metadata columns will flag the deleted rows by setting the `_SDC_DELETED_AT` metadata column. Without the `add_metadata_columns` option the deleted rows from singer taps will not be recongisable in Postgres."
    required: "false"
    protected: false
  - name: hard_delete
    label: Hard Delete
    value: "false"
    kind: hidden
    description: "When `hard_delete` option is true then DELETE SQL commands will be performed in Postgres to delete rows in tables. It's achieved by continuously checking the `_SDC_DELETED_AT` metadata column sent by the singer tap. Due to deleting rows requires metadata columns, `hard_delete` option automatically enables the `add_metadata_columns` option as well."
    required: "false"
    protected: false
  - name: data_flattening_max_level
    label: Data Flattening Max Level
    value: 10
    kind: hidden
    description: Object type RECORD items from taps can be transformed to flattened columns by creating columns automatically. When value is 0 (default) then flattening functionality is turned off.
    required: "false"
    protected: false
  - name: primary_key_required
    label: Primary Key Required
    value: "false"
    kind: boolean
    description: "Log based and Incremental replications on tables with no Primary Key cause duplicates when merging UPDATE events. When set to true, stop loading data if no Primary Key is defined."
    required: "false"
    protected: false
  - name: validate_records
    label: Validate Records
    value: "false"
    kind: boolean
    description: Validate every single record message to the corresponding JSON schema. This option is disabled by default and invalid RECORD messages will fail only at load time by Postgres. Enabling this option will detect invalid records earlier but could cause performance degradation.
    required: "false"
    protected: false
  - name: temp_dir
    label: Temporary Directory
    kind: hidden
    description: "(Default: platform-dependent) Directory of temporary CSV files with RECORD messages."
    required: "false"
    protected: false
- id: f602af3b-f1f5-4bbe-ad9c-1d64d4033b62
  name: target-snowflake
  namespace: snowflake
  variant: meltano
  label: Snowflake
  description: |-
    Snowflake Data Warehouse

    Accelerate your analytics with the data platform built to enable the modern cloud data warehouse.

    ## Settings
    ### Account
    Snowflake account

    Example: `XC93539`

    ### Username
    Snowflake login username

    ### Password
    Snowflake login password

    ### Role
    Name of the role used for loading data - this role is granted to all created schemas and tables

    Example: `PUBLIC`

    ### Database
    Snowflake database name

    ### Warehouse
    Snowflake virtual warehouse name

    Example: `COMPUTE_WH`

    ### Schema
    Name of the schema where tables will be created (no database prefix)

    ### Batch Size
    Maximum number of rows in each [batch](https://docs.snowflake.com/en/sql-reference/external-functions-best-practices.html#process-one-row-at-a-time)

    ### Timestamp Column
    Name of the column used to store timestamps corresponding to when data was loaded into the table
  logo_url: /assets/images/datastore/target-snowflake.png
  hidden: false
  docs: https://hub.meltano.com/loaders/snowflake--meltano.html
  pip_url: git+https://github.com/Matatika/[email protected]0.1.1
  repo: https://github.com/Matatika/target-snowflake
  settings:
  - name: account
    label: Account
    kind: string
    description: Account Name in Snowflake (https://XXXXX.snowflakecomputing.com)
    protected: false
  - name: username
    label: Username
    kind: string
    description: The username you use for logging in
    protected: false
  - name: password
    label: Password
    kind: password
    description: The password you use for logging in
    protected: false
  - name: role
    label: Role
    kind: string
    description: "Role to be used for loading the data, e.g. `LOADER`. Also this role is GRANTed usage to all tables and schemas created"
    protected: false
  - name: database
    label: Database
    kind: string
    description: The name of the Snowflake database you want to use
    protected: false
  - name: warehouse
    label: Warehouse
    kind: string
    description: The name of the Snowflake warehouse you want to use
    protected: false
  - name: schema
    label: Schema
    value: analytics
    kind: string
    protected: false
    value_processor: upcase_string
  - name: batch_size
    label: Batch Size
    value: 5000
    kind: integer
    description: How many records are sent to Snowflake at a time?
    protected: false
  - name: timestamp_column
    label: Timestamp Column
    value: __loaded_at
    kind: string
    description: Name of the column used for recording the timestamp when Data are uploaded to Snowflake.
    protected: false
transforms:
- id: 21f9362a-aea3-4f6b-be5f-88e3ca47d468
  name: dbt-tap-test
  variant: sit
  hidden: false
- id: 7f87c991-733a-4068-9277-7e4b3030893a
  name: dbt-google-analytics
  namespace: tap_google_analytics
  variant: meltano
  hidden: false
  pip_url: https://github.com/Matatika/[email protected]0.4.0
  repo: https://github.com/Matatika/dbt-tap-google-analytics
  vars:
    schema: ""
- id: 724fdc95-6395-4068-b74b-e6716c297887
  name: dbt-googleads
  namespace: tap_googleads
  variant: meltano
  hidden: false
  pip_url: https://github.com/Matatika/[email protected]0.4.0
  repo: https://github.com/Matatika/dbt-tap-googleads
  vars:
    schema: ""
- id: 4f3acdb4-898b-4ddf-a70f-1141f7b73129
  name: dbt-solarvista
  namespace: tap_solarvista
  variant: matatika
  hidden: false
  pip_url: https://github.com/Matatika/[email protected]0.4.0
  repo: https://github.com/Matatika/dbt-tap-solarvista
  vars:
    schema: ""
- id: 0f5da286-225b-4f8d-80f5-47f8239c8f03
  name: dbt-meltano
  namespace: tap_meltano
  variant: meltano
  hidden: false
  pip_url: https://github.com/Matatika/[email protected]0.4.0
  repo: https://github.com/Matatika/dbt-tap-meltano
  vars:
    schema: ""
- id: 41ee7734-ee87-4216-801b-b565eb79a613
  name: dbt-spotify
  namespace: tap_spotify
  variant: spotify
  hidden: false
  pip_url: https://github.com/Matatika/[email protected]0.4.0
  repo: https://github.com/Matatika/dbt-tap-spotify
  vars:
    schema: ""
- id: 512c097b-df0e-4437-ba9a-3374557a30d9
  name: dbt-tap-trello
  namespace: tap_trello
  variant: matatika
  hidden: false
  pip_url: https://github.com/Matatika/[email protected]0.4.0
  vars:
    schema: ""


View a dataplugin

GET

/api/dataplugins/{dataplugin-id}

Returns the dataplugin {dataplugin-id}.

Prerequisites

  • Dataplugin {dataplugin-id} must exist

Request

Example Snippets

  • cURL
curl -H "Authorization: Bearer $ACCESS_TOKEN" 'https://catalog.matatika.com/api/dataplugins/549aab3e-1c5a-4b54-ba08-8491a27cecc1' -i -X GET \
    -H 'Accept: application/json, application/javascript, text/javascript, text/json' \
    -H 'Content-Type: application/json'
  • Python (requests)
import requests

url = "https://catalog.matatika.com/api/dataplugins/549aab3e-1c5a-4b54-ba08-8491a27cecc1"

headers = {
  'Authorization': ACCESS_TOKEN
}

response = requests.request("GET", url, headers=headers)

print(response.text.encode('utf8'))

Response

200 OK

Dataplugin with HAL links.

{
  "id" : "549aab3e-1c5a-4b54-ba08-8491a27cecc1",
  "pluginType" : "TRANSFORMER",
  "name" : "dbt",
  "namespace" : "dbt",
  "variant" : "dbt",
  "logoUrl" : "/assets/images/transformer/dbt.png",
  "hidden" : false,
  "docs" : "https://meltano.com/docs/transforms.html",
  "pipUrl" : "dbt-core~=1.0.0 dbt-postgres~=1.0.0 dbt-redshift~=1.0.0 dbt-snowflake~=1.0.0 dbt-bigquery~=1.0.0\n",
  "repo" : "https://github.com/dbt-labs/dbt-core",
  "capabilities" : [ ],
  "select" : [ ],
  "update" : { },
  "vars" : { },
  "settings" : [ {
    "name" : "project_dir",
    "aliases" : [ ],
    "value" : "$MELTANO_PROJECT_ROOT/transform",
    "kind" : "STRING",
    "protected" : false
  }, {
    "name" : "profiles_dir",
    "aliases" : [ ],
    "value" : "$MELTANO_PROJECT_ROOT/transform/profile",
    "kind" : "STRING",
    "env" : "DBT_PROFILES_DIR",
    "protected" : false
  }, {
    "name" : "target",
    "aliases" : [ ],
    "value" : "$MELTANO_LOAD__DIALECT",
    "kind" : "STRING",
    "protected" : false
  }, {
    "name" : "source_schema",
    "aliases" : [ ],
    "value" : "$MELTANO_LOAD__TARGET_SCHEMA",
    "kind" : "STRING",
    "protected" : false
  }, {
    "name" : "target_schema",
    "aliases" : [ ],
    "value" : "analytics",
    "kind" : "STRING",
    "protected" : false
  }, {
    "name" : "models",
    "aliases" : [ ],
    "value" : "$MELTANO_TRANSFORM__PACKAGE_NAME $MELTANO_EXTRACTOR_NAMESPACE my_meltano_project",
    "kind" : "STRING",
    "protected" : false
  } ],
  "variants" : [ ],
  "commands" : {
    "compile" : {
      "args" : "compile",
      "description" : "Generates executable SQL from source model, test, and analysis files. Compiled SQL files are written to the target/ directory."
    },
    "seed" : {
      "args" : "seed",
      "description" : "Load data from csv files into your data warehouse."
    },
    "test" : {
      "args" : "test",
      "description" : "Runs tests on data in deployed models."
    },
    "deps" : {
      "args" : "deps",
      "description" : "Pull the most recent version of the dependencies listed in packages.yml"
    },
    "run" : {
      "args" : "run",
      "description" : "Compile SQL and execute against the current target database."
    },
    "clean" : {
      "args" : "clean",
      "description" : "Delete all folders in the clean-targets list (usually the dbt_modules and target directories.)"
    },
    "snapshot" : {
      "args" : "snapshot",
      "description" : "Execute snapshots defined in your project."
    }
  },
  "matatikaHidden" : false,
  "_links" : {
    "self" : {
      "href" : "https://catalog.matatika.com/api/dataplugins/549aab3e-1c5a-4b54-ba08-8491a27cecc1"
    },
    "update dataplugin" : {
      "href" : "https://catalog.matatika.com/api/workspaces/374ebea3-6784-403f-8722-db2002c8fd83/dataplugins/549aab3e-1c5a-4b54-ba08-8491a27cecc1",
      "type" : "PUT"
    }
  }
}

Initialise a new dataplugin

POST

/api/dataplugins

Initialises a new dataplugin.

Request

Example Snippets

  • cURL
curl -H "Authorization: Bearer $ACCESS_TOKEN" 'https://catalog.matatika.com/api/workspaces/374ebea3-6784-403f-8722-db2002c8fd83/dataplugins' -i -X POST \
    -H 'Accept: application/json, application/javascript, text/javascript, text/json' \
    -H 'Content-Type: application/json'
  • Python (requests)
import requests

url = "https://catalog.matatika.com/api/workspaces/374ebea3-6784-403f-8722-db2002c8fd83/dataplugins"

headers = {
  'Authorization': ACCESS_TOKEN
}

response = requests.request("POST", url, headers=headers)

print(response.text.encode('utf8'))

Response

200 OK

Dataplugin with HAL links.

{
  "id" : "ca779797-f25f-457a-95cf-801bedfd65c2",
  "hidden" : false,
  "capabilities" : [ ],
  "select" : [ ],
  "update" : { },
  "vars" : { },
  "settings" : [ ],
  "variants" : [ ],
  "commands" : { },
  "matatikaHidden" : false,
  "_links" : {
    "self" : {
      "href" : "https://catalog.matatika.com/api/dataplugins/ca779797-f25f-457a-95cf-801bedfd65c2"
    },
    "create dataplugin" : {
      "href" : "https://catalog.matatika.com/api/workspaces/374ebea3-6784-403f-8722-db2002c8fd83/dataplugins/ca779797-f25f-457a-95cf-801bedfd65c2",
      "type" : "PUT"
    }
  }
}

Publish dataplugins from a discovery.yml

POST

/api/workspaces/{workspace-id}/discovery.yml

Publishes dataplugins from a Meltano discovery.yml.

Prerequisites

  • Workspace {workspace-id} must exist

Request

Body

Meltano discovery.yml

version: 19
extractors:
- name: tap-test
  variant: sit
loaders:
- name: target-test
  variant: sit
transforms:
- name: dbt-tap-test
  variant: sit
files:
- name: analyze-test
  variant: sit


Example Snippets

  • cURL
curl -H "Authorization: Bearer $ACCESS_TOKEN" 'https://catalog.matatika.com/api/workspaces/374ebea3-6784-403f-8722-db2002c8fd83/discovery' -i -X POST \
    -H 'Accept: application/json, application/javascript, text/javascript, text/json' \
    -H 'Content-Type: application/json' \
    -d 'version: 19
extractors:
- name: tap-test
  variant: sit
loaders:
- name: target-test
  variant: sit
transforms:
- name: dbt-tap-test
  variant: sit
files:
- name: analyze-test
  variant: sit

'
  • Python (requests)
import requests

url = "https://catalog.matatika.com/api/workspaces/374ebea3-6784-403f-8722-db2002c8fd83/discovery"

data = version: 19
extractors:
- name: tap-test
  variant: sit
loaders:
- name: target-test
  variant: sit
transforms:
- name: dbt-tap-test
  variant: sit
files:
- name: analyze-test
  variant: sit


headers = {
  'Authorization': ACCESS_TOKEN
}

response = requests.request("POST", url, headers=headers, data=data)

print(response.text.encode('utf8'))

Response

201 Created

Dataplugin collection with HAL links.

{
  "_embedded" : {
    "dataplugins" : [ {
      "id" : "27fcfbb9-213a-4222-b689-6cfa4ed1f838",
      "pluginType" : "EXTRACTOR",
      "name" : "tap-test",
      "variant" : "sit",
      "hidden" : false,
      "capabilities" : [ ],
      "select" : [ ],
      "update" : { },
      "vars" : { },
      "settings" : [ ],
      "variants" : [ ],
      "commands" : { },
      "matatikaHidden" : false,
      "_links" : {
        "self" : {
          "href" : "https://catalog.matatika.com/api/dataplugins/27fcfbb9-213a-4222-b689-6cfa4ed1f838"
        },
        "update dataplugin" : {
          "href" : "https://catalog.matatika.com/api/workspaces/374ebea3-6784-403f-8722-db2002c8fd83/dataplugins/27fcfbb9-213a-4222-b689-6cfa4ed1f838",
          "type" : "PUT"
        },
        "delete dataplugin" : {
          "href" : "https://catalog.matatika.com/api/dataplugins/27fcfbb9-213a-4222-b689-6cfa4ed1f838",
          "type" : "DELETE"
        }
      }
    }, {
      "id" : "6476b752-45a1-4e2e-81e8-608f906660d2",
      "pluginType" : "LOADER",
      "name" : "target-test",
      "variant" : "sit",
      "hidden" : false,
      "capabilities" : [ ],
      "select" : [ ],
      "update" : { },
      "vars" : { },
      "settings" : [ ],
      "variants" : [ ],
      "commands" : { },
      "matatikaHidden" : false,
      "_links" : {
        "self" : {
          "href" : "https://catalog.matatika.com/api/dataplugins/6476b752-45a1-4e2e-81e8-608f906660d2"
        },
        "update dataplugin" : {
          "href" : "https://catalog.matatika.com/api/workspaces/374ebea3-6784-403f-8722-db2002c8fd83/dataplugins/6476b752-45a1-4e2e-81e8-608f906660d2",
          "type" : "PUT"
        },
        "delete dataplugin" : {
          "href" : "https://catalog.matatika.com/api/dataplugins/6476b752-45a1-4e2e-81e8-608f906660d2",
          "type" : "DELETE"
        }
      }
    }, {
      "id" : "21f9362a-aea3-4f6b-be5f-88e3ca47d468",
      "pluginType" : "TRANSFORM",
      "name" : "dbt-tap-test",
      "variant" : "sit",
      "hidden" : false,
      "capabilities" : [ ],
      "select" : [ ],
      "update" : { },
      "vars" : { },
      "settings" : [ ],
      "variants" : [ ],
      "commands" : { },
      "matatikaHidden" : false,
      "_links" : {
        "self" : {
          "href" : "https://catalog.matatika.com/api/dataplugins/21f9362a-aea3-4f6b-be5f-88e3ca47d468"
        },
        "update dataplugin" : {
          "href" : "https://catalog.matatika.com/api/workspaces/374ebea3-6784-403f-8722-db2002c8fd83/dataplugins/21f9362a-aea3-4f6b-be5f-88e3ca47d468",
          "type" : "PUT"
        },
        "delete dataplugin" : {
          "href" : "https://catalog.matatika.com/api/dataplugins/21f9362a-aea3-4f6b-be5f-88e3ca47d468",
          "type" : "DELETE"
        }
      }
    }, {
      "id" : "82592e17-4191-4a0d-bcb7-f5d1b7487ec3",
      "pluginType" : "FILE",
      "name" : "analyze-test",
      "variant" : "sit",
      "hidden" : false,
      "capabilities" : [ ],
      "select" : [ ],
      "update" : { },
      "vars" : { },
      "settings" : [ ],
      "variants" : [ ],
      "commands" : { },
      "matatikaHidden" : false,
      "_links" : {
        "self" : {
          "href" : "https://catalog.matatika.com/api/dataplugins/82592e17-4191-4a0d-bcb7-f5d1b7487ec3"
        },
        "update dataplugin" : {
          "href" : "https://catalog.matatika.com/api/workspaces/374ebea3-6784-403f-8722-db2002c8fd83/dataplugins/82592e17-4191-4a0d-bcb7-f5d1b7487ec3",
          "type" : "PUT"
        },
        "delete dataplugin" : {
          "href" : "https://catalog.matatika.com/api/dataplugins/82592e17-4191-4a0d-bcb7-f5d1b7487ec3",
          "type" : "DELETE"
        }
      }
    } ]
  },
  "_links" : {
    "self" : {
      "href" : "https://catalog.matatika.com/api/dataplugins"
    }
  }
}

Create a dataplugin

PUT

/api/dataplugins/{dataplugin-id}

Creates the dataplugin {dataplugin-id}.

Request

Body

Dataplugin resource.

{
  "name" : "tap-custom-test",
  "variant" : "sit",
  "pluginType" : "EXTRACTOR",
  "label" : "Tap Custom Test",
  "description" : "A dataplugin created during an SIT run",
  "pipUrl" : "git+https://github.com/Matatika/example-repository",
  "settings" : [ {
    "name" : "username",
    "kind" : "STRING",
    "label" : "Username",
    "placeholder" : "username",
    "description" : "The username login credential.",
    "protected" : false
  }, {
    "name" : "email",
    "kind" : "EMAIL",
    "label" : "Email",
    "description" : "The email login credential.",
    "placeholder" : "[email protected]",
    "protected" : false
  }, {
    "name" : "start_date",
    "kind" : "DATE_ISO8601",
    "label" : "Start Date",
    "description" : "The data to begin extracting data from, in ISO 8601 format.",
    "placeholder" : "2020-01-01T00:00:00Z",
    "protected" : false
  } ]
}

Example Snippets

  • cURL
curl -H "Authorization: Bearer $ACCESS_TOKEN" 'https://catalog.matatika.com/api/workspaces/374ebea3-6784-403f-8722-db2002c8fd83/dataplugins/ca779797-f25f-457a-95cf-801bedfd65c2' -i -X PUT \
    -H 'Accept: application/json, application/javascript, text/javascript, text/json' \
    -H 'Content-Type: application/hal+json; charset=ISO-8859-1' \
    -d '{
  "name" : "tap-custom-test",
  "variant" : "sit",
  "pluginType" : "EXTRACTOR",
  "label" : "Tap Custom Test",
  "description" : "A dataplugin created during an SIT run",
  "pipUrl" : "git+https://github.com/Matatika/example-repository",
  "settings" : [ {
    "name" : "username",
    "kind" : "STRING",
    "label" : "Username",
    "placeholder" : "username",
    "description" : "The username login credential.",
    "protected" : false
  }, {
    "name" : "email",
    "kind" : "EMAIL",
    "label" : "Email",
    "description" : "The email login credential.",
    "placeholder" : "[email protected]",
    "protected" : false
  }, {
    "name" : "start_date",
    "kind" : "DATE_ISO8601",
    "label" : "Start Date",
    "description" : "The data to begin extracting data from, in ISO 8601 format.",
    "placeholder" : "2020-01-01T00:00:00Z",
    "protected" : false
  } ]
}'
  • Python (requests)
import requests

url = "https://catalog.matatika.com/api/workspaces/374ebea3-6784-403f-8722-db2002c8fd83/dataplugins/ca779797-f25f-457a-95cf-801bedfd65c2"

data = {
  "name" : "tap-custom-test",
  "variant" : "sit",
  "pluginType" : "EXTRACTOR",
  "label" : "Tap Custom Test",
  "description" : "A dataplugin created during an SIT run",
  "pipUrl" : "git+https://github.com/Matatika/example-repository",
  "settings" : [ {
    "name" : "username",
    "kind" : "STRING",
    "label" : "Username",
    "placeholder" : "username",
    "description" : "The username login credential.",
    "protected" : false
  }, {
    "name" : "email",
    "kind" : "EMAIL",
    "label" : "Email",
    "description" : "The email login credential.",
    "placeholder" : "[email protected]",
    "protected" : false
  }, {
    "name" : "start_date",
    "kind" : "DATE_ISO8601",
    "label" : "Start Date",
    "description" : "The data to begin extracting data from, in ISO 8601 format.",
    "placeholder" : "2020-01-01T00:00:00Z",
    "protected" : false
  } ]
}
headers = {
  'Authorization': ACCESS_TOKEN
}

response = requests.request("PUT", url, headers=headers, data=data)

print(response.text.encode('utf8'))

Response

201 Created

Dataplugin with HAL links.

{
  "id" : "ca779797-f25f-457a-95cf-801bedfd65c2",
  "pluginType" : "EXTRACTOR",
  "name" : "tap-custom-test",
  "variant" : "sit",
  "label" : "Tap Custom Test",
  "description" : "A dataplugin updated during an SIT run",
  "hidden" : false,
  "pipUrl" : "git+https://github.com/Matatika/example-repository-2",
  "capabilities" : [ ],
  "select" : [ ],
  "update" : { },
  "vars" : { },
  "settings" : [ {
    "name" : "username",
    "aliases" : [ ],
    "label" : "Username",
    "placeholder" : "username",
    "kind" : "STRING",
    "description" : "The username login credential.",
    "protected" : false
  }, {
    "name" : "email",
    "aliases" : [ ],
    "label" : "Email",
    "placeholder" : "[email protected]",
    "kind" : "EMAIL",
    "description" : "The email login credential.",
    "protected" : false
  }, {
    "name" : "start_date",
    "aliases" : [ ],
    "label" : "Start Date",
    "placeholder" : "2020-01-01T00:00:00Z",
    "kind" : "DATE_ISO8601",
    "description" : "The data to begin extracting data from, in ISO 8601 format.",
    "protected" : false
  } ],
  "variants" : [ ],
  "commands" : { },
  "matatikaHidden" : false,
  "_links" : {
    "self" : {
      "href" : "https://catalog.matatika.com/api/dataplugins/ca779797-f25f-457a-95cf-801bedfd65c2"
    },
    "update dataplugin" : {
      "href" : "https://catalog.matatika.com/api/workspaces/374ebea3-6784-403f-8722-db2002c8fd83/dataplugins/ca779797-f25f-457a-95cf-801bedfd65c2",
      "type" : "PUT"
    },
    "delete dataplugin" : {
      "href" : "https://catalog.matatika.com/api/dataplugins/ca779797-f25f-457a-95cf-801bedfd65c2",
      "type" : "DELETE"
    }
  }
}

Update a dataplugin

PUT

/api/dataplugins/{dataplugin-id}

Updates the dataplugin {dataplugin-id}.

Prerequisites

  • Dataplugin {dataplugin-id} must exist

Request

Body

Dataplugin resource.

{
  "name" : "tap-custom-test",
  "variant" : "sit",
  "pluginType" : "EXTRACTOR",
  "label" : "Tap Custom Test",
  "description" : "A dataplugin updated during an SIT run",
  "pipUrl" : "git+https://github.com/Matatika/example-repository-2",
  "settings" : [ {
    "name" : "username",
    "kind" : "STRING",
    "label" : "Username",
    "placeholder" : "username",
    "description" : "The username login credential.",
    "protected" : false
  }, {
    "name" : "email",
    "kind" : "EMAIL",
    "label" : "Email",
    "description" : "The email login credential.",
    "placeholder" : "[email protected]",
    "protected" : false
  }, {
    "name" : "start_date",
    "kind" : "DATE_ISO8601",
    "label" : "Start Date",
    "description" : "The data to begin extracting data from, in ISO 8601 format.",
    "placeholder" : "2020-01-01T00:00:00Z",
    "protected" : false
  } ]
}
Path Type Description Constraints
description String A description of the dataplugin  
repositoryUrl String A URL to the dataplugin repository  
settings Array of Settings The dataplugin settings  

Example Snippets

  • cURL
curl -H "Authorization: Bearer $ACCESS_TOKEN" 'https://catalog.matatika.com/api/workspaces/374ebea3-6784-403f-8722-db2002c8fd83/dataplugins/ca779797-f25f-457a-95cf-801bedfd65c2' -i -X PUT \
    -H 'Accept: application/json, application/javascript, text/javascript, text/json' \
    -H 'Content-Type: application/hal+json; charset=ISO-8859-1' \
    -d '{
  "name" : "tap-custom-test",
  "variant" : "sit",
  "pluginType" : "EXTRACTOR",
  "label" : "Tap Custom Test",
  "description" : "A dataplugin updated during an SIT run",
  "pipUrl" : "git+https://github.com/Matatika/example-repository-2",
  "settings" : [ {
    "name" : "username",
    "kind" : "STRING",
    "label" : "Username",
    "placeholder" : "username",
    "description" : "The username login credential.",
    "protected" : false
  }, {
    "name" : "email",
    "kind" : "EMAIL",
    "label" : "Email",
    "description" : "The email login credential.",
    "placeholder" : "[email protected]",
    "protected" : false
  }, {
    "name" : "start_date",
    "kind" : "DATE_ISO8601",
    "label" : "Start Date",
    "description" : "The data to begin extracting data from, in ISO 8601 format.",
    "placeholder" : "2020-01-01T00:00:00Z",
    "protected" : false
  } ]
}'
  • Python (requests)
import requests

url = "https://catalog.matatika.com/api/workspaces/374ebea3-6784-403f-8722-db2002c8fd83/dataplugins/ca779797-f25f-457a-95cf-801bedfd65c2"

data = {
  "name" : "tap-custom-test",
  "variant" : "sit",
  "pluginType" : "EXTRACTOR",
  "label" : "Tap Custom Test",
  "description" : "A dataplugin updated during an SIT run",
  "pipUrl" : "git+https://github.com/Matatika/example-repository-2",
  "settings" : [ {
    "name" : "username",
    "kind" : "STRING",
    "label" : "Username",
    "placeholder" : "username",
    "description" : "The username login credential.",
    "protected" : false
  }, {
    "name" : "email",
    "kind" : "EMAIL",
    "label" : "Email",
    "description" : "The email login credential.",
    "placeholder" : "[email protected]",
    "protected" : false
  }, {
    "name" : "start_date",
    "kind" : "DATE_ISO8601",
    "label" : "Start Date",
    "description" : "The data to begin extracting data from, in ISO 8601 format.",
    "placeholder" : "2020-01-01T00:00:00Z",
    "protected" : false
  } ]
}
headers = {
  'Authorization': ACCESS_TOKEN
}

response = requests.request("PUT", url, headers=headers, data=data)

print(response.text.encode('utf8'))

Response

200 OK

Dataplugin with HAL links.

{
  "id" : "ca779797-f25f-457a-95cf-801bedfd65c2",
  "pluginType" : "EXTRACTOR",
  "name" : "tap-custom-test",
  "variant" : "sit",
  "label" : "Tap Custom Test",
  "description" : "A dataplugin updated during an SIT run",
  "hidden" : false,
  "pipUrl" : "git+https://github.com/Matatika/example-repository-2",
  "capabilities" : [ ],
  "select" : [ ],
  "update" : { },
  "vars" : { },
  "settings" : [ {
    "name" : "username",
    "aliases" : [ ],
    "label" : "Username",
    "placeholder" : "username",
    "kind" : "STRING",
    "description" : "The username login credential.",
    "protected" : false
  }, {
    "name" : "email",
    "aliases" : [ ],
    "label" : "Email",
    "placeholder" : "[email protected]",
    "kind" : "EMAIL",
    "description" : "The email login credential.",
    "protected" : false
  }, {
    "name" : "start_date",
    "aliases" : [ ],
    "label" : "Start Date",
    "placeholder" : "2020-01-01T00:00:00Z",
    "kind" : "DATE_ISO8601",
    "description" : "The data to begin extracting data from, in ISO 8601 format.",
    "protected" : false
  } ],
  "variants" : [ ],
  "commands" : { },
  "matatikaHidden" : false,
  "_links" : {
    "self" : {
      "href" : "https://catalog.matatika.com/api/dataplugins/ca779797-f25f-457a-95cf-801bedfd65c2"
    },
    "update dataplugin" : {
      "href" : "https://catalog.matatika.com/api/workspaces/374ebea3-6784-403f-8722-db2002c8fd83/dataplugins/ca779797-f25f-457a-95cf-801bedfd65c2",
      "type" : "PUT"
    },
    "delete dataplugin" : {
      "href" : "https://catalog.matatika.com/api/dataplugins/ca779797-f25f-457a-95cf-801bedfd65c2",
      "type" : "DELETE"
    }
  }
}

Delete a dataplugin

DELETE

/api/dataplugins/{dataplugin-id}

Deletes the dataplugin {dataplugin-id}.

Request

Example Snippets

  • cURL
curl -H "Authorization: Bearer $ACCESS_TOKEN" 'https://catalog.matatika.com/api/dataplugins/ca779797-f25f-457a-95cf-801bedfd65c2' -i -X DELETE \
    -H 'Accept: application/json, application/javascript, text/javascript, text/json' \
    -H 'Content-Type: application/json'
  • Python (requests)
import requests

url = "https://catalog.matatika.com/api/dataplugins/ca779797-f25f-457a-95cf-801bedfd65c2"

headers = {
  'Authorization': ACCESS_TOKEN
}

response = requests.request("DELETE", url, headers=headers)

print(response.text.encode('utf8'))

Response

204 No Content

No response body provided.