Custom scripts can be used in Pipelines by choosing
Section 2 - Clean, transform and organise when creating or editing your data import or by defining an ‘inline_script’ in your Pipeline YAML.
When you provide a
Script to a pipeline, we will still add your plugins properties to the pipeline environment. Other than that, you are now in complete control of the environment, installation of plugins and execution of your pipeline.
Generally at a minimum you will need to:
MELTANO_ENVIRONMENT=<env_name> meltano environment add <env_name> export MELTANO_ENVIRONMENT meltano install meltano run <tap_name> <target_name>
Adding a Meltano environment will allow your tap to save a state for next time it runs, if your tap supports this.
meltano install will install all plugins in your workspace. You can run
meltano install <plugin_type> <plugin_name> to install specific ones if you wish.
meltano run <tap_name> <target_name> is the actual command to run the data import and get your data into your data store.
meltano invoke dbt deps meltano invoke dbt run
By invoking other plugins through Meltano, you gain the benefit of Meltano taking base level environment variables and passing them these plugins to use. This isn’t perfect in every case, but generally will get you around setting a lot of environment variables manually.