Custom Scripts
Custom scripts can be used in Pipelines by choosing Script from Section 2 - Clean, transform and organise when creating or editing your data import or by defining an ‘inline_script’ in your Pipeline YAML.
Basics
Custom scripts are bash scripts that generally invoke Meltano commands. As mentioned before you can also control the data import’s environment in these scripts.
When you provide a Script to a pipeline, we will still add your plugins properties to the pipeline environment. Other than that, you are now in complete control of the environment, installation of plugins and execution of your pipeline.
Recommendations
Minimal Script
Generally at a minimum you will need to:
MELTANO_ENVIRONMENT=<env_name>
meltano environment add <env_name>
export MELTANO_ENVIRONMENT
meltano install
meltano run <tap_name> <target_name>
Adding a Meltano environment will allow your tap to save a state for next time it runs, if your tap supports this.
Running meltano install will install all plugins in your workspace. You can run meltano install <plugin_type> <plugin_name> to install specific ones if you wish.
meltano run <tap_name> <target_name> is the actual command to run the data import and get your data into your data store.
Using Meltano To Invoke Other Plugins
meltano invoke dbt deps
meltano invoke dbt run
By invoking other plugins through Meltano, you gain the benefit of Meltano taking base level environment variables and passing them these plugins to use. This isn’t perfect in every case, but generally will get you around setting a lot of environment variables manually.