Create a Custom Pipeline
Time required: 5 minutes
You must have already:
- Signed up for a Matatika account
- Created a workspace through the Matatika app or API
In this how-to guide, we will go through the steps of creating a custom pipeline.
With a custom pipeline you are in complete control of all the plugins being run and in what order. We use custom pipeline to run reports, send emails and even some to just run tests.
Set Up Steps
- In your workspace, go to the Lab, then the Pipelines page.
- Click the
+ PIPELINEbutton in the top right of the page.
- Fill in the name field then click the empty square with the plus sign.
- Find the plugin you want to add in either the
- You will need to install the plugins you want to use, find out how here: Adding A Plugin To Your Workspace
- Repeat step 4 adding as many plugins as you want.
- Expand the
Settingsmenu and then each of the plugins you have added.
- Fill in at least all the required
*settings for each plugin.
- In the second section
Clean, transform and organise, decide if you want to use our default data import script by leaving
Default Actionsselected, or supply your own custom actions or script.
- In the third and last section
Automate your import, you can choose how often your data import should run. There are a few example schedules or you can use
Customto create your own.
Save. A green bar will appear at the top of your screen telling you your data import was saved.
- Head back to the
Pipelinesscreen, where for the next 1 to 2 minutes a
config jobwill be running on your data import. This will set everything up and commit your new data imports’s changes to your workspace repository.
- Once the
config jobhas completed you are free to run your data import, or leave it to its schedule.
When using data sources in your pipeline you can create a type of pipeline called an import pipeline. By using an import pipeline Matatika will automatically add the Matatika supported plugins for that data source. These include pre build data models and datasets that you can see in your app.
Find out how to create a import pipeline here: Create a Data Import Pipeline