Product Update: July

Longer days mean we get more done!  Right?  Too often we find the opposite to be true – data teams are stretching and straining to gather the data the business demands, but are producing fewer insights, facing more data quality issues, and are fully occupied just keeping their data pipelines running.  Matatika’s latest features to help you get more done and cost you less include ways to automate updates, schedule custom workloads, and track your Snowflake costs.

TL;DR

News, examples, features, and data

• Following Matatika’s Transport 3.0 collective intelligence pitch, the UK Parliament Transport Select Committee has launched a further detailed inquiry – Read article…
• Accelerate change and improve quality with automated updates from a private and secure data plugin hub – Learn more…
• Schedule custom workloads including any python based workloads and Jupyter Notebooks – Learn more…
• Track your Snowflake costs – Learn more…

Transport 3.0 collective intelligence

In April 2023, Matatika was invited to present in person, or ‘give evidence’ as Westminster prefers to say, to the UK Parliament Transport Select Committee.  As one of 12 companies selected from more than 200 industry leaders, we felt the weight of the occasion, but also felt we had a unique perspective on how data should be collected, and how the ecosystem of innovations could thrive given the right infrastructure.  We are pleased to report that the Select Committee agreed and has launched a further inquiry calling out our proposal as a specific area of interest.  Learn more…

Read the announcement

Accelerate change and improve quality

Matatika was founded with the novel idea that technology and code were not the problem for companies, management of this code and infrastructure was!  With our platform’s native support for DevOps and DataOps workflows, you can rapidly release changes with total trust in your data.  Now we have released an Enterprise ready hub that allows large data teams to automate updates from a private and secure data plugin hub.  Another big step towards providing a completely managed solution for data-driven companies.  Learn more…

Schedule custom workloads

Several new features have made our pipelines easier to configure and even simpler to manage.  Python based workloads, including Jupyter Notebooks, are sometimes performing a vast amount of work and can need to run for several hours.  Our latest features make timeouts simple to configure in our Lab, and schedules now handle the long-running work too by skipping the scheduled run if the previous job has not finished yet.  Learn more about scheduling custom workloads…

Track your Snowflake costs

Many of our customers use Snowflake to store and process their data.  Often we’ve seen their costs gradually increasing as redundant copies of data or unused compute resources start to consume unnecessary credits.  Reducing these costs is often a simple matter of configuration and not a technical job for a DBA, which is a huge benefit of the Snowflake platform, but knowing what to tweak and when can be a challenge.  Now we’ve made this easy!  Our Snowflake monitoring bundle helps teams manage their Snowflake and keep costs down – for free!  Learn more…

Features

• Create a SKIPPED job if the launch is requested by a schedule for a pipeline that is already running
• Make pipeline timeout configurable through UI
• Added new task logic for configuring any unconfigured data stores in your workspace.

Fixes and updates

• Fix support for tap-trello schema property types
• Fix serialisation of full_description in plugin .lock files
• Default tap-mssql replication method to FULL_TABLE
• Prevent accidental multiple plugin install requests
• Fix an issue where chaning the default data store did not update in the current user workspace context, causing future actions to use the old data store.
• Fix delete datacomponent endpoint which is no longer incorrectly read-only
• Fix local profile, grant member and admin access to all local workspaces
• Update pipeline screen to order jobs response by end time, instead of created at

Finally, if any of the above has piqued your interest, consider giving us a star on GitHub!