Fivetran is one of the leaders in data ingestion. They make it incredibly easy to load data from the SaaS tools you already use to the database of your choice in a matter of minutes. In fact, we use them ourselves here at Shipyard!

However, as you start relying on Fivetran to load more and more data, the complexity can start to ramp up. How do you verify that all 10+ data sources loaded today? How do you make sure your models, dashboards, and other activation efforts aren't using incomplete data? How do you reduce the time delays between all of your data processes? How do you course correct when Fivetran hits delays or syncs need to retry?

Those questions can be tough to address right now, but it's imperative to have an answer as you build out a sustainable data operations practice. Fortunately, we made it our mission to find ways to connect Fivetran to the rest of your modern data stack in a seamless manner.

Introducing our Fivetran Blueprints

As of today, Shipyard now has 4 new low-code Blueprints to help you quickly work with Fivetran. This powerful combination of Shipyard + Fivetran will enable your team to continue quickly loading data into your databases with Fivetran while connecting these syncs to the rest of your data operations processes.

By providing a Fivetran API Key/Secret and a specific Connector ID, you can mix and match Blueprints to:

Check out our demo of this workflow!

Run other Data Processes before and after Fivetran

With the current state of Fivetran, your data loads live in isolation from the rest of your data operations, despite typically being the initial source of your data. You know roughly when syncs will start and finish every day, but there's no way to guarantee that downstream processes only run once the data has loaded. You're forced into a situation where your business-critical data jobs are running with finicky schedule-based workflows, resulting in longer runtimes and potentially inaccurate data.

By using the "Execute Sync" and "Run Sync Status" Blueprints in Shipyard, you can build a workflow that runs Fivetran connectors, waits for the moment when data was successfully loaded, then immediately runs downstream processes like dbt, Dataform, BI data refreshes, and more. You can separate this out per datasource so that processes related to Stripe run immediately after Stripe data is loaded, while holistic data processes don't run until every connector has finished syncing successfully.

On the other hand, Shipyard will also automatically prevent downstream jobs from running if your Fivetran data syncs fail, giving you the ability to troubleshoot the issue directly in Fivetran, or run your own customized alerting and resolution process. Wouldn't it be great if the right clients or teams could automatically know when their specific data will be delayed?

Enhanced Fivetran Scheduling Control

By default, Fivetran gives you the ability to sync your connectors every 5, 15, 30, 60, 120, 180, 360, 480, 720, and 1440 minutes. While this offers a lot of flexibility, it's not precise. You can only choose the time your sync runs if you're syncing data every 24 hours and syncs won't run at the exact specified time due to offsets.

By using Shipyard to orchestrate your Fivetran connectors, you can control the exact times that your syncs happen. With Hourly, Daily, Weekly, and Monthly scheduled options, you can choose when you'd like to sync your data. Plus, the addition of Webhooks and workflow connections lets you use Fivetran as part of an event-driven workflow, ensuring that it only runs after other internal processes finish.

Through this powerful combination, you can precisely control when data sources, schemas, or tables are loaded, rather than relying on an all-or-nothing approach to syncing.

Connect All Of Your Data Tools

Shipyard gives you the power to quickly connect the execution of Fivetran connectors to any script that you write in Python, Bash, or Node.js. You can also connect it to other common processes that need to be run against external databases (Snowflake, Redshift, BigQuery, etc.) or data storage systems (AWS S3, Google Cloud Storage, etc.) using our low-code Blueprint Library.

Shipyard automates and connects ANY data tool that you might be using - not just Fivetran. With the Shipyard platform, your team has greater flexibility to create a modular pipeline where each step shares data with each other, rather than relying on flimsy schedule-based pipelines between siloed systems.

Here's a few examples of how you can connect Fivetran to other services to create a seamless solution for your Data Team.

Trigger dbt Cloud or dbt Core jobs after a successful Fivetran data load
Refresh Tableau extracts after successfully loading Fivetran data
Run a Python script to obfuscate PII information immediately after Fivetran loads the data.
Send Slack alerts to notify the appropriate teams when Fivetran syncs fail
Send raw data to a stakeholder's SFTP as soon as it loads.

Get Started Today

The Fivetran Blueprints are now available to all users and can be tested with any account. For more details, see our documentation.

Shipyard is making it easier than ever to automate your entire data stack. Sign up for our free Developer Plan to get started automating your Fivetran syncs.

We're looking forward to seeing how users will take advantage of this new Blueprint to connect Fivetran to the rest of their modern data stack.

About Shipyard:
Shipyard is a modern data orchestration platform for data engineers to easily connect tools, automate workflows, and build a solid data infrastructure from day one.

Shipyard offers low-code templates that are configured using a visual interface, replacing the need to write code to build data workflows while enabling data engineers to get their work into production faster. If a solution can’t be built with existing templates, engineers can always automate scripts in the language of their choice to bring any internal or external process into their workflows.

The Shipyard team has built data products for some of the largest brands in business and deeply understands the problems that come with scale. Observability and alerting are built into the Shipyard platform, ensuring that breakages are identified before being discovered downstream by business teams.

With a high level of concurrency and end-to-end encryption, Shipyard enables data teams to accomplish more without relying on other teams or worrying about infrastructure challenges, while also ensuring that business teams trust the data made available to them.

For more information, visit www.shipyardapp.com or get started for free.