When you automate data workflows, schedule systems don't always cut it. Timing between steps can be flaky and the scheduled run time is usually someone's "best guess". That's why event driven workflows are a key part of any good data operations. They help you react as soon as data is available or as soon as someone clicks a button. That's why a year ago we released the ability to trigger Fleets via a webhook.

But what if simply running a workflow based on an event isn't good enough? Sometimes you might need to run the workflow using the context of the event. What was the file name that was uploaded? Who clicked the button? What time was the workflow triggered? This data can be used to customize the workflow to run in a more dynamic and accurate way.

Sending Parameters to Webhooks

Today, we're launching our latest enhancements to webhooks: parameters. When you trigger a Fleet using a webhook, we now capture any data sent in the header or the body and store it as a file that's available to every Vessel in that Fleet. Data can be in any given format (JSON, YAML, TXT, etc.), giving you the freedom to design your workflows as you see fit without limiting the types of applications that can externally trigger Shipyard.

When you're ready to use this data, your scripts can access the contents by reading in the file (whose path is available through the new environment variables of SHIPYARD_WEBHOOK_HEADERS_FILE and SHIPYARD_WEBHOOK_BODY_FILE).

Previously, triggering a Fleet via a webhook meant that you were executing a static configuration of the Fleet. Every time you ran it, it would run exactly the same way. Now, your Fleet can react to the data it was sent at runtime. This opens up a world of new possibilities to make your Fleet's more dynamic and reactive in nature.

Unlocking a World of New Use Cases

Trigger via Text

If you use a messaging service like Twilio, you know that a webhook can be triggered every time a number is texted or called. With our latest update, you can use Twilio to trigger your Fleets by simply sending a text. If you want to go one step further, your entire Fleet can be customized to run in a specific way based on the contents of that text message. This gives you the ability to allow external and internal partners to execute your Fleets without ever needing access to the platform.

Process a Recently Uploaded File

What if your data pipelines rely on infrequent file deliveries from your partners? You may need to build a data pipeline to process files every time they are uploaded to an FTP server or an S3 bucket, but you don't know what the file name is going to be.

With our new webhook parameters, you can use existing file-watching services to POST a request to a Shipyard webhook, with the file_path included, every time a new file is uploaded. This could trigger a Fleet that processes the data, checks it for any errors, uploads it to the appropriate database, then deletes or renames the file so it won't get processed again.

Interactive Slack Messages

Internal messaging tools like Slack allow you to send messages with interactive buttons to keep your team's workflow all in one place. For example, you could use Shipyard to send Slack messages about a data issue. Rather than letting the notification just exist in the void, you want your team to "Claim" the issue or mark it as "Fixed".

With our latest update, you could include buttons in your Slack data alerts that trigger a Fleet's webhook when clicked. When someone clicks "Claim", a Fleet is triggered that updates the current message to indicate which user is currently looking into the data issue. When someone clicks "Fixed", the current message could be updated to include a green checkmark emoji tied to the user who clicked the button.

Automated Project Management

Internal project management tools like Asana, Shortcut, or GitHub can be configured to send a POST request to a webhook for every change being made. That means that Shipyard can be used as the central server to accept and process these webhooks. All you need to do is build a Fleet to interpret the data being sent to kick off other Fleets (via webhook). With this setup you can accomplish uses cases like:

• Automatically moving tasks from one project to another once they reach a specific stage
• Syncing comments between your project management tool and your internal communication tool
• Notifying external parties once a task has been completed

The Powerful Backend for your Application

You can use Shipyard as the backend to process heavy-lifting data use cases for your application. For example, you might want to allow users to export their data from your application at the click of a button.

In Shipyard, you can build a Fleet that runs a SQL query to process a specific customer's data in Snowflake and send that customer an email with their data attached.  In your application, you can create a button that sends the user's org_id, email, and selected_date_range to a Shipyard webhook. At runtime, your Fleet can pass the org_id and selected_date_range into the SQL query to ensure that the right data is grabbed. Once the data is secured, an email can be sent directly to the user's email.

Getting Started

We built a tutorial from the ground up to help you understand how to start working with webhook parameters. This tutorial walks through the steps of triggering a Fleet using a CURL command and running a python script that captures and prints out the data associated with the CURL command. It's the best way to learn more about how to start making your Fleets run dynamically!

Currently, this functionality is limited to users writing their own scripts to interact with the webhook data, but we are working on ways to allow all of our existing Blueprints to interact with webhook data as well.

Webhook Parameters are now available to all paid subscribers and can be tested with any account. Sign up for our free Developer Plan to start creating and editing data pipelines with complex logic today.

We're looking forward to seeing how users will take advantage of this functionality to quickly launch, monitor, and share data pipelines!

About Shipyard:
Shipyard is a modern data orchestration platform for data engineers to easily connect tools, automate workflows, and build a solid data infrastructure from day one.

Shipyard offers low-code templates that are configured using a visual interface, replacing the need to write code to build data workflows while enabling data engineers to get their work into production faster. If a solution can’t be built with existing templates, engineers can always automate scripts in the language of their choice to bring any internal or external process into their workflows.

The Shipyard team has built data products for some of the largest brands in business and deeply understands the problems that come with scale. Observability and alerting are built into the Shipyard platform, ensuring that breakages are identified before being discovered downstream by business teams.

With a high level of concurrency and end-to-end encryption, Shipyard enables data teams to accomplish more without relying on other teams or worrying about infrastructure challenges, while also ensuring that business teams trust the data made available to them.

For more information, visit www.shipyardapp.com or get started with our free Developer Plan.