When you're working on automating the entirety of your company's data operations, you'll quickly start running into edge cases around scheduling. Not every workflow needs to happen on a set schedule and you don't always know exactly when something needs to run.
- Can you only run a specific workflow once you've been sent a file?
- Do you only need to run the workflow when code is updated?
- Does your process heavily rely on someone else's work completing first?
- What if you need to run a solution more frequently during peak season, but much less outside of that?
If you've found yourself trying to build for these types of situations then we have just the thing to make it easier.
Execute from Anywhere, at Anytime
Today, we're announcing the introduction of Webhook Triggers for Vessels and Fleets in Shipyard. This new feature will enable teams to programmatically execute solutions living in Shipyard with a simple POST request.
After you create your Vessel or Fleet, you can navigate to the Triggers tab to add a webhook. Each Vessel or Fleet can have exactly one webhook attached to it.
After you successfully run a POST request against a Vessel or Fleet webhook, it will immediately be scheduled to run. The best part is that with webhooks, you'll still be able to run hundreds of jobs at once and have resources scale dynamically, just like you've come to expect from Shipyard. Plus, you'll still be able to run everything on-demand or with your existing schedules.
With this new trigger type, you can now build your solutions to be entirely reactive, rather than trying to rely 100% on schedule-based triggers. This update makes Shipyard even easier to execute any data workflow that your team could build.
Integrate with External Tools
The introduction of webhooks opens up a world of possibilities to integrate Shipyard with other tools that you use every day. We want to build Shipyard in a way that makes it easier to "play nicely" with other tools, while still building the foundation that will help the platform become your central source for running data operations. The following examples show how easy it is to connect Shipyard to your other tools:
AWS Cloudwatch + Lamda - Use AWS Cloudwatch to monitor AWS resources, like S3, for new files. When a new file gets delivered, kick off a Lambda job that runs a POST command against Shipyard. With this setup, you can keep Shipyard running all of your heavy-duty, long-running data jobs while keeping Lambda responsible for quickly triggering Shipyard at the right moment.
Zapier - Run your existing zaps that connect SaaS apps together with small "if this, then that" rules, such as loading in new emails from a Google form to your CRM of choice. Once the job completes, run a Webhook to kick off a Fleet in Shipyard that processes and manipulates the newly loaded data in bulk.
GitHub Actions - While our GitHub Code Sync integration ensures that Shipyard is always executing up-to-date code, the platform never knows the exact moment that your code is updated. If you want your code to be executed as soon as new code exists, you can set up GitHub Actions to run a POST request to Shipyard every time you merge PRs or make commits.
This is only a short list of possibilities you can open up with the latest support for webhooks. You can just as easily integrate with:
- CI/CD tools like Jenkins, Travis and CircleCI
- Function as a Service tools like Pipedream or Cloud Functions
- Business workflow tools like Tray.io, IFTTT, and n8n.io
- Your own internal applications
While this update makes it easier for other services to trigger Shipyard, the inverse is true as well! Shipyard can always run an HTTP request (POST, GET, UPDATE, PATCH) to any URL to make updates and push changes between your favorite SaaS tools.
Building Internal References
Shipyard's security measures prevent Vessels or Fleets located in different projects from talking to each other. This setup is ideal for data governance, preventing data from leaking between clients or teams. However, it has sometimes prevented customers from connecting Fleets across projects or organizations. This is a perfectly legitimate use case that we wanted to help support.
Now that Shipyard has webhooks, you can make one Vessel or Fleet trigger another, potentially in a completely different project or organization. In this way, you can build out a workflow that connects Fleet A -> Fleet B -> Fleet C with ease!
This setup is perfect for working as part of a larger data team, where Data Engineering may want their jobs to trigger Data Science jobs, but the teams don't want each other having direct access to their mission-critical workflows.
This setup could also work in situations where a company needs to deliver files to an external vendor using Shipyard. The company could run their own data processing jobs on Shipyard and once the Fleet finishes running, trigger the Vendor's Shipyard webhook to pick up the file and process it. With this process in place, data loads are tracked more specifically, rather than relying on finicky file watchers that eat up resources.
Webhooks are now available to all subscribers and can be tested with any paid account. Sign up for our free Developer Plan to start building workflows and subscribe to trigger any script you write via a webhook. For more information about how to set up and use these webhooks, view our documentation.
We're looking forward to seeing how users will take advantage of this functionality to quickly develop and deploy data solutions.
About Shipyard:
Shipyard is a modern data orchestration platform for data engineers to easily connect tools, automate workflows, and build a solid data infrastructure from day one.
Shipyard offers low-code templates that are configured using a visual interface, replacing the need to write code to build data workflows while enabling data engineers to get their work into production faster. If a solution can’t be built with existing templates, engineers can always automate scripts in the language of their choice to bring any internal or external process into their workflows.
The Shipyard team has built data products for some of the largest brands in business and deeply understands the problems that come with scale. Observability and alerting are built into the Shipyard platform, ensuring that breakages are identified before being discovered downstream by business teams.
With a high level of concurrency and end-to-end encryption, Shipyard enables data teams to accomplish more without relying on other teams or worrying about infrastructure challenges, while also ensuring that business teams trust the data made available to them.
For more information, visit www.shipyardapp.com or get started for free.
