In today's digital landscape, where data is being constantly exchanged between different applications and systems, ensuring the security and confidentiality of sensitive information has become paramount. Whether this information is a set of usernames and passwords, API Keys, or even configuration files, these credentials have to be safeguarded and reused to gain access to valuable data.

But sometimes, credentials themselves can be the source of headaches. Processes break because an employee leaves and everything tied to their login needs to be updated. API Keys get accidentally pushed to GitHub so they need to be rotated. Configuration files now have to use a new format. Tracking down the scripts that use these credentials and managing these updates can sometimes be difficult.

You can even have situations where analysts need to be able to access a system, but the engineering team doesn't want them to have direct access to the platform. As a consequence, a tug-of-war ensues where analysts perceive their productivity being hampered, while Engineering gets viewed as an obstacle, despite their valid security concerns.

We've seen most of these situations play out time and time again and we're dedicated to making credentials easier to work with overall.

Introducing Credential Management

We are excited to announce that Shipyard now has built-in support for storing, updating, and reusing credentials.

Every user, regardless of their plan, now has access to the credentials page which contains the ability to create and edit credential groups. You can select to make something custom or get started with vendor-specific credentials.

Every credential group will contain one or more credential items. These items can then be mapped to environment variables or open source low code Blueprint Inputs when you're building out a Fleet or a Blueprint.

We built our entire credential management flow to be flexible too. If you're actively building a new Fleet and realized you need to use some credentials, you can create and immediately use those new credentials without ever leaving the Fleet Builder.

Let's dig into a few new possibilities that Credential Management will open up for data teams that use Shipyard.

Reusability

In the past, if you set up 100s of Vessels with Email, you would have to enter your credentials from scratch for every Vessel.

Now, you can select "Use Credential" and watch the fields auto-populate with all the right values.  You can also mix and match credentials by choosing individual credential items from multiple groups. The choice is yours.

If your credentials ever need to be updated, you can make the change from one location and ensure that every Vessel starts using the new credentials automatically. This seamless transition of credential management will make for much smoother sailing.

Security

Just like our Environment Variables, credentials are secure by default. Any values you enter are encrypted at rest and in transit. Plus, these values are never sent back to the UI afterwards. Their values only resolve at runtime and instead show up as (hidden) in our platform.

This gives our users increased confidence and trust that their credentials are safe in our system.

Shareability

Everywhere you use credentials, you'll see them as a nice looking tag. This setup means that engineering teams can load in credentials securely on the backend that anyone in the organization can use without any user needing to know the underlying values.

This has some huge implications for self-service data flows.

Maybe you're a vendor that wants to deliver data directly to a client's database. The client can provide credentials to their database that you never have access to see, but now you have the ability to send directly to their database.

Or maybe the roles are reversed. You want your clients to send data from their warehouse directly to your S3 buckets so you can augment their data on your platform. However, you don't want to share S3 credentials with them. Instead, you give them access to Shipyard to set up their own data delivery flows, where they can insert your S3 credentials without needing to know what they are.

Thinking Outside the Box

Credentials can also be referenced anywhere in the application through the use of the #{CREDENTIAL_GROUP_REFERENCE.CREDENTIAL_ITEM} syntax. This means that you can now set custom values at the organization level that can be referenced by any Fleet or Vessel.

• Want every Vessel to install the same version of Pandas? Make it a credential and reference it when installing packages.
• Want your Python scripts to dynamically reference a specific dataset naming convention? Make the dataset name a credential and pass it as an environment variable that your script reads directly.
• Want to have development, staging, and production credentials for your database? Now you can make and flip between these seamlessly.

The Future of Credentials

This is just the beginning for credential management. Over the course of 2023, we'll be continuing to explore additional features like:

• Testing Credentials before saving
• Integrations with external services like 1Password or AWS Secrets Manager
• Ability to Upload Files, like RSA Keys or configuration files
• User Access Management per credential
• Dashboards to visually see credential usage
• API access to manage credentials programmatically

If there's something that you'd like to see, let us know about it.

Get Started Today

These use cases only scratch the surface of what's now possible with Shipyard. Credential Management is now available to all users.

Sign up for a free Developer Account to start automating data workflows with a seamless credential experience. Learn more about credentials through our documentation.