Did you miss out on the first day of the Modern Data Stack conference? Want a quick recap? We've got you covered!
Go here for takeaways from the 2nd day of the conference.
Our team attended sessions yesterday and packed our notebooks with lots of great tidbits. Here's what we took away from the first day of this great conference!
1. The Future of Analytics is Collaborative Tooling and Further Abstraction
In an opening session with George Fraser at Fivetran, Ali Ghodsi at Databricks, and Rohan Kumar at Azure Data, these heads of industry giants discussed where they see the future of data analytics going.
It's now easier than ever to gather your data and deliver it where you need to. The current modern data stack lets you worry less about the nuts and bolts of how to get access to data so you can focus more about driving value with that data.
The next 5 years will be spent bridging the gaps between the tools that data engineers use and the tools that Data Science and Business Analysts use. These teams need to be talking to each other constantly and have the ability to "link into" what the others are working on so that fresh data is immediately powering realtime insights, business intelligence, and operational analytics.
As the creators of a collaborative data orchestration platform, we couldn't agree more. Seeing how data gets used in your organization end-to-end will be a difference maker in how effectively businesses are able to act on their data.
2. Reverse ETL is driving Product Led Growth
Now that most teams have their data centralized in a cloud data warehouse, people are starting to ask the question - "How do I make use of all this data"? Enter reverse ETL.
Typical ETL (Extract Transform Load) is a process to move data from an external service into your data warehouse. Reverse ETL is the opposite - a process to move data from your data warehouse into an external service.
By having their data team build models with cleaned product data directly in the data warehouse, companies like Loom are able to send that data to marketing, support, and sales tools every few hours. This puts every team at the company in a position where they can all see a holistic view of a customer while still working in the tools that work best for them.
As big fans of services that help you action on your data quickly, we're excited to see how this space evolves and how reverse ETL makes it easier to keep all of your tools on the same page as your database.
3. Better Data Products start with dbt
Spencer Taylor at 4 Mile Analytics and JB at BetterHelp talked through how they build solid data experiences for internal and external stakeholders. Beyond creating fantastic looking data products (animated dashboards - my how far we've come), the team had some exceptional insight around how they constructed the backend to power these beautiful experiences that help team visualize and immediately act on their data.
Implementing dbt helped their teams solve the problem of maintaining hundreds of tables and views that powered their dashboards. The team could visualize how all the data connected together while keeping everything well documented and tested in easy-to-manage YAML files.
It's a common story that we keep hearing. Once you know how to deploy dbt to the cloud, your Analytics Engineering team can become a powerhouse of rock solid data that's well tested and error-proof, resulting in the data team finding potential issues with the data before vendors or engineering teams notice!
Plus, JB shared how they use a great new package that merges Great Expectations testing logic within dbt, called dbt-expectations. How cool is that?
4. MVP Data Stacks Help you make an impact in 90 days
Utsav Kaushish at User Interviews led a great sessions about lessons he learned while spinning up a data stack. When you first land in a new role, your job is create small wins in the first 2 weeks and have a big win within the first 90 days. You need to move quick so you can prove out value and help build a culture that focuses on data. Miss the mark and it will be difficult to get buy-in down the road.
The easiest way to accomplish to get started is by quickly setting up an MVP Modern Data Stack.
- Set up a cloud datawarehouse like Snowflake, Bigquery, or Snowflake
- Use Fivetran to load data from your SaaS tools into that Data Warehouse
- Use a visualization tool like Mode, Power BI, or Data Studio to explore the data.
Once these are in place, you have the basic tools you need to start making an impact with your data.
5. Enterprise Data Supply Chains are Strained
Rahul Dabke at Immuta presented recent findings in a research study called the DataOps Dilemma. When you think about DataOps, it's helpful to reframe it as a supply chain for data. In that supply chain, you have data suppliers and data consumers, where data moves back and forth between the two parties that exist both internally and externally.
While it's easy to map out how the data moves across the data supply chain, there are plenty of gaps that exist. Organizations largely reported that lack of security, automation, and the right skillsets were contributing to the bottlenecks in their data supply chains. As a result, 59% of organizations stated that their DataOps practice was just barely in existence.
Fortunately, there are a lot of tools popping up int he modern data stack, like Shipyard, that make automating your data workflows easy even if you don't have DevOps skills. Hopefully the data stack will continue to expand to offer more ways to improve the data supply chain for everyone!
Check out more of this great research study here.
And that's a wrap! While we didn't get to cover all the great presentations we saw, we hope this gives you a taste of what day 1 of the conference had to offer. Come check out our booth at the 2nd day of the conference!
About Shipyard:
Shipyard is a modern data orchestration platform for data engineers to easily connect tools, automate workflows, and build a solid data infrastructure from day one.
Shipyard offers low-code templates that are configured using a visual interface, replacing the need to write code to build data workflows while enabling data engineers to get their work into production faster. If a solution can’t be built with existing templates, engineers can always automate scripts in the language of their choice to bring any internal or external process into their workflows.
The Shipyard team has built data products for some of the largest brands in business and deeply understands the problems that come with scale. Observability and alerting are built into the Shipyard platform, ensuring that breakages are identified before being discovered downstream by business teams.
With a high level of concurrency and end-to-end encryption, Shipyard enables data teams to accomplish more without relying on other teams or worrying about infrastructure challenges, while also ensuring that business teams trust the data made available to them.
For more information, visit www.shipyardapp.com or get started with our free Developer Plan.
