StreamSets Deepens Snowflake Partnership with New Data Pipeline Engine for Snowpark

By AIT News Desk On Jun 10, 2021

StreamSets, the provider of the industry’s first DataOps platform, announced the preview of its new engine for running data pipelines using the new developer experience, Snowpark, from Snowflake, the Data Cloud company. StreamSets’ engine for Snowpark marries StreamSets’ powerful DataOps capabilities with the flexibility in Snowpark to use languages such as Java and Scala. Snowpark and the StreamSets engine for Snowpark provides a unique combination of power and ease-of-use for data engineers to build, operate and monitor smart data pipelines for the Snowflake Data Cloud.

Many Snowflake customers already use the StreamSets DataOps Platform to build and operate smart data pipelines that ingest data in streaming, batch or change data capture (CDC) modes into the Snowflake Data Cloud. Now, with Snowpark and the StreamSets engine for Snowpark , data engineers can go beyond SQL to express powerful data pipeline logic with the StreamSets DataOps Platform utilizing their preferred language, such as Scala and Java, through an intuitive graphical interface. Those pipelines can then execute directly on Snowflake’s Data Cloud, giving even more users the ability to achieve powerful data insights without having to manage additional processing systems.

Specifically, Snowflake customers can use the StreamSets DataOps Platform and the StreamSets engine for Snowpark to get even more out of their Data Cloud investments with the following key benefits:

Bluenote Raises $10Million from Lux Capital & Elad Gil to Transform Life Sciences Workflows with AI

Dec 20, 2024

Dynamic Infrastructure Introduces Groundbreaking AI Capability to Predict Future Deterioration in Civil Infrastructure

Dec 20, 2024

Pollo AI Launches Video-to-Video Feature, Revolutionizing Creative AI Video Creation and Outshining Competitors

Dec 20, 2024

Prev Next 1 of 40,564

A high-performance data engineering platform harnessing the power and flexibility of Scala while executing entirely in Snowflake’s Data Cloud– eliminating the need for additional Spark-based systems that add cost, complexity and latency
A powerful design environment that provides 10x productivity to data engineers building and orchestrating enterprise-grade data pipelines, combining the ease of drag-and-drop design with the extensibility of custom code and a comprehensive SDK
A single experience for building, operating and monitoring all pipelines for Snowflake’s Data Cloud, including streaming, CDC and batch data pipelines ingesting data to the Data Cloud, as well as ELT pipelines executing workloads directly in the Data Cloud via Snowpark
Smart data pipelines that are designed to deliver data on a continuous basis to support a DataOps practice, with built-in resiliency to changes in data structures, schema and semantics
Data engineering delivered as a cloud service with no infrastructure to build or manage

“Snowpark is transformative for data engineers and developers who prefer languages other than SQL for development,” said Isaac Kunen, Senior Product Manager, Snowflake. “With StreamSets’ new engine for Snowpark, data engineers will be able to build and operate Scala-based data pipelines designed for DataOps, bringing the power and performance of Snowflake’s Data Cloud to the savviest enterprise data engineers.”

“Our customers have been making amazing strides employing DataOps practices and democratizing their data with Snowflake,” said Judy Ko, Chief Product Officer, StreamSets. “We see Snowpark as the next big step, making sophisticated data engineering accessible to everyone, so all types of data, from all sources, can be delivered continuously through the Data Cloud.”