Overview

This document details the process flow for near-real time ingestion of data from standard RDBMS using log based change data capture into Snowflake. The capabilities explained in this document also go ahead to explain the key Data Engineering concepts within Snowflake specifically around Orchestration of ELT processes once the CDC data lands in Snowflake. This is achieved using the combination of Snowpipe (wrapped under the Kafka-Snowflake connector), streams (CDC like capability within snowflake) and tasks (orchestration process).

The purpose of this document is:

  1. Enable developers to setup a working demo in their environment and get a quick POV done

Karthik Venkatraman

Sales Engineer - Snowflake - https://www.linkedin.com/in/kvenkatr/

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store