Architecture and Security

Architecture

The following diagram provides an overview of how Kumo integrates with your Databricks environment as a native application:

The control plane is deployed in Kumo’s VPC.
The Kumo Data engine will be Databricks compute clusters in your Databricks workspace.
The Kumo AI engine will be deployed in Kumo’s VPC, operating only on ephemeral data read from the data store erased immediately after training/batch prediction complete.
The data store may be either a Kumo-owned bucket encrypted at rest on S3, or on your own data storage layer (e.g. UC volume).

Security

The native app for Databricks supports Kumo's end-to-end machine learning (ML) platform. Kumo enables enterprises to leverage state-of-the-art predictive analytics to make predictions, allowing data scientists to immediately tackle many prediction problems by first registering data sources and then issuing different SQL-like predictive query interfaces that specify their ML tasks. Kumo then executes the predictive query and automates the entire process of feature preparation, label engineering, training dataset creation, model optimization, and MLOps, making it easy for users to build multiple ML models.

With the Kumo native app for Databricks:

There is no hardware (virtual or physical) to select, install, configure, or manage.
There is no software to install, configure, or manage.
Kumo handles ongoing maintenance, management, upgrades, and tuning.

Additionally, the native app for Databricks provides additional security benefits for your organization, since all data resides within your Databricks environment. Because Kumo's data engine runs on your own all-purpose Databricks compute clusters, you can be sure that all your data—including your query results and predictions—are secure within your Databricks environment. However, keep in mind that the setup of Kumo as a Databricks native app will require assistance from Kumo during installation and the PoC process.

For a simpler deployment mode that requires less maintenance overhead, please see Kumo's SaaS version.