External Data Sources

Note

This API is in public beta. The contract may change before general availability.

An external data source is a reusable, named connection that lets a transformation read from storage outside Cognite Data Fusion (CDF). You configure the credentials and the location once, then reference the source from transformation SQL by its external ID. Microsoft Fabric OneLake is currently the supported system.

Using this API requires the Fabric connector to be enabled for the project. Contact Cognite Support to have it enabled.

Warning

Store the client secret securely, for example in an environment variable or a secret manager. All the examples below read it from the environment rather than hard-coding it.

Concepts

• Format — the external system the source connects to. one_lake is the supported value, and each format has its own pair of data classes: OneLakeExternalDataSourceWrite for writing and OneLakeExternalDataSource for reading.

• Credentials — the Microsoft Entra ID application credentials, held by OneLakeCredentialsWrite. The client secret is write-only: it is stored encrypted and never returned when you read a data source, which is why the read model, OneLakeCredentials, has no client_secret.

• Location — where in OneLake the source points, held by OneLakeLocationDescription. Both IDs are GUIDs from the Fabric portal: the workspace ID under Workspace settings > Workspace ID and the lakehouse ID under Lakehouse settings > Item ID.

• Data set scoping — an optional data_set_id that scopes access to the data source through CDF data set permissions.

Permissions

Every operation is governed by transformationsExternalDataSourcesAcl, available in the SDK as TransformationsExternalDataSourcesAcl:

Operation	Action
list()	READ
upsert()	WRITE
delete()	WRITE
verify_usability()	USE

Running a transformation that reads from a data source also requires USE, in addition to the usual transformation and destination capabilities.

Register a data source

An upsert creates the data source if it does not exist and replaces it in full if it does, matched on external ID. Every request must therefore carry complete settings, including the client secret:

import os

from cognite.client import CogniteClient
from cognite.client.data_classes.transformations.externaldata import (
    OneLakeCredentialsWrite,
    OneLakeExternalDataSourceWrite,
    OneLakeLocationDescription,
    OneLakeSettingsWrite,
)

client = CogniteClient()

data_source = OneLakeExternalDataSourceWrite(
    external_id="fabric-lakehouse-prod",
    name="Production lakehouse",
    data_set_id=123456,
    settings=OneLakeSettingsWrite(
        credentials=OneLakeCredentialsWrite(
            client_id=os.environ["ONELAKE_CLIENT_ID"],
            tenant_id=os.environ["ONELAKE_TENANT_ID"],
            client_secret=os.environ["ONELAKE_CLIENT_SECRET"],
        ),
        location_description=OneLakeLocationDescription(
            workspace_id=os.environ["ONELAKE_WORKSPACE_ID"],
            container_id=os.environ["ONELAKE_CONTAINER_ID"],
        ),
    ),
)
registered = client.transformations.external_data_sources.upsert(data_source)

Verify that a data source is usable

Before you wire a data source into a transformation, check that CDF can reach the storage location with the stored credentials. That way you find configuration problems here instead of in a failed transformation job:

usability = client.transformations.external_data_sources.verify_usability("fabric-lakehouse-prod")

if not usability.is_usable:
    raise RuntimeError("The data source is missing, or its credentials do not grant access")

Read OneLake tables from transformation SQL

A registered data source is referenced from transformation SQL through the ext_onelake() table-valued function, which takes the external ID of the data source and the table name, plus an optional schema name:

SELECT * FROM ext_onelake('fabric-lakehouse-prod', 'my_table')
SELECT * FROM ext_onelake('fabric-lakehouse-prod', 'my_table', 'my_schema')

Data read this way can be written to any transformation destination. This example writes it into a data model view:

from cognite.client.data_classes import TransformationDestination, TransformationWrite
from cognite.client.data_classes.transformations.common import NonceCredentials, ViewInfo

session = client.iam.sessions.create()
nonce = NonceCredentials(session.id, session.nonce, client.config.project)

transformation = client.transformations.create(
    TransformationWrite(
        external_id="onelake-to-data-model",
        name="OneLake to data model",
        query="""
            SELECT
                externalId AS externalId,
                description AS name
            FROM ext_onelake('fabric-lakehouse-prod', 'my_table')
        """,
        destination=TransformationDestination.nodes(
            view=ViewInfo(space="my-model-space", external_id="MyView", version="v1"),
            instance_space="my-instance-space",
        ),
        conflict_mode="upsert",
        source_nonce=nonce,
        destination_nonce=nonce,
    )
)
job = client.transformations.run(transformation_id=transformation.id)

See Transformations for the rest of the transformation API, including the other ways of supplying credentials.

List data sources

Listing returns read models, so OneLake data sources come back as OneLakeExternalDataSource:

for data_source in client.transformations.external_data_sources.list(limit=None):
    print(data_source.external_id, data_source.settings.location_description.workspace_id)

To iterate without holding every data source in memory, call the API directly, optionally in chunks:

for data_source in client.transformations.external_data_sources():
    ...  # do something with the data source

for chunk in client.transformations.external_data_sources(chunk_size=25):
    ...  # do something with the chunk

Rotate credentials

Because a read model never carries the client secret, it cannot be turned back into a write model — as_write() raises a TypeError. Rotating a secret means building a new write model with the new secret and upserting it under the same external ID, which replaces the stored data source:

rotated = OneLakeExternalDataSourceWrite(
    external_id="fabric-lakehouse-prod",
    name="Production lakehouse",
    data_set_id=123456,
    settings=OneLakeSettingsWrite(
        credentials=OneLakeCredentialsWrite(
            client_id=os.environ["ONELAKE_CLIENT_ID"],
            tenant_id=os.environ["ONELAKE_TENANT_ID"],
            client_secret=os.environ["ONELAKE_NEW_CLIENT_SECRET"],
        ),
        location_description=OneLakeLocationDescription(
            workspace_id=os.environ["ONELAKE_WORKSPACE_ID"],
            container_id=os.environ["ONELAKE_CONTAINER_ID"],
        ),
    ),
)
client.transformations.external_data_sources.upsert(rotated)

Delete a data source

Transformations that still reference a deleted data source fail the next time they run:

client.transformations.external_data_sources.delete("fabric-lakehouse-prod")

API reference

The methods are documented under Transformation External Data Sources, and the data classes under Data classes.