Transformations

TransformationsAPI

Create transformations

TransformationsAPI.create(transformation: Union[cognite.client.data_classes.transformations.Transformation, Sequence[cognite.client.data_classes.transformations.Transformation]]) → Union[cognite.client.data_classes.transformations.Transformation, cognite.client.data_classes.transformations.TransformationList]

Create one or more transformations.

Parameters:transformation (Union[Transformation, List[Transformation]]) – Transformation or list of transformations to create.
Returns:Created transformation(s)

Examples

Create new transformations:

>>> from cognite.client import CogniteClient
>>> from cognite.client.data_classes import Transformation, TransformationDestination
>>> from cognite.client.data_classes.transformations.common import ViewInfo, EdgeType, DataModelInfo
>>> c = CogniteClient()
>>> transformations = [
>>>     Transformation(
>>>         name="transformation1",
>>>         destination=TransformationDestination.assets()
>>>     ),
>>>     Transformation(
>>>         name="transformation2",
>>>         destination=TransformationDestination.raw("myDatabase", "myTable")
>>>     ),
>>>      Transformation(
>>>      name="transformation3",
>>>      view = ViewInfo(space="TypeSpace", external_id="TypeExtId", version="version"),
>>>      destination=TransformationDestination.nodes(view, "InstanceSpace")
>>>      ),
>>>      Transformation(
>>>      name="transformation4",
>>>      view = ViewInfo(space="TypeSpace", external_id="TypeExtId", version="version"),
>>>      destination=TransformationDestination.edges(view, "InstanceSpace")
>>>      ),
>>>      Transformation(
>>>      name="transformation5",
>>>      edge_type = EdgeType(space="TypeSpace", external_id="TypeExtId"),
>>>      destination=TransformationDestination.edges(edge_type,"InstanceSpace")
>>>      ),
>>>      Transformation(
>>>      name="transformation6",
>>>      data_model = DataModelInfo(space="modelSpace", external_id="modelExternalId",version="modelVersion",destination_type="viewExternalId"),
>>>      destination=TransformationDestination.instances(data_model,"InstanceSpace")
>>>      ),
>>>      Transformation(
>>>      name="transformation7",
>>>      data_model = DataModelInfo(space="modelSpace", external_id="modelExternalId",version="modelVersion",destination_type="viewExternalId", destination_relationship_from_type="connectionPropertyName"),
>>>      destination=TransformationDestination.instances(data_model,"InstanceSpace")
>>>      ),
>>> ]
>>> res = c.transformations.create(transformations)

Retrieve transformations by id

TransformationsAPI.retrieve(id: Optional[int] = None, external_id: Optional[str] = None) → Optional[cognite.client.data_classes.transformations.Transformation]

Retrieve a single transformation by id.

Parameters:id (int, optional) – ID
Returns:Requested transformation or None if it does not exist.
Return type:Optional[Transformation]

Examples

Get transformation by id:

>>> from cognite.client import CogniteClient
>>> c = CogniteClient()
>>> res = c.transformations.retrieve(id=1)

Get transformation by external id:

>>> from cognite.client import CogniteClient
>>> c = CogniteClient()
>>> res = c.transformations.retrieve(external_id="1")
TransformationsAPI.retrieve_multiple(ids: Optional[Sequence[int]] = None, external_ids: Optional[Sequence[str]] = None, ignore_unknown_ids: bool = False) → cognite.client.data_classes.transformations.TransformationList

Retrieve multiple transformations.

Parameters:
  • ids (List[int]) – List of ids to retrieve.
  • external_ids (List[str]) – List of external ids to retrieve.
  • ignore_unknown_ids (bool) – Ignore IDs and external IDs that are not found rather than throw an exception.
Returns:

Requested transformation or None if it does not exist.

Return type:

TransformationList

Examples

Get multiple transformations:

>>> from cognite.client import CogniteClient
>>> c = CogniteClient()
>>> res = c.transformations.retrieve_multiple(ids=[1,2,3], external_ids=['transform-1','transform-2'])

Run transformations by id

TransformationsAPI.run(transformation_id: Optional[int] = None, transformation_external_id: Optional[str] = None, wait: bool = True, timeout: Optional[float] = None) → cognite.client.data_classes.transformations.jobs.TransformationJob

Run a transformation.

Parameters:
  • transformation_id (int) – Transformation internal id
  • transformation_external_id (str) – Transformation external id
  • wait (bool) – Wait until the transformation run is finished. Defaults to True.
  • timeout (Optional[float]) – maximum time (s) to wait, default is None (infinite time). Once the timeout is reached, it returns with the current status. Won’t have any effect if wait is False.
Returns:

Created transformation job

Examples

Run transformation to completion by id:

>>> from cognite.client import CogniteClient
>>> c = CogniteClient()
>>>
>>> res = c.transformations.run(transformation_id = 1)

Start running transformation by id:

>>> from cognite.client import CogniteClient
>>> c = CogniteClient()
>>>
>>> res = c.transformations.run(transformation_id = 1, wait = False)
TransformationsAPI.run_async(transformation_id: Optional[int] = None, transformation_external_id: Optional[str] = None, timeout: Optional[float] = None) → cognite.client.data_classes.transformations.jobs.TransformationJob

Run a transformation to completion asynchronously.

Parameters:
  • transformation_id (int) – internal Transformation id
  • transformation_external_id (str) – external Transformation id
  • timeout (Optional[float]) – maximum time (s) to wait, default is None (infinite time). Once the timeout is reached, it returns with the current status.
Returns:

Completed (if finished) or running (if timeout reached) transformation job.

Examples

Run transformation asyncronously by id:

>>> import asyncio
>>> from cognite.client import CogniteClient
>>>
>>> c = CogniteClient()
>>>
>>> async def run_transformation():
>>>     res = await c.transformations.run_async(id = 1)
>>>
>>> loop = asyncio.get_event_loop()
>>> loop.run_until_complete(run_transformation())
>>> loop.close()

Preview transformations

TransformationsAPI.preview(query: Optional[str] = None, convert_to_string: bool = False, limit: int = 100, source_limit: Optional[int] = 100, infer_schema_limit: Optional[int] = 1000) → cognite.client.data_classes.transformations.TransformationPreviewResult

Preview the result of a query.

Parameters:
  • query (str) – SQL query to run for preview.
  • convert_to_string (bool) – Stringify values in the query results, default is False.
  • limit (int) – Maximum number of rows to return in the final result, default is 100.
  • source_limit (Union[int,str]) – Maximum number of items to read from the data source or None to run without limit, default is 100.
  • infer_schema_limit – Limit for how many rows that are used for inferring result schema, default is 1000.
Returns:

Result of the executed query

Examples

Preview transformation results as schema and list of rows:

>>> from cognite.client import CogniteClient
>>> c = CogniteClient()
>>>
>>> query_result = c.transformations.preview(query="select * from _cdf.assets")

Preview transformation results as pandas dataframe:

>>> from cognite.client import CogniteClient
>>> c = CogniteClient()
>>>
>>> df = c.transformations.preview(query="select * from _cdf.assets").to_pandas()

Cancel transformation run by id

TransformationsAPI.cancel(transformation_id: Optional[int] = None, transformation_external_id: Optional[str] = None) → None

Cancel a running transformation.

Parameters:
  • transformation_id (int) – Transformation internal id
  • transformation_external_id (str) – Transformation external id

Examples

Wait transformation for 1 minute and cancel if still running:

>>> from cognite.client import CogniteClient
>>> from cognite.client.data_classes import TransformationJobStatus
>>> c = CogniteClient()
>>>
>>> res = c.transformations.run(id = 1, timeout = 60.0)
>>> if res.status == TransformationJobStatus.RUNNING:
>>>     res.cancel()

List transformations

TransformationsAPI.list(include_public: bool = True, name_regex: Optional[str] = None, query_regex: Optional[str] = None, destination_type: Optional[str] = None, conflict_mode: Optional[str] = None, cdf_project_name: Optional[str] = None, has_blocked_error: Optional[bool] = None, created_time: Union[Dict[str, Any], cognite.client.data_classes.shared.TimestampRange, None] = None, last_updated_time: Union[Dict[str, Any], cognite.client.data_classes.shared.TimestampRange, None] = None, data_set_ids: Optional[List[int]] = None, data_set_external_ids: Optional[List[str]] = None, tags: Optional[cognite.client.data_classes.transformations.TagsFilter] = None, limit: Optional[int] = 25) → cognite.client.data_classes.transformations.TransformationList

List all transformations.

Parameters:
  • include_public (bool) – Whether public transformations should be included in the results. (default true).
  • name_regex (str) – Regex expression to match the transformation name
  • query_regex (str) – Regex expression to match the transformation query
  • destination_type (str) – Transformation destination resource name to filter by.
  • conflict_mode (str) – Filters by a selected transformation action type: abort/create, upsert, update, delete
  • cdf_project_name (str) – Project name to filter by configured source and destination project
  • has_blocked_error (bool) – Whether only the blocked transformations should be included in the results.
  • created_time (Union[Dict[str, Any], TimestampRange]) – Range between two timestamps
  • last_updated_time (Union[Dict[str, Any], TimestampRange]) – Range between two timestamps
  • data_set_ids (List[int]) – Return only transformations in the specified data sets with these ids.
  • data_set_external_ids (List[str]) – Return only transformations in the specified data sets with these external ids.
  • tags (TagsFilter) – Return only the resource matching the specified tags constraints. It only supports ContainsAny as of now.
  • limit (int) – Limits the number of results to be returned. To retrieve all results use limit=-1, default limit is 25.
Returns:

List of transformations

Return type:

TransformationList

Example

List transformations:

>>> from cognite.client import CogniteClient
>>> c = CogniteClient()
>>> transformations_list = c.transformations.list()

Update transformations

TransformationsAPI.update(item: Union[cognite.client.data_classes.transformations.Transformation, cognite.client.data_classes.transformations.TransformationUpdate, Sequence[Union[cognite.client.data_classes.transformations.Transformation, cognite.client.data_classes.transformations.TransformationUpdate]]]) → Union[cognite.client.data_classes.transformations.Transformation, cognite.client.data_classes.transformations.TransformationList]

Update one or more transformations

Parameters:item (Union[Transformation, TransformationUpdate, List[Union[Transformation, TransformationUpdate]]]) – Transformation(s) to update
Returns:Updated transformation(s)
Return type:Union[Transformation, TransformationList]

Examples

Update a transformation that you have fetched. This will perform a full update of the transformation:

>>> from cognite.client import CogniteClient
>>> c = CogniteClient()
>>> transformation = c.transformations.retrieve(id=1)
>>> transformation.query = "SELECT * FROM _cdf.assets"
>>> res = c.transformations.update(transformation)

Perform a partial update on a transformation, updating the query and making it private:

>>> from cognite.client import CogniteClient
>>> from cognite.client.data_classes import TransformationUpdate
>>> c = CogniteClient()
>>> my_update = TransformationUpdate(id=1).query.set("SELECT * FROM _cdf.assets").is_public.set(False)
>>> res = c.transformations.update(my_update)

Delete transformations

TransformationsAPI.delete(id: Union[int, Sequence[int], None] = None, external_id: Union[str, Sequence[str], None] = None, ignore_unknown_ids: bool = False) → None

Delete one or more transformations.

Parameters:
  • id (Union[int, List[int]) – Id or list of ids.
  • external_id (Union[str, List[str]]) – External ID or list of external ids.
  • ignore_unknown_ids (bool) – Ignore IDs and external IDs that are not found rather than throw an exception.
Returns:

None

Example

Delete transformations by id or external id:

>>> from cognite.client import CogniteClient
>>> c = CogniteClient()
>>> c.transformations.delete(id=[1,2,3], external_id="function3")

Transformation Schedules

Create transformation Schedules

TransformationSchedulesAPI.create(schedule: Union[cognite.client.data_classes.transformations.schedules.TransformationSchedule, Sequence[cognite.client.data_classes.transformations.schedules.TransformationSchedule]]) → Union[cognite.client.data_classes.transformations.schedules.TransformationSchedule, cognite.client.data_classes.transformations.schedules.TransformationScheduleList]

Schedule the specified transformation with the specified configuration(s).

Parameters:schedule (Union[TransformationSchedule, Sequence[TransformationSchedule]]) – Configuration or list of configurations of the schedules to create.
Returns:Created schedule(s)

Examples

Create new schedules:

>>> from cognite.client import CogniteClient
>>> from cognite.client.data_classes import TransformationSchedule
>>> c = CogniteClient()
>>> schedules = [TransformationSchedule(id = 1, interval = "0 * * * *"), TransformationSchedule(external_id="transformation2", interval = "5 * * * *"))]
>>> res = c.transformations.schedules.create(schedules)

Retrieve transformation schedules

TransformationSchedulesAPI.retrieve(id: Optional[int] = None, external_id: Optional[str] = None) → Optional[cognite.client.data_classes.transformations.schedules.TransformationSchedule]

Retrieve a single transformation schedule by the id or external id of its transformation.

Parameters:
  • id (int, optional) – transformation ID
  • external_id (str, optional) – transformation External ID
Returns:

Requested transformation schedule or None if it does not exist.

Return type:

Optional[TransformationSchedule]

Examples

Get transformation schedule by transformation id:

>>> from cognite.client import CogniteClient
>>> c = CogniteClient()
>>> res = c.transformations.schedules.retrieve(id=1)

Get transformation schedule by transformation external id:

>>> from cognite.client import CogniteClient
>>> c = CogniteClient()
>>> res = c.transformations.schedules.retrieve(external_id="1")

Retrieve multiple transformation schedules

TransformationSchedulesAPI.retrieve_multiple(ids: Optional[Sequence[int]] = None, external_ids: Optional[Sequence[str]] = None, ignore_unknown_ids: bool = False) → cognite.client.data_classes.transformations.schedules.TransformationScheduleList

Retrieve multiple transformation schedules by the ids or external ids of the corresponding transformations.

Parameters:
  • ids (int, optional) – transformation IDs
  • external_ids (str, optional) – transformation External IDs
  • ignore_unknown_ids (bool) – Ignore IDs and external IDs that are not found rather than throw an exception.
Returns:

Requested transformation schedules.

Return type:

TransformationScheduleList

Examples

Get transformation schedules by transformation ids:

>>> from cognite.client import CogniteClient
>>> c = CogniteClient()
>>> res = c.transformations.schedules.retrieve_multiple(ids=[1, 2, 3])

Get transformation schedules by transformation external ids:

>>> from cognite.client import CogniteClient
>>> c = CogniteClient()
>>> res = c.transformations.schedules.retrieve_multiple(external_ids=["t1", "t2"])

List transformation schedules

TransformationSchedulesAPI.list(include_public: bool = True, limit: Optional[int] = 25) → cognite.client.data_classes.transformations.schedules.TransformationScheduleList

List all transformation schedules.

Parameters:
  • include_public (bool) – Whether public transformations should be included in the results. (default true).
  • cursor (str) – Cursor for paging through results.
  • limit (int) – Limits the number of results to be returned. To retrieve all results use limit=-1, default limit is 25.
Returns:

List of schedules

Return type:

TransformationScheduleList

Example

List schedules:

>>> from cognite.client import CogniteClient
>>> c = CogniteClient()
>>> schedules_list = c.transformations.schedules.list()

Update transformation schedules

TransformationSchedulesAPI.update(item: Union[cognite.client.data_classes.transformations.schedules.TransformationSchedule, cognite.client.data_classes.transformations.schedules.TransformationScheduleUpdate, Sequence[Union[cognite.client.data_classes.transformations.schedules.TransformationSchedule, cognite.client.data_classes.transformations.schedules.TransformationScheduleUpdate]]]) → Union[cognite.client.data_classes.transformations.schedules.TransformationSchedule, cognite.client.data_classes.transformations.schedules.TransformationScheduleList]

Update one or more transformation schedules

Parameters:item (Union[TransformationSchedule, TransformationScheduleUpdate, Sequence[Union[TransformationSchedule, TransformationScheduleUpdate]]]) – Transformation schedule(s) to update
Returns:Updated transformation schedule(s)
Return type:Union[TransformationSchedule, TransformationScheduleList]

Examples

Update a transformation schedule that you have fetched. This will perform a full update of the schedule:

>>> from cognite.client import CogniteClient
>>> c = CogniteClient()
>>> transformation_schedule = c.transformations.schedules.retrieve(id=1)
>>> transformation_schedule.is_paused = True
>>> res = c.transformations.schedules.update(transformation_schedule)

Perform a partial update on a transformation schedule, updating the interval and unpausing it:

>>> from cognite.client import CogniteClient
>>> from cognite.client.data_classes import TransformationScheduleUpdate
>>> c = CogniteClient()
>>> my_update = TransformationScheduleUpdate(id=1).interval.set("0 * * * *").is_paused.set(False)
>>> res = c.transformations.schedules.update(my_update)

Delete transformation schedules

TransformationSchedulesAPI.delete(id: Union[int, Sequence[int], None] = None, external_id: Union[str, Sequence[str], None] = None, ignore_unknown_ids: bool = False) → None

Unschedule one or more transformations

Parameters:
  • id (Union[int, Sequence[int]) – Id or list of ids
  • external_id (Union[str, Sequence[str]]) – External ID or list of external ids
  • ignore_unknown_ids (bool) – Ignore IDs and external IDs that are not found rather than throw an exception.
Returns:

None

Examples

Delete schedules by id or external id:

>>> from cognite.client import CogniteClient
>>> c = CogniteClient()
>>> c.transformations.schedules.delete(id=[1,2,3], external_id="3")

Transformation Notifications

Create transformation notifications

TransformationNotificationsAPI.create(notification: Union[cognite.client.data_classes.transformations.notifications.TransformationNotification, Sequence[cognite.client.data_classes.transformations.notifications.TransformationNotification]]) → Union[cognite.client.data_classes.transformations.notifications.TransformationNotification, cognite.client.data_classes.transformations.notifications.TransformationNotificationList]

Subscribe for notifications on the transformation errors.

Parameters:notification (Union[TransformationNotification, Sequence[TransformationNotification]]) – Notification or list of notifications to create.
Returns:Created notification(s)

Examples

Create new notifications:

>>> from cognite.client import CogniteClient
>>> from cognite.client.data_classes import TransformationNotification
>>> c = CogniteClient()
>>> notifications = [TransformationNotification(transformation_id = 1, destination="[email protected]"), TransformationNotification(transformation_external_id="transformation2", destination="[email protected]"))]
>>> res = c.transformations.notifications.create(notifications)

List transformation notifications

TransformationNotificationsAPI.list(transformation_id: Optional[int] = None, transformation_external_id: Optional[str] = None, destination: Optional[str] = None, limit: Optional[int] = 25) → cognite.client.data_classes.transformations.notifications.TransformationNotificationList

List notification subscriptions.

Parameters:
  • transformation_id (Optional[int]) – Filter by transformation internal numeric ID.
  • transformation_external_id (str) – Filter by transformation externalId.
  • destination (str) – Filter by notification destination.
  • limit (int) – Limits the number of results to be returned. To retrieve all results use limit=-1, default limit is 25.
Returns:

List of transformation notifications

Return type:

TransformationNotificationList

Example

List all notifications:

>>> from cognite.client import CogniteClient
>>> c = CogniteClient()
>>> notifications_list = c.transformations.notifications.list()

List all notifications by transformation id:

>>> from cognite.client import CogniteClient
>>> c = CogniteClient()
>>> notifications_list = c.transformations.notifications.list(transformation_id = 1)

Delete transformation notifications

TransformationNotificationsAPI.delete(id: Union[int, Sequence[int], None] = None) → None

Deletes the specified notification subscriptions on the transformation. Does nothing when the subscriptions already don’t exist

Parameters:id (Union[int, Sequence[int]) – Id or list of transformation notification ids
Returns:None

Examples

Delete schedules by id or external id:

>>> from cognite.client import CogniteClient
>>> c = CogniteClient()
>>> c.transformations.notifications.delete(id=[1,2,3])

Transformation Jobs

Retrieve transformation jobs

TransformationJobsAPI.retrieve(id: int) → Optional[cognite.client.data_classes.transformations.jobs.TransformationJob]

Retrieve a single transformation job by id.

Parameters:id (int) – Job internal Id
Returns:Requested transformation job or None if it does not exist.
Return type:Optional[TransformationJob]

Examples

Get transformation job by id:

>>> from cognite.client import CogniteClient
>>> c = CogniteClient()
>>> res = c.transformations.jobs.retrieve(id=1)
TransformationJobsAPI.retrieve_multiple(ids: Sequence[int], ignore_unknown_ids: bool = False) → cognite.client.data_classes.transformations.jobs.TransformationJobList

Retrieve multiple transformation jobs by id.

Parameters:
  • ids (Sequence[int]) – Job internal Ids
  • ignore_unknown_ids (bool) – Ignore IDs that are not found rather than throw an exception.
Returns:

Requested transformation jobs.

Return type:

TransformationJobList

Examples

Get jobs by id:

>>> from cognite.client import CogniteClient
>>> c = CogniteClient()
>>> res = c.transformations.jobs.retrieve_multiple(ids=[1, 2, 3])

List transformation jobs

TransformationJobsAPI.list(limit: Optional[int] = 25, transformation_id: Optional[int] = None, transformation_external_id: Optional[str] = None) → cognite.client.data_classes.transformations.jobs.TransformationJobList

List all running transformation jobs.

Parameters:
  • limit (int) – Limits the number of results to be returned. To retrieve all results use limit=-1, default limit is 25.
  • transformation_id (int) – Filters the results by the internal transformation id.
  • transformation_external_id (str) – Filters the results by the external transformation id.
Returns:

List of transformation jobs

Return type:

TransformationJobList

Example

List transformation jobs:

>>> from cognite.client import CogniteClient
>>> c = CogniteClient()
>>> transformation_jobs_list = c.transformations.jobs.list()

List transformation jobs of a single transformation:

>>> from cognite.client import CogniteClient
>>> c = CogniteClient()
>>> transformation_jobs_list = c.transformations.jobs.list(transformation_id = 1)

Transformation Schema

Get transformation schema

TransformationSchemaAPI.retrieve(destination: cognite.client.data_classes.transformations.common.TransformationDestination, conflict_mode: Optional[str] = None) → cognite.client.data_classes.transformations.schema.TransformationSchemaColumnList

Get expected schema for a transformation destination.

Parameters:
  • destination (TransformationDestination) – destination for which the schema is requested.
  • conflict_mode (Optional[str]) – conflict mode for which the schema is requested.
Returns:

List of column descriptions

Return type:

TransformationSchemaColumnList

Example

Get the schema for a transformation producing assets:

>>> from cognite.client import CogniteClient
>>> from cognite.client.data_classes import TransformationDestination
>>> c = CogniteClient()
>>> columns = c.transformations.schema.retrieve(destination = TransformationDestination.assets())

Data classes

class cognite.client.data_classes.transformations.ContainsAny(tags: Optional[List[str]] = None)

Bases: cognite.client.data_classes.transformations.TagsFilter

Return transformations that has one of the tags specified.

Parameters:tags (List[str]) – The resource item contains at least one of the listed tags. The tags are defined by a list of external ids.

Examples

List only resources marked as a PUMP or as a VALVE:

>>> from cognite.client.data_classes import ContainsAny
>>> my_tag_filter = ContainsAny(tags=["PUMP", "VALVE"])
class cognite.client.data_classes.transformations.SessionDetails(session_id: Optional[int] = None, client_id: Optional[str] = None, project_name: Optional[str] = None)

Bases: object

Details of a source session which provid.

Parameters:
  • session_id (int) – CDF source session ID
  • client_id (str) – Idp source client ID
  • project_name (str) – CDF source project name
dump(camel_case: bool = False) → Dict[str, Any]

Dump the instance into a json serializable Python data type.

Parameters:camel_case (bool) – Use camelCase for attribute names. Defaults to False.
Returns:A dictionary representation of the instance.
Return type:Dict[str, Any]
class cognite.client.data_classes.transformations.Transformation(id: int = None, external_id: str = None, name: str = None, query: str = None, destination: TransformationDestination = None, conflict_mode: str = None, is_public: bool = True, ignore_null_fields: bool = False, source_oidc_credentials: Optional[OidcCredentials] = None, destination_oidc_credentials: Optional[OidcCredentials] = None, created_time: Optional[int] = None, last_updated_time: Optional[int] = None, owner: str = None, owner_is_current_user: bool = True, has_source_oidc_credentials: Optional[bool] = None, has_destination_oidc_credentials: Optional[bool] = None, running_job: TransformationJob = None, last_finished_job: TransformationJob = None, blocked: TransformationBlockedInfo = None, schedule: TransformationSchedule = None, data_set_id: int = None, cognite_client: CogniteClient = None, source_nonce: Optional[NonceCredentials] = None, destination_nonce: Optional[NonceCredentials] = None, source_session: Optional[SessionDetails] = None, destination_session: Optional[SessionDetails] = None, tags: Optional[List[str]] = None)

Bases: cognite.client.data_classes._base.CogniteResource

The transformations resource allows transforming data in CDF.

Parameters:
  • id (int) – A server-generated ID for the object.
  • external_id (str) – The external ID provided by the client. Must be unique for the resource type.
  • name (str) – The name of the Transformation.
  • query (str) – SQL query of the transformation.
  • destination (TransformationDestination) – see TransformationDestination for options.
  • conflict_mode (str) – What to do in case of id collisions: either “abort”, “upsert”, “update” or “delete”
  • is_public (bool) – Indicates if the transformation is visible to all in project or only to the owner.
  • ignore_null_fields (bool) – Indicates how null values are handled on updates: ignore or set null.
  • source_oidc_credentials (Optional[OidcCredentials]) – Configures the transformation to authenticate with the given oidc credentials key on the destination.
  • destination_oidc_credentials (Optional[OidcCredentials]) – Configures the transformation to authenticate with the given oidc credentials on the destination.
  • created_time (int) – The number of milliseconds since 00:00:00 Thursday, 1 January 1970, Coordinated Universal Time (UTC), minus leap seconds.
  • last_updated_time (int) – The number of milliseconds since 00:00:00 Thursday, 1 January 1970, Coordinated Universal Time (UTC), minus leap seconds.
  • owner (str) – Owner of the transformation: requester’s identity.
  • owner_is_current_user (bool) – Indicates if the transformation belongs to the current user.
  • has_source_oidc_credentials (bool) – Indicates if the transformation is configured with a source oidc credentials set.
  • has_destination_oidc_credentials (bool) – Indicates if the transformation is configured with a destination oidc credentials set.
  • running_job (TransformationJob) – Details for the job of this transformation currently running.
  • last_finished_job (TransformationJob) – Details for the last finished job of this transformation.
  • blocked (TransformationBlockedInfo) – Provides reason and time if the transformation is blocked.
  • schedule (TransformationSchedule) – Details for the schedule if the transformation is scheduled.
  • cognite_client (CogniteClient) – The client to associate with this object.
  • source_nonce (NonceCredentials) – Single use credentials to bind to a CDF session for reading.
  • destination_nonce (NonceCredentials) – Single use credentials to bind to a CDF session for writing.
  • source_session (SessionDetails) – Details for the session used to read from the source project.
  • destination_session (SessionDetails) – Details for the session used to write to the destination project.
dump(camel_case: bool = False) → Dict[str, Any]

Dump the instance into a json serializable Python data type.

Parameters:camel_case (bool) – Use camelCase for attribute names. Defaults to False.
Returns:A dictionary representation of the instance.
Return type:Dict[str, Any]
class cognite.client.data_classes.transformations.TransformationFilter(include_public: bool = True, name_regex: Optional[str] = None, query_regex: Optional[str] = None, destination_type: Optional[str] = None, conflict_mode: Optional[str] = None, cdf_project_name: Optional[str] = None, has_blocked_error: Optional[bool] = None, created_time: Union[Dict[str, Any], cognite.client.data_classes.shared.TimestampRange, None] = None, last_updated_time: Union[Dict[str, Any], cognite.client.data_classes.shared.TimestampRange, None] = None, data_set_ids: Optional[List[Dict[str, Any]]] = None, tags: Optional[cognite.client.data_classes.transformations.TagsFilter] = None)

Bases: cognite.client.data_classes._base.CogniteFilter

No description.

Parameters:
  • include_public (bool) – Whether public transformations should be included in the results. The default is true.
  • name_regex (str) – Regex expression to match the transformation name
  • query_regex (str) – Regex expression to match the transformation query
  • destination_type (str) – Transformation destination resource name to filter by.
  • conflict_mode (str) – Filters by a selected transformation action type: abort/create, upsert, update, delete
  • cdf_project_name (str) – Project name to filter by configured source and destination project
  • has_blocked_error (bool) – Whether only the blocked transformations should be included in the results.
  • created_time (Union[Dict[str, Any], TimestampRange]) – Range between two timestamps
  • last_updated_time (Union[Dict[str, Any], TimestampRange]) – Range between two timestamps
  • data_set_ids (List[Dict[str, Any]]) – Return only transformations in the specified data sets with these ids.
  • tags (TagsFilter) – Return only the resource matching the specified tags constraints. It only supports ContainsAny as of now.
dump(camel_case: bool = True) → Dict[str, Any]

Dump the instance into a json serializable Python data type.

Parameters:camel_case (bool) – Use camelCase for attribute names. Defaults to False.
Returns:A dictionary representation of the instance.
Return type:Dict[str, Any]
class cognite.client.data_classes.transformations.TransformationList(resources: Collection[Any], cognite_client: CogniteClient = None)

Bases: cognite.client.data_classes._base.CogniteResourceList

class cognite.client.data_classes.transformations.TransformationPreviewResult(schema: TransformationSchemaColumnList = None, results: List[Dict] = None, cognite_client: CogniteClient = None)

Bases: cognite.client.data_classes._base.CogniteResource

Allows previewing the result of a sql transformation before executing it.

Parameters:
  • schema (TransformationSchemaColumnList) – List of column descriptions.
  • results (List[Dict]) – List of resulting rows. Each row is a dictionary where the key is the column name and the value is the entrie.
dump(camel_case: bool = False) → Dict[str, Any]

Dump the instance into a json serializable Python data type.

Parameters:camel_case (bool) – Use camelCase for attribute names. Defaults to False.
Returns:A dictionary representation of the instance.
Return type:Dict[str, Any]
class cognite.client.data_classes.transformations.TransformationUpdate(id: Optional[int] = None, external_id: Optional[str] = None)

Bases: cognite.client.data_classes._base.CogniteUpdate

Changes applied to transformation

Parameters:
  • id (int) – A server-generated ID for the object.
  • external_id (str) – External Id provided by client. Should be unique within the project.
dump(camel_case: bool = True) → Dict[str, Any]

Dump the instance into a json serializable Python data type.

Returns:A dictionary representation of the instance.
Return type:Dict[str, Any]
class cognite.client.data_classes.transformations.schedules.TransformationSchedule(id: int = None, external_id: str = None, created_time: int = None, last_updated_time: int = None, interval: str = None, is_paused: bool = False, cognite_client: CogniteClient = None)

Bases: cognite.client.data_classes._base.CogniteResource

The transformation schedules resource allows running recurrent transformations.

Parameters:
  • id (int) – Transformation id.
  • external_id (str) – Transformation externalId.
  • created_time (int) – Time when the schedule was created.
  • last_updated_time (int) – Time when the schedule was last updated.
  • interval (str) – Cron expression describes when the function should be called. Use http://www.cronmaker.com to create a cron expression.
  • is_paused (bool) – If true, the transformation is not scheduled.
  • cognite_client (CogniteClient) – The client to associate with this object.
class cognite.client.data_classes.transformations.schedules.TransformationScheduleList(resources: Collection[Any], cognite_client: CogniteClient = None)

Bases: cognite.client.data_classes._base.CogniteResourceList

class cognite.client.data_classes.transformations.schedules.TransformationScheduleUpdate(id: Optional[int] = None, external_id: Optional[str] = None)

Bases: cognite.client.data_classes._base.CogniteUpdate

Changes applied to transformation schedule

Parameters:
  • id (int) – Transformation id.
  • external_id (str) – Transformation externalId.
class cognite.client.data_classes.transformations.notifications.TransformationNotification(id: int = None, transformation_id: int = None, transformation_external_id: str = None, destination: str = None, created_time: int = None, last_updated_time: int = None, cognite_client: CogniteClient = None)

Bases: cognite.client.data_classes._base.CogniteResource

The transformation notification resource allows configuring email alerts on events related to a transformation run.

Parameters:
  • id (int) – A server-generated ID for the object.
  • transformation_id (int) – Transformation Id.
  • transformation_external_id (str) – Transformation external Id.
  • destination (str) – Email address where notifications should be sent.
  • created_time (int) – Time when the notification was created.
  • last_updated_time (int) – Time when the notification was last updated.
  • cognite_client (CogniteClient) – The client to associate with this object.
class cognite.client.data_classes.transformations.notifications.TransformationNotificationFilter(transformation_id: Optional[int] = None, transformation_external_id: Optional[str] = None, destination: Optional[str] = None)

Bases: cognite.client.data_classes._base.CogniteFilter

Parameters:
  • transformation_id (Optional[int]) – Filter by transformation internal numeric ID.
  • transformation_external_id (str) – Filter by transformation externalId.
  • destination (str) – Filter by notification destination.
class cognite.client.data_classes.transformations.notifications.TransformationNotificationList(resources: Collection[Any], cognite_client: CogniteClient = None)

Bases: cognite.client.data_classes._base.CogniteResourceList

class cognite.client.data_classes.transformations.jobs.TransformationJob(id: int = None, status: TransformationJobStatus = None, transformation_id: int = None, transformation_external_id: str = None, source_project: str = None, destination_project: str = None, destination: TransformationDestination = None, conflict_mode: str = None, query: str = None, error: str = None, ignore_null_fields: bool = False, created_time: int = None, started_time: int = None, finished_time: int = None, last_seen_time: int = None, cognite_client: CogniteClient = None)

Bases: cognite.client.data_classes._base.CogniteResource

The transformation job resource allows following the status of execution of a transformation run.

Parameters:
  • id (int) – A server-generated ID for the object.
  • status (TransformationJobStatus) – Status of the job.
  • transformation_id (int) – Server-generated ID of the transformation.
  • transformation_external_id (str) – external ID of the transformation.
  • source_project (str) – Name of the CDF project the data will be read from.
  • destination_project (str) – Name of the CDF project the data will be written to.
  • destination_type (str) – Target resource type of the transformation.
  • destination_database (str) – Target database if the destination type is raw.
  • destination_table (str) – Target table name if the destination type is RAW.
  • conflict_mode (str) – What to do in case of id collisions: either “abort”, “upsert”, “update” or “delete”.
  • query (str) – Query of the transformation that is being executed.
  • error (str) – Error message from the server.
  • ignore_null_fields (bool) – Indicates how null values are handled on updates: ignore or set null.
  • created_time (int) – Time when the job was created.
  • started_time (int) – Time when the job started running.
  • finished_time (int) – Time when the job finished running.
  • last_seen_time (int) – Time of the last status update from the job.
  • cognite_client (CogniteClient) – The client to associate with this object.
metrics() → cognite.client.data_classes.transformations.jobs.TransformationJobMetricList

Get job metrics.

update() → None

Get updated job status.

wait(polling_interval: float = 1, timeout: Optional[float] = None) → cognite.client.data_classes.transformations.jobs.TransformationJob

Waits for the job to finish.

Parameters:
  • polling_interval (float) – time (s) to wait between job status updates, default is one second.
  • timeout (Optional[float]) – maximum time (s) to wait, default is None (infinite time). Once the timeout is reached, it returns with the current status.
Returns:

self.

Return type:

TransformationJob

Examples

run transformations 1 and 2 in parallel, and run 3 once they finish successfully:

>>> from cognite.client import CogniteClient
>>> c = CogniteClient()
>>>
>>> job1 = c.transformations.run(id = 1, wait = False)
>>> job2 = c.transformations.run(id = 2, wait = False)
>>> job1.wait()
>>> job2.wait()
>>> if TransformationJobStatus.FAILED not in [job1.status, job2.status]:
>>>     c.transformations.run(id = 3, wait = False)

wait transformation for 5 minutes and do something if still running:

>>> from cognite.client import CogniteClient
>>> c = CogniteClient()
>>>
>>> job = c.transformations.run(id = 1, wait = False)
>>> job.wait(timeout = 5.0*60)
>>> if job.status == TransformationJobStatus.FAILED:
>>>     # do something if job failed
>>> elif job.status == TransformationJobStatus.COMPLETED:
>>>     # do something if job completed successfully
>>> else:
>>>     # do something if job is still running
wait_async(polling_interval: float = 1, timeout: Optional[float] = None) → cognite.client.data_classes.transformations.jobs.TransformationJob

Asyncio coroutine, waits for the job to finish asynchronously.

Parameters:
  • polling_interval (float) – time (s) to wait between job status updates, default is one second.
  • timeout (Optional[float]) – maximum time (s) to wait, default is None (infinite time). Once the timeout is reached, it returns with the current status.
Returns:

coroutine object that will finish when the job finishes and resolves to self.

Return type:

Awaitable[TransformationJob]

Examples

run transformations 1 and 2 in parallel, and run 3 once they finish successfully:

>>> from asyncio import ensure_future
>>> from cognite.client import CogniteClient
>>> c = CogniteClient()
>>>
>>> async def run_successive_transformations():
>>>     job1 = c.transformations.run(id = 1, wait = False)
>>>     job2 = c.transformations.run(id = 2, wait = False)
>>>     await job1.wait_async()
>>>     await job2.wait_async()
>>>     if TransformationJobStatus.FAILED not in [job1.status, job2.status]:
>>>         c.transformations.run(id = 3, wait = False)
>>>
>>> ensure_future(run_successive_transformations())

wait transformation for 5 minutes and do something if still running:

>>> from asyncio import ensure_future
>>> from cognite.client import CogniteClient
>>> c = CogniteClient()
>>>
>>> async def run_successive_transformations():
>>>     job = c.transformations.run(id = 1, wait = False)
>>>     await job.wait_async(timeout = 5.0*60)
>>>     if job.status == TransformationJobStatus.FAILED:
>>>         # do something if job failed
>>>     elif job.status == TransformationJobStatus.COMPLETED:
>>>         # do something if job completed successfully
>>>     else:
>>>         # do something if job is still running
>>>
>>> ensure_future(run_successive_transformations())
class cognite.client.data_classes.transformations.jobs.TransformationJobFilter(transformation_id: Optional[int] = None, transformation_external_id: Optional[str] = None)

Bases: cognite.client.data_classes._base.CogniteFilter

Parameters:
  • transformation_id (Optional[int]) – Filter jobs by transformation internal numeric ID.
  • transformation_external_id (str) – Filter jobs by transformation external ID.
class cognite.client.data_classes.transformations.jobs.TransformationJobList(resources: Collection[Any], cognite_client: CogniteClient = None)

Bases: cognite.client.data_classes._base.CogniteResourceList

class cognite.client.data_classes.transformations.jobs.TransformationJobMetric(id: int = None, timestamp: int = None, name: str = None, count: int = None, cognite_client: CogniteClient = None)

Bases: cognite.client.data_classes._base.CogniteResource

The transformation job metric resource allows following details of execution of a transformation run.

Parameters:
  • timestamp (int) – Time of the last metric update.
  • name (str) – Name of the metric.
  • count (int) – Value of the metric.
  • cognite_client (CogniteClient) – The client to associate with this object.
class cognite.client.data_classes.transformations.jobs.TransformationJobMetricList(resources: Collection[Any], cognite_client: CogniteClient = None)

Bases: cognite.client.data_classes._base.CogniteResourceList

class cognite.client.data_classes.transformations.jobs.TransformationJobStatus

Bases: str, enum.Enum

An enumeration.

class cognite.client.data_classes.transformations.schema.TransformationSchemaColumn(name: str = None, sql_type: str = None, type: TransformationSchemaType = None, nullable: bool = False, cognite_client: CogniteClient = None)

Bases: cognite.client.data_classes._base.CogniteResource

Represents a column of the expected sql structure for a destination type.

Parameters:
  • name (str) – Column name
  • sql_type (str) – Type of the column in sql format.
  • type (TransformationSchemaType) – Type of the column in json format.
  • nullable (bool) – Values for the column can be null or not
  • cognite_client (CogniteClient) – The client to associate with this object.
class cognite.client.data_classes.transformations.schema.TransformationSchemaColumnList(resources: Collection[Any], cognite_client: CogniteClient = None)

Bases: cognite.client.data_classes._base.CogniteResourceList

class cognite.client.data_classes.transformations.common.Edges(view: Optional[cognite.client.data_classes.transformations.common.ViewInfo] = None, instance_space: Optional[str] = None, edge_type: Optional[cognite.client.data_classes.transformations.common.EdgeType] = None)

Bases: cognite.client.data_classes.transformations.common.TransformationDestination

class cognite.client.data_classes.transformations.common.Instances(data_model: Optional[cognite.client.data_classes.transformations.common.DataModelInfo] = None, instance_space: Optional[str] = None)

Bases: cognite.client.data_classes.transformations.common.TransformationDestination

class cognite.client.data_classes.transformations.common.Nodes(view: Optional[cognite.client.data_classes.transformations.common.ViewInfo] = None, instance_space: Optional[str] = None)

Bases: cognite.client.data_classes.transformations.common.TransformationDestination

class cognite.client.data_classes.transformations.common.RawTable(database: Optional[str] = None, table: Optional[str] = None)

Bases: cognite.client.data_classes.transformations.common.TransformationDestination

class cognite.client.data_classes.transformations.common.SequenceRows(external_id: Optional[str] = None)

Bases: cognite.client.data_classes.transformations.common.TransformationDestination

class cognite.client.data_classes.transformations.common.TransformationBlockedInfo(reason: Optional[str] = None, created_time: Optional[int] = None)

Bases: object

Information about the reason why and when a transformation is blocked.

Parameters:
  • reason (str) – Reason why the transformation is blocked.
  • created_time (Optional[int]) – Timestamp when the transformation was blocked.
class cognite.client.data_classes.transformations.common.TransformationDestination(type: Optional[str] = None)

Bases: object

TransformationDestination has static methods to define the target resource type of a transformation

Parameters:type (str) – Used as data type identifier on transformation creation/retrieval.
static asset_hierarchy() → cognite.client.data_classes.transformations.common.TransformationDestination

To be used when the transformation is meant to produce asset hierarchies.

static assets() → cognite.client.data_classes.transformations.common.TransformationDestination

To be used when the transformation is meant to produce assets.

static data_sets() → cognite.client.data_classes.transformations.common.TransformationDestination

To be used when the transformation is meant to produce data sets.

static datapoints() → cognite.client.data_classes.transformations.common.TransformationDestination

To be used when the transformation is meant to produce numeric data points.

static edges(view: Optional[cognite.client.data_classes.transformations.common.ViewInfo] = None, instance_space: Optional[str] = None, edge_type: Optional[cognite.client.data_classes.transformations.common.EdgeType] = None) → cognite.client.data_classes.transformations.common.Edges
Parameters:
  • view (ViewInfo) – information of the view.
  • instance_space (str) – space id of the instance.
  • edge_type (EdgeType) – information about the type of the edge
Returns:

pointing to the target flexible data model.

Return type:

Edges

static events() → cognite.client.data_classes.transformations.common.TransformationDestination

To be used when the transformation is meant to produce events.

static files() → cognite.client.data_classes.transformations.common.TransformationDestination

To be used when the transformation is meant to produce files.

static instances(data_model: Optional[cognite.client.data_classes.transformations.common.DataModelInfo] = None, instance_space: Optional[str] = None) → cognite.client.data_classes.transformations.common.Instances
Parameters:
  • data_model (DataModelInfo) – information of the Data Model.
  • instance_space (str) – space id of the instance.
Returns:

pointing to the target centric data model.

Return type:

Instances

static labels() → cognite.client.data_classes.transformations.common.TransformationDestination

To be used when the transformation is meant to produce labels.

static nodes(view: Optional[cognite.client.data_classes.transformations.common.ViewInfo] = None, instance_space: Optional[str] = None) → cognite.client.data_classes.transformations.common.Nodes
Parameters:
  • view (ViewInfo) – information of the view.
  • instance_space (str) – space id of the instance.
Returns:

pointing to the target flexible data model.

Return type:

Nodes

static raw(database: str = '', table: str = '') → cognite.client.data_classes.transformations.common.RawTable

To be used when the transformation is meant to produce raw table rows.

Parameters:
  • database (str) – database name of the target raw table.
  • table (str) – name of the target raw table
Returns:

TransformationDestination pointing to the target table

static relationships() → cognite.client.data_classes.transformations.common.TransformationDestination

To be used when the transformation is meant to produce relationships.

static sequence_rows(external_id: str = '') → cognite.client.data_classes.transformations.common.SequenceRows

To be used when the transformation is meant to produce sequence rows.

Parameters:external_id (str) – Sequence external id.
Returns:TransformationDestination pointing to the target sequence rows
static sequences() → cognite.client.data_classes.transformations.common.TransformationDestination

To be used when the transformation is meant to produce sequences.

static string_datapoints() → cognite.client.data_classes.transformations.common.TransformationDestination

To be used when the transformation is meant to produce string data points.

static timeseries() → cognite.client.data_classes.transformations.common.TransformationDestination

To be used when the transformation is meant to produce time series.