Transformations¶
TransformationsAPI¶
Create transformations¶
-
TransformationsAPI.
create
(transformation: Union[cognite.client.data_classes.transformations.Transformation, Sequence[cognite.client.data_classes.transformations.Transformation]]) → Union[cognite.client.data_classes.transformations.Transformation, cognite.client.data_classes.transformations.TransformationList]¶ Create one or more transformations.
Parameters: transformation (Union[Transformation, List[Transformation]]) – Transformation or list of transformations to create. Returns: Created transformation(s) Examples
Create new transformations:
>>> from cognite.client import CogniteClient >>> from cognite.client.data_classes import Transformation, TransformationDestination >>> from cognite.client.data_classes.transformations.common import ViewInfo, EdgeType, DataModelInfo >>> c = CogniteClient() >>> transformations = [ >>> Transformation( >>> name="transformation1", >>> destination=TransformationDestination.assets() >>> ), >>> Transformation( >>> name="transformation2", >>> destination=TransformationDestination.raw("myDatabase", "myTable") >>> ), >>> Transformation( >>> name="transformation3", >>> view = ViewInfo(space="TypeSpace", external_id="TypeExtId", version="version"), >>> destination=TransformationDestination.nodes(view, "InstanceSpace") >>> ), >>> Transformation( >>> name="transformation4", >>> view = ViewInfo(space="TypeSpace", external_id="TypeExtId", version="version"), >>> destination=TransformationDestination.edges(view, "InstanceSpace") >>> ), >>> Transformation( >>> name="transformation5", >>> edge_type = EdgeType(space="TypeSpace", external_id="TypeExtId"), >>> destination=TransformationDestination.edges(edge_type,"InstanceSpace") >>> ), >>> Transformation( >>> name="transformation6", >>> data_model = DataModelInfo(space="modelSpace", external_id="modelExternalId",version="modelVersion",destination_type="viewExternalId"), >>> destination=TransformationDestination.instances(data_model,"InstanceSpace") >>> ), >>> Transformation( >>> name="transformation7", >>> data_model = DataModelInfo(space="modelSpace", external_id="modelExternalId",version="modelVersion",destination_type="viewExternalId", destination_relationship_from_type="connectionPropertyName"), >>> destination=TransformationDestination.instances(data_model,"InstanceSpace") >>> ), >>> ] >>> res = c.transformations.create(transformations)
Retrieve transformations by id¶
-
TransformationsAPI.
retrieve
(id: Optional[int] = None, external_id: Optional[str] = None) → Optional[cognite.client.data_classes.transformations.Transformation]¶ Retrieve a single transformation by id.
Parameters: id (int, optional) – ID Returns: Requested transformation or None if it does not exist. Return type: Optional[Transformation] Examples
Get transformation by id:
>>> from cognite.client import CogniteClient >>> c = CogniteClient() >>> res = c.transformations.retrieve(id=1)
Get transformation by external id:
>>> from cognite.client import CogniteClient >>> c = CogniteClient() >>> res = c.transformations.retrieve(external_id="1")
-
TransformationsAPI.
retrieve_multiple
(ids: Optional[Sequence[int]] = None, external_ids: Optional[Sequence[str]] = None, ignore_unknown_ids: bool = False) → cognite.client.data_classes.transformations.TransformationList¶ Retrieve multiple transformations.
Parameters: - ids (List[int]) – List of ids to retrieve.
- external_ids (List[str]) – List of external ids to retrieve.
- ignore_unknown_ids (bool) – Ignore IDs and external IDs that are not found rather than throw an exception.
Returns: Requested transformation or None if it does not exist.
Return type: Examples
Get multiple transformations:
>>> from cognite.client import CogniteClient >>> c = CogniteClient() >>> res = c.transformations.retrieve_multiple(ids=[1,2,3], external_ids=['transform-1','transform-2'])
Run transformations by id¶
-
TransformationsAPI.
run
(transformation_id: Optional[int] = None, transformation_external_id: Optional[str] = None, wait: bool = True, timeout: Optional[float] = None) → cognite.client.data_classes.transformations.jobs.TransformationJob¶ -
Parameters: - transformation_id (int) – Transformation internal id
- transformation_external_id (str) – Transformation external id
- wait (bool) – Wait until the transformation run is finished. Defaults to True.
- timeout (Optional[float]) – maximum time (s) to wait, default is None (infinite time). Once the timeout is reached, it returns with the current status. Won’t have any effect if wait is False.
Returns: Created transformation job
Examples
Run transformation to completion by id:
>>> from cognite.client import CogniteClient >>> c = CogniteClient() >>> >>> res = c.transformations.run(transformation_id = 1)
Start running transformation by id:
>>> from cognite.client import CogniteClient >>> c = CogniteClient() >>> >>> res = c.transformations.run(transformation_id = 1, wait = False)
-
TransformationsAPI.
run_async
(transformation_id: Optional[int] = None, transformation_external_id: Optional[str] = None, timeout: Optional[float] = None) → cognite.client.data_classes.transformations.jobs.TransformationJob¶ Run a transformation to completion asynchronously.
Parameters: - transformation_id (int) – internal Transformation id
- transformation_external_id (str) – external Transformation id
- timeout (Optional[float]) – maximum time (s) to wait, default is None (infinite time). Once the timeout is reached, it returns with the current status.
Returns: Completed (if finished) or running (if timeout reached) transformation job.
Examples
Run transformation asyncronously by id:
>>> import asyncio >>> from cognite.client import CogniteClient >>> >>> c = CogniteClient() >>> >>> async def run_transformation(): >>> res = await c.transformations.run_async(id = 1) >>> >>> loop = asyncio.get_event_loop() >>> loop.run_until_complete(run_transformation()) >>> loop.close()
Preview transformations¶
-
TransformationsAPI.
preview
(query: Optional[str] = None, convert_to_string: bool = False, limit: int = 100, source_limit: Optional[int] = 100, infer_schema_limit: Optional[int] = 1000) → cognite.client.data_classes.transformations.TransformationPreviewResult¶ Preview the result of a query.
Parameters: - query (str) – SQL query to run for preview.
- convert_to_string (bool) – Stringify values in the query results, default is False.
- limit (int) – Maximum number of rows to return in the final result, default is 100.
- source_limit (Union[int,str]) – Maximum number of items to read from the data source or None to run without limit, default is 100.
- infer_schema_limit – Limit for how many rows that are used for inferring result schema, default is 1000.
Returns: Result of the executed query
Examples
Preview transformation results as schema and list of rows:
>>> from cognite.client import CogniteClient >>> c = CogniteClient() >>> >>> query_result = c.transformations.preview(query="select * from _cdf.assets")
Preview transformation results as pandas dataframe:
>>> from cognite.client import CogniteClient >>> c = CogniteClient() >>> >>> df = c.transformations.preview(query="select * from _cdf.assets").to_pandas()
Cancel transformation run by id¶
-
TransformationsAPI.
cancel
(transformation_id: Optional[int] = None, transformation_external_id: Optional[str] = None) → None¶ Cancel a running transformation.
Parameters: - transformation_id (int) – Transformation internal id
- transformation_external_id (str) – Transformation external id
Examples
Wait transformation for 1 minute and cancel if still running:
>>> from cognite.client import CogniteClient >>> from cognite.client.data_classes import TransformationJobStatus >>> c = CogniteClient() >>> >>> res = c.transformations.run(id = 1, timeout = 60.0) >>> if res.status == TransformationJobStatus.RUNNING: >>> res.cancel()
List transformations¶
-
TransformationsAPI.
list
(include_public: bool = True, name_regex: Optional[str] = None, query_regex: Optional[str] = None, destination_type: Optional[str] = None, conflict_mode: Optional[str] = None, cdf_project_name: Optional[str] = None, has_blocked_error: Optional[bool] = None, created_time: Union[Dict[str, Any], cognite.client.data_classes.shared.TimestampRange, None] = None, last_updated_time: Union[Dict[str, Any], cognite.client.data_classes.shared.TimestampRange, None] = None, data_set_ids: Optional[List[int]] = None, data_set_external_ids: Optional[List[str]] = None, tags: Optional[cognite.client.data_classes.transformations.TagsFilter] = None, limit: Optional[int] = 25) → cognite.client.data_classes.transformations.TransformationList¶ -
Parameters: - include_public (bool) – Whether public transformations should be included in the results. (default true).
- name_regex (str) – Regex expression to match the transformation name
- query_regex (str) – Regex expression to match the transformation query
- destination_type (str) – Transformation destination resource name to filter by.
- conflict_mode (str) – Filters by a selected transformation action type: abort/create, upsert, update, delete
- cdf_project_name (str) – Project name to filter by configured source and destination project
- has_blocked_error (bool) – Whether only the blocked transformations should be included in the results.
- created_time (Union[Dict[str, Any], TimestampRange]) – Range between two timestamps
- last_updated_time (Union[Dict[str, Any], TimestampRange]) – Range between two timestamps
- data_set_ids (List[int]) – Return only transformations in the specified data sets with these ids.
- data_set_external_ids (List[str]) – Return only transformations in the specified data sets with these external ids.
- tags (TagsFilter) – Return only the resource matching the specified tags constraints. It only supports ContainsAny as of now.
- limit (int) – Limits the number of results to be returned. To retrieve all results use limit=-1, default limit is 25.
Returns: List of transformations
Return type: Example
List transformations:
>>> from cognite.client import CogniteClient >>> c = CogniteClient() >>> transformations_list = c.transformations.list()
Update transformations¶
-
TransformationsAPI.
update
(item: Union[cognite.client.data_classes.transformations.Transformation, cognite.client.data_classes.transformations.TransformationUpdate, Sequence[Union[cognite.client.data_classes.transformations.Transformation, cognite.client.data_classes.transformations.TransformationUpdate]]]) → Union[cognite.client.data_classes.transformations.Transformation, cognite.client.data_classes.transformations.TransformationList]¶ Update one or more transformations
Parameters: item (Union[Transformation, TransformationUpdate, List[Union[Transformation, TransformationUpdate]]]) – Transformation(s) to update Returns: Updated transformation(s) Return type: Union[Transformation, TransformationList] Examples
Update a transformation that you have fetched. This will perform a full update of the transformation:
>>> from cognite.client import CogniteClient >>> c = CogniteClient() >>> transformation = c.transformations.retrieve(id=1) >>> transformation.query = "SELECT * FROM _cdf.assets" >>> res = c.transformations.update(transformation)
Perform a partial update on a transformation, updating the query and making it private:
>>> from cognite.client import CogniteClient >>> from cognite.client.data_classes import TransformationUpdate >>> c = CogniteClient() >>> my_update = TransformationUpdate(id=1).query.set("SELECT * FROM _cdf.assets").is_public.set(False) >>> res = c.transformations.update(my_update)
Delete transformations¶
-
TransformationsAPI.
delete
(id: Union[int, Sequence[int], None] = None, external_id: Union[str, Sequence[str], None] = None, ignore_unknown_ids: bool = False) → None¶ Delete one or more transformations.
Parameters: - id (Union[int, List[int]) – Id or list of ids.
- external_id (Union[str, List[str]]) – External ID or list of external ids.
- ignore_unknown_ids (bool) – Ignore IDs and external IDs that are not found rather than throw an exception.
Returns: None
Example
Delete transformations by id or external id:
>>> from cognite.client import CogniteClient >>> c = CogniteClient() >>> c.transformations.delete(id=[1,2,3], external_id="function3")
Transformation Schedules¶
Create transformation Schedules¶
-
TransformationSchedulesAPI.
create
(schedule: Union[cognite.client.data_classes.transformations.schedules.TransformationSchedule, Sequence[cognite.client.data_classes.transformations.schedules.TransformationSchedule]]) → Union[cognite.client.data_classes.transformations.schedules.TransformationSchedule, cognite.client.data_classes.transformations.schedules.TransformationScheduleList]¶ Schedule the specified transformation with the specified configuration(s).
Parameters: schedule (Union[TransformationSchedule, Sequence[TransformationSchedule]]) – Configuration or list of configurations of the schedules to create. Returns: Created schedule(s) Examples
Create new schedules:
>>> from cognite.client import CogniteClient >>> from cognite.client.data_classes import TransformationSchedule >>> c = CogniteClient() >>> schedules = [TransformationSchedule(id = 1, interval = "0 * * * *"), TransformationSchedule(external_id="transformation2", interval = "5 * * * *"))] >>> res = c.transformations.schedules.create(schedules)
Retrieve transformation schedules¶
-
TransformationSchedulesAPI.
retrieve
(id: Optional[int] = None, external_id: Optional[str] = None) → Optional[cognite.client.data_classes.transformations.schedules.TransformationSchedule]¶ Retrieve a single transformation schedule by the id or external id of its transformation.
Parameters: - id (int, optional) – transformation ID
- external_id (str, optional) – transformation External ID
Returns: Requested transformation schedule or None if it does not exist.
Return type: Optional[TransformationSchedule]
Examples
Get transformation schedule by transformation id:
>>> from cognite.client import CogniteClient >>> c = CogniteClient() >>> res = c.transformations.schedules.retrieve(id=1)
Get transformation schedule by transformation external id:
>>> from cognite.client import CogniteClient >>> c = CogniteClient() >>> res = c.transformations.schedules.retrieve(external_id="1")
Retrieve multiple transformation schedules¶
-
TransformationSchedulesAPI.
retrieve_multiple
(ids: Optional[Sequence[int]] = None, external_ids: Optional[Sequence[str]] = None, ignore_unknown_ids: bool = False) → cognite.client.data_classes.transformations.schedules.TransformationScheduleList¶ -
Parameters: - ids (int, optional) – transformation IDs
- external_ids (str, optional) – transformation External IDs
- ignore_unknown_ids (bool) – Ignore IDs and external IDs that are not found rather than throw an exception.
Returns: Requested transformation schedules.
Return type: Examples
Get transformation schedules by transformation ids:
>>> from cognite.client import CogniteClient >>> c = CogniteClient() >>> res = c.transformations.schedules.retrieve_multiple(ids=[1, 2, 3])
Get transformation schedules by transformation external ids:
>>> from cognite.client import CogniteClient >>> c = CogniteClient() >>> res = c.transformations.schedules.retrieve_multiple(external_ids=["t1", "t2"])
List transformation schedules¶
-
TransformationSchedulesAPI.
list
(include_public: bool = True, limit: Optional[int] = 25) → cognite.client.data_classes.transformations.schedules.TransformationScheduleList¶ List all transformation schedules.
Parameters: - include_public (bool) – Whether public transformations should be included in the results. (default true).
- cursor (str) – Cursor for paging through results.
- limit (int) – Limits the number of results to be returned. To retrieve all results use limit=-1, default limit is 25.
Returns: List of schedules
Return type: Example
List schedules:
>>> from cognite.client import CogniteClient >>> c = CogniteClient() >>> schedules_list = c.transformations.schedules.list()
Update transformation schedules¶
-
TransformationSchedulesAPI.
update
(item: Union[cognite.client.data_classes.transformations.schedules.TransformationSchedule, cognite.client.data_classes.transformations.schedules.TransformationScheduleUpdate, Sequence[Union[cognite.client.data_classes.transformations.schedules.TransformationSchedule, cognite.client.data_classes.transformations.schedules.TransformationScheduleUpdate]]]) → Union[cognite.client.data_classes.transformations.schedules.TransformationSchedule, cognite.client.data_classes.transformations.schedules.TransformationScheduleList]¶ Update one or more transformation schedules
Parameters: item (Union[TransformationSchedule, TransformationScheduleUpdate, Sequence[Union[TransformationSchedule, TransformationScheduleUpdate]]]) – Transformation schedule(s) to update Returns: Updated transformation schedule(s) Return type: Union[TransformationSchedule, TransformationScheduleList] Examples
Update a transformation schedule that you have fetched. This will perform a full update of the schedule:
>>> from cognite.client import CogniteClient >>> c = CogniteClient() >>> transformation_schedule = c.transformations.schedules.retrieve(id=1) >>> transformation_schedule.is_paused = True >>> res = c.transformations.schedules.update(transformation_schedule)
Perform a partial update on a transformation schedule, updating the interval and unpausing it:
>>> from cognite.client import CogniteClient >>> from cognite.client.data_classes import TransformationScheduleUpdate >>> c = CogniteClient() >>> my_update = TransformationScheduleUpdate(id=1).interval.set("0 * * * *").is_paused.set(False) >>> res = c.transformations.schedules.update(my_update)
Delete transformation schedules¶
-
TransformationSchedulesAPI.
delete
(id: Union[int, Sequence[int], None] = None, external_id: Union[str, Sequence[str], None] = None, ignore_unknown_ids: bool = False) → None¶ Unschedule one or more transformations
Parameters: - id (Union[int, Sequence[int]) – Id or list of ids
- external_id (Union[str, Sequence[str]]) – External ID or list of external ids
- ignore_unknown_ids (bool) – Ignore IDs and external IDs that are not found rather than throw an exception.
Returns: None
Examples
Delete schedules by id or external id:
>>> from cognite.client import CogniteClient >>> c = CogniteClient() >>> c.transformations.schedules.delete(id=[1,2,3], external_id="3")
Transformation Notifications¶
Create transformation notifications¶
-
TransformationNotificationsAPI.
create
(notification: Union[cognite.client.data_classes.transformations.notifications.TransformationNotification, Sequence[cognite.client.data_classes.transformations.notifications.TransformationNotification]]) → Union[cognite.client.data_classes.transformations.notifications.TransformationNotification, cognite.client.data_classes.transformations.notifications.TransformationNotificationList]¶ Subscribe for notifications on the transformation errors.
Parameters: notification (Union[TransformationNotification, Sequence[TransformationNotification]]) – Notification or list of notifications to create. Returns: Created notification(s) Examples
Create new notifications:
>>> from cognite.client import CogniteClient >>> from cognite.client.data_classes import TransformationNotification >>> c = CogniteClient() >>> notifications = [TransformationNotification(transformation_id = 1, destination="[email protected]"), TransformationNotification(transformation_external_id="transformation2", destination="[email protected]"))] >>> res = c.transformations.notifications.create(notifications)
List transformation notifications¶
-
TransformationNotificationsAPI.
list
(transformation_id: Optional[int] = None, transformation_external_id: Optional[str] = None, destination: Optional[str] = None, limit: Optional[int] = 25) → cognite.client.data_classes.transformations.notifications.TransformationNotificationList¶ List notification subscriptions.
Parameters: - transformation_id (Optional[int]) – Filter by transformation internal numeric ID.
- transformation_external_id (str) – Filter by transformation externalId.
- destination (str) – Filter by notification destination.
- limit (int) – Limits the number of results to be returned. To retrieve all results use limit=-1, default limit is 25.
Returns: List of transformation notifications
Return type: Example
List all notifications:
>>> from cognite.client import CogniteClient >>> c = CogniteClient() >>> notifications_list = c.transformations.notifications.list()
List all notifications by transformation id:
>>> from cognite.client import CogniteClient >>> c = CogniteClient() >>> notifications_list = c.transformations.notifications.list(transformation_id = 1)
Delete transformation notifications¶
-
TransformationNotificationsAPI.
delete
(id: Union[int, Sequence[int], None] = None) → None¶ -
Parameters: id (Union[int, Sequence[int]) – Id or list of transformation notification ids Returns: None Examples
Delete schedules by id or external id:
>>> from cognite.client import CogniteClient >>> c = CogniteClient() >>> c.transformations.notifications.delete(id=[1,2,3])
Transformation Jobs¶
Retrieve transformation jobs¶
-
TransformationJobsAPI.
retrieve
(id: int) → Optional[cognite.client.data_classes.transformations.jobs.TransformationJob]¶ Retrieve a single transformation job by id.
Parameters: id (int) – Job internal Id Returns: Requested transformation job or None if it does not exist. Return type: Optional[TransformationJob] Examples
Get transformation job by id:
>>> from cognite.client import CogniteClient >>> c = CogniteClient() >>> res = c.transformations.jobs.retrieve(id=1)
-
TransformationJobsAPI.
retrieve_multiple
(ids: Sequence[int], ignore_unknown_ids: bool = False) → cognite.client.data_classes.transformations.jobs.TransformationJobList¶ Retrieve multiple transformation jobs by id.
Parameters: - ids (Sequence[int]) – Job internal Ids
- ignore_unknown_ids (bool) – Ignore IDs that are not found rather than throw an exception.
Returns: Requested transformation jobs.
Return type: Examples
Get jobs by id:
>>> from cognite.client import CogniteClient >>> c = CogniteClient() >>> res = c.transformations.jobs.retrieve_multiple(ids=[1, 2, 3])
List transformation jobs¶
-
TransformationJobsAPI.
list
(limit: Optional[int] = 25, transformation_id: Optional[int] = None, transformation_external_id: Optional[str] = None) → cognite.client.data_classes.transformations.jobs.TransformationJobList¶ List all running transformation jobs.
Parameters: - limit (int) – Limits the number of results to be returned. To retrieve all results use limit=-1, default limit is 25.
- transformation_id (int) – Filters the results by the internal transformation id.
- transformation_external_id (str) – Filters the results by the external transformation id.
Returns: List of transformation jobs
Return type: Example
List transformation jobs:
>>> from cognite.client import CogniteClient >>> c = CogniteClient() >>> transformation_jobs_list = c.transformations.jobs.list()
List transformation jobs of a single transformation:
>>> from cognite.client import CogniteClient >>> c = CogniteClient() >>> transformation_jobs_list = c.transformations.jobs.list(transformation_id = 1)
Transformation Schema¶
Get transformation schema¶
-
TransformationSchemaAPI.
retrieve
(destination: cognite.client.data_classes.transformations.common.TransformationDestination, conflict_mode: Optional[str] = None) → cognite.client.data_classes.transformations.schema.TransformationSchemaColumnList¶ Get expected schema for a transformation destination.
Parameters: - destination (TransformationDestination) – destination for which the schema is requested.
- conflict_mode (Optional[str]) – conflict mode for which the schema is requested.
Returns: List of column descriptions
Return type: Example
Get the schema for a transformation producing assets:
>>> from cognite.client import CogniteClient >>> from cognite.client.data_classes import TransformationDestination >>> c = CogniteClient() >>> columns = c.transformations.schema.retrieve(destination = TransformationDestination.assets())
Data classes¶
-
class
cognite.client.data_classes.transformations.
ContainsAny
(tags: Optional[List[str]] = None)¶ Bases:
cognite.client.data_classes.transformations.TagsFilter
Return transformations that has one of the tags specified.
Parameters: tags (List[str]) – The resource item contains at least one of the listed tags. The tags are defined by a list of external ids. Examples
List only resources marked as a PUMP or as a VALVE:
>>> from cognite.client.data_classes import ContainsAny >>> my_tag_filter = ContainsAny(tags=["PUMP", "VALVE"])
-
class
cognite.client.data_classes.transformations.
SessionDetails
(session_id: Optional[int] = None, client_id: Optional[str] = None, project_name: Optional[str] = None)¶ Bases:
object
Details of a source session which provid.
Parameters: - session_id (int) – CDF source session ID
- client_id (str) – Idp source client ID
- project_name (str) – CDF source project name
-
dump
(camel_case: bool = False) → Dict[str, Any]¶ Dump the instance into a json serializable Python data type.
Parameters: camel_case (bool) – Use camelCase for attribute names. Defaults to False. Returns: A dictionary representation of the instance. Return type: Dict[str, Any]
-
class
cognite.client.data_classes.transformations.
Transformation
(id: int = None, external_id: str = None, name: str = None, query: str = None, destination: TransformationDestination = None, conflict_mode: str = None, is_public: bool = True, ignore_null_fields: bool = False, source_oidc_credentials: Optional[OidcCredentials] = None, destination_oidc_credentials: Optional[OidcCredentials] = None, created_time: Optional[int] = None, last_updated_time: Optional[int] = None, owner: str = None, owner_is_current_user: bool = True, has_source_oidc_credentials: Optional[bool] = None, has_destination_oidc_credentials: Optional[bool] = None, running_job: TransformationJob = None, last_finished_job: TransformationJob = None, blocked: TransformationBlockedInfo = None, schedule: TransformationSchedule = None, data_set_id: int = None, cognite_client: CogniteClient = None, source_nonce: Optional[NonceCredentials] = None, destination_nonce: Optional[NonceCredentials] = None, source_session: Optional[SessionDetails] = None, destination_session: Optional[SessionDetails] = None, tags: Optional[List[str]] = None)¶ Bases:
cognite.client.data_classes._base.CogniteResource
The transformations resource allows transforming data in CDF.
Parameters: - id (int) – A server-generated ID for the object.
- external_id (str) – The external ID provided by the client. Must be unique for the resource type.
- name (str) – The name of the Transformation.
- query (str) – SQL query of the transformation.
- destination (TransformationDestination) – see TransformationDestination for options.
- conflict_mode (str) – What to do in case of id collisions: either “abort”, “upsert”, “update” or “delete”
- is_public (bool) – Indicates if the transformation is visible to all in project or only to the owner.
- ignore_null_fields (bool) – Indicates how null values are handled on updates: ignore or set null.
- source_oidc_credentials (Optional[OidcCredentials]) – Configures the transformation to authenticate with the given oidc credentials key on the destination.
- destination_oidc_credentials (Optional[OidcCredentials]) – Configures the transformation to authenticate with the given oidc credentials on the destination.
- created_time (int) – The number of milliseconds since 00:00:00 Thursday, 1 January 1970, Coordinated Universal Time (UTC), minus leap seconds.
- last_updated_time (int) – The number of milliseconds since 00:00:00 Thursday, 1 January 1970, Coordinated Universal Time (UTC), minus leap seconds.
- owner (str) – Owner of the transformation: requester’s identity.
- owner_is_current_user (bool) – Indicates if the transformation belongs to the current user.
- has_source_oidc_credentials (bool) – Indicates if the transformation is configured with a source oidc credentials set.
- has_destination_oidc_credentials (bool) – Indicates if the transformation is configured with a destination oidc credentials set.
- running_job (TransformationJob) – Details for the job of this transformation currently running.
- last_finished_job (TransformationJob) – Details for the last finished job of this transformation.
- blocked (TransformationBlockedInfo) – Provides reason and time if the transformation is blocked.
- schedule (TransformationSchedule) – Details for the schedule if the transformation is scheduled.
- cognite_client (CogniteClient) – The client to associate with this object.
- source_nonce (NonceCredentials) – Single use credentials to bind to a CDF session for reading.
- destination_nonce (NonceCredentials) – Single use credentials to bind to a CDF session for writing.
- source_session (SessionDetails) – Details for the session used to read from the source project.
- destination_session (SessionDetails) – Details for the session used to write to the destination project.
-
dump
(camel_case: bool = False) → Dict[str, Any]¶ Dump the instance into a json serializable Python data type.
Parameters: camel_case (bool) – Use camelCase for attribute names. Defaults to False. Returns: A dictionary representation of the instance. Return type: Dict[str, Any]
-
class
cognite.client.data_classes.transformations.
TransformationFilter
(include_public: bool = True, name_regex: Optional[str] = None, query_regex: Optional[str] = None, destination_type: Optional[str] = None, conflict_mode: Optional[str] = None, cdf_project_name: Optional[str] = None, has_blocked_error: Optional[bool] = None, created_time: Union[Dict[str, Any], cognite.client.data_classes.shared.TimestampRange, None] = None, last_updated_time: Union[Dict[str, Any], cognite.client.data_classes.shared.TimestampRange, None] = None, data_set_ids: Optional[List[Dict[str, Any]]] = None, tags: Optional[cognite.client.data_classes.transformations.TagsFilter] = None)¶ Bases:
cognite.client.data_classes._base.CogniteFilter
No description.
Parameters: - include_public (bool) – Whether public transformations should be included in the results. The default is true.
- name_regex (str) – Regex expression to match the transformation name
- query_regex (str) – Regex expression to match the transformation query
- destination_type (str) – Transformation destination resource name to filter by.
- conflict_mode (str) – Filters by a selected transformation action type: abort/create, upsert, update, delete
- cdf_project_name (str) – Project name to filter by configured source and destination project
- has_blocked_error (bool) – Whether only the blocked transformations should be included in the results.
- created_time (Union[Dict[str, Any], TimestampRange]) – Range between two timestamps
- last_updated_time (Union[Dict[str, Any], TimestampRange]) – Range between two timestamps
- data_set_ids (List[Dict[str, Any]]) – Return only transformations in the specified data sets with these ids.
- tags (TagsFilter) – Return only the resource matching the specified tags constraints. It only supports ContainsAny as of now.
-
dump
(camel_case: bool = True) → Dict[str, Any]¶ Dump the instance into a json serializable Python data type.
Parameters: camel_case (bool) – Use camelCase for attribute names. Defaults to False. Returns: A dictionary representation of the instance. Return type: Dict[str, Any]
-
class
cognite.client.data_classes.transformations.
TransformationList
(resources: Collection[Any], cognite_client: CogniteClient = None)¶ Bases:
cognite.client.data_classes._base.CogniteResourceList
-
class
cognite.client.data_classes.transformations.
TransformationPreviewResult
(schema: TransformationSchemaColumnList = None, results: List[Dict] = None, cognite_client: CogniteClient = None)¶ Bases:
cognite.client.data_classes._base.CogniteResource
Allows previewing the result of a sql transformation before executing it.
Parameters: - schema (TransformationSchemaColumnList) – List of column descriptions.
- results (List[Dict]) – List of resulting rows. Each row is a dictionary where the key is the column name and the value is the entrie.
-
dump
(camel_case: bool = False) → Dict[str, Any]¶ Dump the instance into a json serializable Python data type.
Parameters: camel_case (bool) – Use camelCase for attribute names. Defaults to False. Returns: A dictionary representation of the instance. Return type: Dict[str, Any]
-
class
cognite.client.data_classes.transformations.
TransformationUpdate
(id: Optional[int] = None, external_id: Optional[str] = None)¶ Bases:
cognite.client.data_classes._base.CogniteUpdate
Changes applied to transformation
Parameters: - id (int) – A server-generated ID for the object.
- external_id (str) – External Id provided by client. Should be unique within the project.
-
dump
(camel_case: bool = True) → Dict[str, Any]¶ Dump the instance into a json serializable Python data type.
Returns: A dictionary representation of the instance. Return type: Dict[str, Any]
-
class
cognite.client.data_classes.transformations.schedules.
TransformationSchedule
(id: int = None, external_id: str = None, created_time: int = None, last_updated_time: int = None, interval: str = None, is_paused: bool = False, cognite_client: CogniteClient = None)¶ Bases:
cognite.client.data_classes._base.CogniteResource
The transformation schedules resource allows running recurrent transformations.
Parameters: - id (int) – Transformation id.
- external_id (str) – Transformation externalId.
- created_time (int) – Time when the schedule was created.
- last_updated_time (int) – Time when the schedule was last updated.
- interval (str) – Cron expression describes when the function should be called. Use http://www.cronmaker.com to create a cron expression.
- is_paused (bool) – If true, the transformation is not scheduled.
- cognite_client (CogniteClient) – The client to associate with this object.
-
class
cognite.client.data_classes.transformations.schedules.
TransformationScheduleList
(resources: Collection[Any], cognite_client: CogniteClient = None)¶ Bases:
cognite.client.data_classes._base.CogniteResourceList
-
class
cognite.client.data_classes.transformations.schedules.
TransformationScheduleUpdate
(id: Optional[int] = None, external_id: Optional[str] = None)¶ Bases:
cognite.client.data_classes._base.CogniteUpdate
Changes applied to transformation schedule
Parameters: - id (int) – Transformation id.
- external_id (str) – Transformation externalId.
-
class
cognite.client.data_classes.transformations.notifications.
TransformationNotification
(id: int = None, transformation_id: int = None, transformation_external_id: str = None, destination: str = None, created_time: int = None, last_updated_time: int = None, cognite_client: CogniteClient = None)¶ Bases:
cognite.client.data_classes._base.CogniteResource
The transformation notification resource allows configuring email alerts on events related to a transformation run.
Parameters: - id (int) – A server-generated ID for the object.
- transformation_id (int) – Transformation Id.
- transformation_external_id (str) – Transformation external Id.
- destination (str) – Email address where notifications should be sent.
- created_time (int) – Time when the notification was created.
- last_updated_time (int) – Time when the notification was last updated.
- cognite_client (CogniteClient) – The client to associate with this object.
-
class
cognite.client.data_classes.transformations.notifications.
TransformationNotificationFilter
(transformation_id: Optional[int] = None, transformation_external_id: Optional[str] = None, destination: Optional[str] = None)¶ Bases:
cognite.client.data_classes._base.CogniteFilter
Parameters: - transformation_id (Optional[int]) – Filter by transformation internal numeric ID.
- transformation_external_id (str) – Filter by transformation externalId.
- destination (str) – Filter by notification destination.
-
class
cognite.client.data_classes.transformations.notifications.
TransformationNotificationList
(resources: Collection[Any], cognite_client: CogniteClient = None)¶ Bases:
cognite.client.data_classes._base.CogniteResourceList
-
class
cognite.client.data_classes.transformations.jobs.
TransformationJob
(id: int = None, status: TransformationJobStatus = None, transformation_id: int = None, transformation_external_id: str = None, source_project: str = None, destination_project: str = None, destination: TransformationDestination = None, conflict_mode: str = None, query: str = None, error: str = None, ignore_null_fields: bool = False, created_time: int = None, started_time: int = None, finished_time: int = None, last_seen_time: int = None, cognite_client: CogniteClient = None)¶ Bases:
cognite.client.data_classes._base.CogniteResource
The transformation job resource allows following the status of execution of a transformation run.
Parameters: - id (int) – A server-generated ID for the object.
- status (TransformationJobStatus) – Status of the job.
- transformation_id (int) – Server-generated ID of the transformation.
- transformation_external_id (str) – external ID of the transformation.
- source_project (str) – Name of the CDF project the data will be read from.
- destination_project (str) – Name of the CDF project the data will be written to.
- destination_type (str) – Target resource type of the transformation.
- destination_database (str) – Target database if the destination type is raw.
- destination_table (str) – Target table name if the destination type is RAW.
- conflict_mode (str) – What to do in case of id collisions: either “abort”, “upsert”, “update” or “delete”.
- query (str) – Query of the transformation that is being executed.
- error (str) – Error message from the server.
- ignore_null_fields (bool) – Indicates how null values are handled on updates: ignore or set null.
- created_time (int) – Time when the job was created.
- started_time (int) – Time when the job started running.
- finished_time (int) – Time when the job finished running.
- last_seen_time (int) – Time of the last status update from the job.
- cognite_client (CogniteClient) – The client to associate with this object.
-
metrics
() → cognite.client.data_classes.transformations.jobs.TransformationJobMetricList¶ Get job metrics.
-
update
() → None¶ Get updated job status.
-
wait
(polling_interval: float = 1, timeout: Optional[float] = None) → cognite.client.data_classes.transformations.jobs.TransformationJob¶ Waits for the job to finish.
Parameters: - polling_interval (float) – time (s) to wait between job status updates, default is one second.
- timeout (Optional[float]) – maximum time (s) to wait, default is None (infinite time). Once the timeout is reached, it returns with the current status.
Returns: self.
Return type: Examples
run transformations 1 and 2 in parallel, and run 3 once they finish successfully:
>>> from cognite.client import CogniteClient >>> c = CogniteClient() >>> >>> job1 = c.transformations.run(id = 1, wait = False) >>> job2 = c.transformations.run(id = 2, wait = False) >>> job1.wait() >>> job2.wait() >>> if TransformationJobStatus.FAILED not in [job1.status, job2.status]: >>> c.transformations.run(id = 3, wait = False)
wait transformation for 5 minutes and do something if still running:
>>> from cognite.client import CogniteClient >>> c = CogniteClient() >>> >>> job = c.transformations.run(id = 1, wait = False) >>> job.wait(timeout = 5.0*60) >>> if job.status == TransformationJobStatus.FAILED: >>> # do something if job failed >>> elif job.status == TransformationJobStatus.COMPLETED: >>> # do something if job completed successfully >>> else: >>> # do something if job is still running
-
wait_async
(polling_interval: float = 1, timeout: Optional[float] = None) → cognite.client.data_classes.transformations.jobs.TransformationJob¶ Asyncio coroutine, waits for the job to finish asynchronously.
Parameters: - polling_interval (float) – time (s) to wait between job status updates, default is one second.
- timeout (Optional[float]) – maximum time (s) to wait, default is None (infinite time). Once the timeout is reached, it returns with the current status.
Returns: coroutine object that will finish when the job finishes and resolves to self.
Return type: Awaitable[TransformationJob]
Examples
run transformations 1 and 2 in parallel, and run 3 once they finish successfully:
>>> from asyncio import ensure_future >>> from cognite.client import CogniteClient >>> c = CogniteClient() >>> >>> async def run_successive_transformations(): >>> job1 = c.transformations.run(id = 1, wait = False) >>> job2 = c.transformations.run(id = 2, wait = False) >>> await job1.wait_async() >>> await job2.wait_async() >>> if TransformationJobStatus.FAILED not in [job1.status, job2.status]: >>> c.transformations.run(id = 3, wait = False) >>> >>> ensure_future(run_successive_transformations())
wait transformation for 5 minutes and do something if still running:
>>> from asyncio import ensure_future >>> from cognite.client import CogniteClient >>> c = CogniteClient() >>> >>> async def run_successive_transformations(): >>> job = c.transformations.run(id = 1, wait = False) >>> await job.wait_async(timeout = 5.0*60) >>> if job.status == TransformationJobStatus.FAILED: >>> # do something if job failed >>> elif job.status == TransformationJobStatus.COMPLETED: >>> # do something if job completed successfully >>> else: >>> # do something if job is still running >>> >>> ensure_future(run_successive_transformations())
-
class
cognite.client.data_classes.transformations.jobs.
TransformationJobFilter
(transformation_id: Optional[int] = None, transformation_external_id: Optional[str] = None)¶ Bases:
cognite.client.data_classes._base.CogniteFilter
Parameters: - transformation_id (Optional[int]) – Filter jobs by transformation internal numeric ID.
- transformation_external_id (str) – Filter jobs by transformation external ID.
-
class
cognite.client.data_classes.transformations.jobs.
TransformationJobList
(resources: Collection[Any], cognite_client: CogniteClient = None)¶ Bases:
cognite.client.data_classes._base.CogniteResourceList
-
class
cognite.client.data_classes.transformations.jobs.
TransformationJobMetric
(id: int = None, timestamp: int = None, name: str = None, count: int = None, cognite_client: CogniteClient = None)¶ Bases:
cognite.client.data_classes._base.CogniteResource
The transformation job metric resource allows following details of execution of a transformation run.
Parameters: - timestamp (int) – Time of the last metric update.
- name (str) – Name of the metric.
- count (int) – Value of the metric.
- cognite_client (CogniteClient) – The client to associate with this object.
-
class
cognite.client.data_classes.transformations.jobs.
TransformationJobMetricList
(resources: Collection[Any], cognite_client: CogniteClient = None)¶ Bases:
cognite.client.data_classes._base.CogniteResourceList
-
class
cognite.client.data_classes.transformations.jobs.
TransformationJobStatus
¶ Bases:
str
,enum.Enum
An enumeration.
-
class
cognite.client.data_classes.transformations.schema.
TransformationSchemaColumn
(name: str = None, sql_type: str = None, type: TransformationSchemaType = None, nullable: bool = False, cognite_client: CogniteClient = None)¶ Bases:
cognite.client.data_classes._base.CogniteResource
Represents a column of the expected sql structure for a destination type.
Parameters: - name (str) – Column name
- sql_type (str) – Type of the column in sql format.
- type (TransformationSchemaType) – Type of the column in json format.
- nullable (bool) – Values for the column can be null or not
- cognite_client (CogniteClient) – The client to associate with this object.
-
class
cognite.client.data_classes.transformations.schema.
TransformationSchemaColumnList
(resources: Collection[Any], cognite_client: CogniteClient = None)¶ Bases:
cognite.client.data_classes._base.CogniteResourceList
-
class
cognite.client.data_classes.transformations.common.
Edges
(view: Optional[cognite.client.data_classes.transformations.common.ViewInfo] = None, instance_space: Optional[str] = None, edge_type: Optional[cognite.client.data_classes.transformations.common.EdgeType] = None)¶ Bases:
cognite.client.data_classes.transformations.common.TransformationDestination
-
class
cognite.client.data_classes.transformations.common.
Instances
(data_model: Optional[cognite.client.data_classes.transformations.common.DataModelInfo] = None, instance_space: Optional[str] = None)¶ Bases:
cognite.client.data_classes.transformations.common.TransformationDestination
-
class
cognite.client.data_classes.transformations.common.
Nodes
(view: Optional[cognite.client.data_classes.transformations.common.ViewInfo] = None, instance_space: Optional[str] = None)¶ Bases:
cognite.client.data_classes.transformations.common.TransformationDestination
-
class
cognite.client.data_classes.transformations.common.
RawTable
(database: Optional[str] = None, table: Optional[str] = None)¶ Bases:
cognite.client.data_classes.transformations.common.TransformationDestination
-
class
cognite.client.data_classes.transformations.common.
SequenceRows
(external_id: Optional[str] = None)¶ Bases:
cognite.client.data_classes.transformations.common.TransformationDestination
-
class
cognite.client.data_classes.transformations.common.
TransformationBlockedInfo
(reason: Optional[str] = None, created_time: Optional[int] = None)¶ Bases:
object
Information about the reason why and when a transformation is blocked.
Parameters: - reason (str) – Reason why the transformation is blocked.
- created_time (Optional[int]) – Timestamp when the transformation was blocked.
-
class
cognite.client.data_classes.transformations.common.
TransformationDestination
(type: Optional[str] = None)¶ Bases:
object
TransformationDestination has static methods to define the target resource type of a transformation
Parameters: type (str) – Used as data type identifier on transformation creation/retrieval. -
static
asset_hierarchy
() → cognite.client.data_classes.transformations.common.TransformationDestination¶ To be used when the transformation is meant to produce asset hierarchies.
-
static
assets
() → cognite.client.data_classes.transformations.common.TransformationDestination¶ To be used when the transformation is meant to produce assets.
-
static
data_sets
() → cognite.client.data_classes.transformations.common.TransformationDestination¶ To be used when the transformation is meant to produce data sets.
-
static
datapoints
() → cognite.client.data_classes.transformations.common.TransformationDestination¶ To be used when the transformation is meant to produce numeric data points.
-
static
edges
(view: Optional[cognite.client.data_classes.transformations.common.ViewInfo] = None, instance_space: Optional[str] = None, edge_type: Optional[cognite.client.data_classes.transformations.common.EdgeType] = None) → cognite.client.data_classes.transformations.common.Edges¶ Parameters: - view (ViewInfo) – information of the view.
- instance_space (str) – space id of the instance.
- edge_type (EdgeType) – information about the type of the edge
Returns: pointing to the target flexible data model.
Return type:
-
static
events
() → cognite.client.data_classes.transformations.common.TransformationDestination¶ To be used when the transformation is meant to produce events.
-
static
files
() → cognite.client.data_classes.transformations.common.TransformationDestination¶ To be used when the transformation is meant to produce files.
-
static
instances
(data_model: Optional[cognite.client.data_classes.transformations.common.DataModelInfo] = None, instance_space: Optional[str] = None) → cognite.client.data_classes.transformations.common.Instances¶ Parameters: - data_model (DataModelInfo) – information of the Data Model.
- instance_space (str) – space id of the instance.
Returns: pointing to the target centric data model.
Return type:
-
static
labels
() → cognite.client.data_classes.transformations.common.TransformationDestination¶ To be used when the transformation is meant to produce labels.
-
static
nodes
(view: Optional[cognite.client.data_classes.transformations.common.ViewInfo] = None, instance_space: Optional[str] = None) → cognite.client.data_classes.transformations.common.Nodes¶ Parameters: - view (ViewInfo) – information of the view.
- instance_space (str) – space id of the instance.
Returns: pointing to the target flexible data model.
Return type:
-
static
raw
(database: str = '', table: str = '') → cognite.client.data_classes.transformations.common.RawTable¶ To be used when the transformation is meant to produce raw table rows.
Parameters: - database (str) – database name of the target raw table.
- table (str) – name of the target raw table
Returns: TransformationDestination pointing to the target table
-
static
relationships
() → cognite.client.data_classes.transformations.common.TransformationDestination¶ To be used when the transformation is meant to produce relationships.
-
static
sequence_rows
(external_id: str = '') → cognite.client.data_classes.transformations.common.SequenceRows¶ To be used when the transformation is meant to produce sequence rows.
Parameters: external_id (str) – Sequence external id. Returns: TransformationDestination pointing to the target sequence rows
-
static
sequences
() → cognite.client.data_classes.transformations.common.TransformationDestination¶ To be used when the transformation is meant to produce sequences.
-
static
string_datapoints
() → cognite.client.data_classes.transformations.common.TransformationDestination¶ To be used when the transformation is meant to produce string data points.
-
static
timeseries
() → cognite.client.data_classes.transformations.common.TransformationDestination¶ To be used when the transformation is meant to produce time series.
-
static