Hosted Extractors

Sources

Create new source

SourcesAPI.create(items: SourceWrite) → Source

SourcesAPI.create(items: Sequence[SourceWrite]) → SourceList

Parameters: items (SourceWrite | Sequence[SourceWrite]) – Source(s) to create.
Returns: Created source(s)
Return type: Source | SourceList

Examples

Create new source:

>>> from cognite.client import CogniteClient
>>> from cognite.client.data_classes.hosted_extractors import EventHubSourceWrite
>>> client = CogniteClient()
>>> source = EventHubSourceWrite('my_event_hub', 'http://myeventhub.com', "My EventHub", 'my_key', 'my_value')
>>> res = client.hosted_extractors.sources.create(source)

Delete source

SourcesAPI.delete(external_ids: Union[str, SequenceNotStr[str]], ignore_unknown_ids: bool = False, force: bool = False) → None

Delete one or more sources

Parameters

external_ids (str | SequenceNotStr[str]) – The external ID provided by the client. Must be unique for the resource type.
ignore_unknown_ids (bool) – Ignore external IDs that are not found rather than throw an exception.
force (bool) – Delete any jobs associated with each item.

Examples

Delete sources by id:

>>> from cognite.client import CogniteClient
>>> client = CogniteClient()
>>> client.hosted_extractors.sources.delete(["myMQTTSource", "MyEventHubSource"])

List sources

SourcesAPI.list(limit: int | None = 25) → SourceList

List sources

Parameters: limit (int | None) – Maximum number of sources to return. Defaults to 25. Set to -1, float(“inf”) or None to return all items.
Returns: List of requested sources
Return type: SourceList

Examples

List sources:

>>> from cognite.client import CogniteClient
>>> client = CogniteClient()
>>> source_list = client.hosted_extractors.sources.list(limit=5)

Iterate over sources:

>>> for source in client.hosted_extractors.sources:
...     source # do something with the source

Iterate over chunks of sources to reduce memory load:

>>> for source_list in client.hosted_extractors.sources(chunk_size=25):
...     source_list # do something with the sources

Retrieve sources

SourcesAPI.retrieve(external_ids: str, ignore_unknown_ids: bool = False) → Source

SourcesAPI.retrieve(external_ids: SequenceNotStr[str], ignore_unknown_ids: bool = False) → SourceList

Retrieve one or more sources.

Parameters

external_ids (str | SequenceNotStr[str]) – The external ID provided by the client. Must be unique for the resource type.
ignore_unknown_ids (bool) – Ignore external IDs that are not found rather than throw an exception.

Returns

Requested sources

Return type

Source | SourceList

Examples

>>> from cognite.client import CogniteClient
>>> client = CogniteClient()
>>> res = client.hosted_extractors.sources.retrieve('myMQTTSource')

Get multiple sources by id:

>>> res = client.hosted_extractors.sources.retrieve(["myMQTTSource", "MyEventHubSource"], ignore_unknown_ids=True)

Update sources

SourcesAPI.update(items: cognite.client.data_classes.hosted_extractors.sources.SourceWrite | cognite.client.data_classes.hosted_extractors.sources.SourceUpdate, mode: Literal['replace_ignore_null', 'patch', 'replace'] = 'replace_ignore_null') → Source

SourcesAPI.update(items: Sequence[cognite.client.data_classes.hosted_extractors.sources.SourceWrite | cognite.client.data_classes.hosted_extractors.sources.SourceUpdate], mode: Literal['replace_ignore_null', 'patch', 'replace'] = 'replace_ignore_null') → SourceList

Update one or more sources.

Parameters

items (SourceWrite | SourceUpdate | Sequence[SourceWrite | SourceUpdate]) – Source(s) to update.
mode (Literal['replace_ignore_null', 'patch', 'replace']) – How to update data when a non-update object is given (SourceWrite). If you use ‘replace_ignore_null’, only the fields you have set will be used to replace existing (default). Using ‘replace’ will additionally clear all the fields that are not specified by you. Last option, ‘patch’, will update only the fields you have set and for container-like fields such as metadata or labels, add the values to the existing. For more details, see Update and Upsert Mode Parameter.

Returns

Updated source(s)

Return type

Source | SourceList

Examples

Update source:

>>> from cognite.client import CogniteClient
>>> from cognite.client.data_classes.hosted_extractors import EventHubSourceUpdate
>>> client = CogniteClient()
>>> source = EventHubSourceUpdate('my_event_hub').event_hub_name.set("My Updated EventHub")
>>> res = client.hosted_extractors.sources.update(source)

Jobs

Create new job

JobsAPI.create(items: JobWrite) → Job

JobsAPI.create(items: Sequence[JobWrite]) → JobList

Create one or more jobs.

Parameters: items (JobWrite | Sequence[JobWrite]) – Job(s) to create.
Returns: Created job(s)
Return type: Job | JobList

Examples

Create new job:

>>> from cognite.client import CogniteClient
>>> from cognite.client.data_classes.hosted_extractors import EventHubSourceWrite
>>> client = CogniteClient()
>>> job_write = EventHubSourceWrite('my_event_hub', 'http://myeventhub.com', "My EventHub", 'my_key', 'my_value')
>>> job = client.hosted_extractors.jobs.create(job_write)

Delete job

JobsAPI.delete(external_ids: Union[str, SequenceNotStr[str]], ignore_unknown_ids: bool = False) → None

Delete one or more jobs

Parameters

external_ids (str | SequenceNotStr[str]) – The external ID provided by the client. Must be unique for the resource type.
ignore_unknown_ids (bool) – Ignore external IDs that are not found

Examples

Delete jobs by external id:

>>> from cognite.client import CogniteClient
>>> client = CogniteClient()
>>> client.hosted_extractors.jobs.delete(["myMQTTJob", "MyEventHubJob"])

List jobs

JobsAPI.list(limit: int | None = 25) → JobList

List jobs

Parameters: limit (int | None) – Maximum number of jobs to return. Defaults to 25. Set to -1, float(“inf”) or None to return all items.
Returns: List of requested jobs
Return type: JobList

Examples

List jobs:

>>> from cognite.client import CogniteClient
>>> client = CogniteClient()
>>> job_list = client.hosted_extractors.jobs.list(limit=5)

Iterate over jobs:

>>> for job in client.hosted_extractors.jobs:
...     job # do something with the job

Iterate over chunks of jobs to reduce memory load:

>>> for job_list in client.hosted_extractors.jobs(chunk_size=25):
...     job_list # do something with the jobs

Retrieve jobs

JobsAPI.retrieve(external_ids: str, ignore_unknown_ids: bool = False) → cognite.client.data_classes.hosted_extractors.jobs.Job | None

JobsAPI.retrieve(external_ids: SequenceNotStr[str], ignore_unknown_ids: bool = False) → JobList

Retrieve one or more jobs.

Parameters

external_ids (str | SequenceNotStr[str]) – The external ID provided by the client. Must be unique for the job type.
ignore_unknown_ids (bool) – Ignore external IDs that are not found

Returns

Requested jobs

Return type

Job | None | JobList

Examples

>>> from cognite.client import CogniteClient
>>> client = CogniteClient()
>>> res = client.hosted_extractors.jobs.retrieve('myJob')

Get multiple jobs by id:

>>> res = client.hosted_extractors.jobs.retrieve(["myJob", "myOtherJob"], ignore_unknown_ids=True)

Update job

JobsAPI.update(items: cognite.client.data_classes.hosted_extractors.jobs.JobWrite | cognite.client.data_classes.hosted_extractors.jobs.JobUpdate, mode: Literal['replace_ignore_null', 'patch', 'replace'] = 'replace_ignore_null') → Job

JobsAPI.update(items: Sequence[cognite.client.data_classes.hosted_extractors.jobs.JobWrite | cognite.client.data_classes.hosted_extractors.jobs.JobUpdate], mode: Literal['replace_ignore_null', 'patch', 'replace'] = 'replace_ignore_null') → JobList

Update one or more jobs.

Parameters

items (JobWrite | JobUpdate | Sequence[JobWrite | JobUpdate]) – Job(s) to update.
mode (Literal['replace_ignore_null', 'patch', 'replace']) – How to update data when a non-update object is given (JobWrite). If you use ‘replace_ignore_null’, only the fields you have set will be used to replace existing (default). Using ‘replace’ will additionally clear all the fields that are not specified by you. Last option, ‘patch’, will update only the fields you have set and for container-like fields such as metadata or labels, add the values to the existing. For more details, see Update and Upsert Mode Parameter.

Returns

Updated job(s)

Return type

Job | JobList

Examples

Update job:

>>> from cognite.client import CogniteClient
>>> from cognite.client.data_classes.hosted_extractors import EventHubSourceUpdate
>>> client = CogniteClient()
>>> job = EventHubSourceUpdate('my_event_hub').event_hub_name.set("My Updated EventHub")
>>> updated_job = client.hosted_extractors.jobs.update(job)

List job logs

JobsAPI.list_logs(job: Optional[str] = None, source: Optional[str] = None, destination: Optional[str] = None, limit: int | None = 25) → JobLogsList

List job logs.

Parameters

job (str | None) – Require returned logs to belong to the job given by this external ID.
source (str | None) – Require returned logs to belong to the any job with source given by this external ID.
destination (str | None) – Require returned logs to belong to the any job with destination given by this external ID.
limit (int | None) – Maximum number of logs to return. Defaults to 25. Set to -1, float(“inf”) or None to return all items.

Returns

List of requested job logs

Return type

JobLogsList

Examples

Reqests logs for a specific job:

>>> from cognite.client import CogniteClient
>>> client = CogniteClient()
>>> res = client.hosted_extractors.jobs.list_logs(job="myJob")

List job metrics

JobsAPI.list_metrics(job: Optional[str] = None, source: Optional[str] = None, destination: Optional[str] = None, limit: int | None = 25) → JobMetricsList

List job metrics.

Parameters

job (str | None) – Require returned metrics to belong to the job given by this external ID.
source (str | None) – Require returned metrics to belong to the any job with source given by this external ID.
destination (str | None) – Require returned metrics to belong to the any job with destination given by this external ID.
limit (int | None) – Maximum number of metrics to return. Defaults to 25. Set to -1, float(“inf”) or None to return all items.

Returns

List of requested job metrics

Return type

JobMetricsList

Examples

Reqests metrics for a specific job:

>>> from cognite.client import CogniteClient
>>> client = CogniteClient()
>>> res = client.hosted_extractors.jobs.list_metrics(job="myJob")

Destinations

Create new destinations

DestinationsAPI.create(items: DestinationWrite) → Destination

DestinationsAPI.create(items: Sequence[DestinationWrite]) → DestinationList

Create one or more destinations.

Parameters: items (DestinationWrite | Sequence[DestinationWrite]) – Destination(s) to create.
Returns: Created destination(s)
Return type: Destination | DestinationList

Examples

Create new destination:

>>> from cognite.client import CogniteClient
>>> from cognite.client.data_classes.hosted_extractors import DestinationWrite, SessionWrite
>>> client = CogniteClient()
>>> destination = DestinationWrite(external_id='my_dest', credentials=SessionWrite("my_nonce"), target_data_set_id=123)
>>> res = client.hosted_extractors.destinations.create(destination)

Delete destinations

DestinationsAPI.delete(external_ids: Union[str, SequenceNotStr[str]], ignore_unknown_ids: bool = False, force: bool = False) → None

Delete one or more destsinations

Parameters

external_ids (str | SequenceNotStr[str]) – The external ID provided by the client. Must be unique for the resource type.
ignore_unknown_ids (bool) – Ignore external IDs that are not found
force (bool) – Delete any jobs associated with each item.

Examples

Delete destinations by id:

>>> from cognite.client import CogniteClient
>>> client = CogniteClient()
>>> client.hosted_extractors.destinations.delete(["myDest", "MyDest2"])

List destinations

DestinationsAPI.list(limit: int | None = 25) → DestinationList

List destinations

Parameters: limit (int | None) – Maximum number of destinations to return. Defaults to 25. Set to -1, float(“inf”) or None to return all items.
Returns: List of requested destinations
Return type: DestinationList

Examples

List destinations:

>>> from cognite.client import CogniteClient
>>> client = CogniteClient()
>>> destination_list = client.hosted_extractors.destinations.list(limit=5)

Iterate over destinations:

>>> for destination in client.hosted_extractors.destinations:
...     destination # do something with the destination

Iterate over chunks of destinations to reduce memory load:

>>> for destination_list in client.hosted_extractors.destinations(chunk_size=25):
...     destination_list # do something with the destinationss

Retrieve destinations

DestinationsAPI.retrieve(external_ids: str, ignore_unknown_ids: bool = False) → Destination

DestinationsAPI.retrieve(external_ids: SequenceNotStr[str], ignore_unknown_ids: bool = False) → DestinationList

Retrieve one or more destinations.

Parameters

external_ids (str | SequenceNotStr[str]) – The external ID provided by the client. Must be unique for the resource type.
ignore_unknown_ids (bool) – Ignore external IDs that are not found

Returns

Requested destinations

Return type

Destination | DestinationList

Examples

>>> from cognite.client import CogniteClient
>>> client = CogniteClient()
>>> res = client.hosted_extractors.destinations.retrieve('myDestination')

Get multiple destinations by id:

>>> res = client.hosted_extractors.destinations.retrieve(["myDestination", "myDestination2"], ignore_unknown_ids=True)

Update destinations

DestinationsAPI.update(items: cognite.client.data_classes.hosted_extractors.destinations.DestinationWrite | cognite.client.data_classes.hosted_extractors.destinations.DestinationUpdate, mode: Literal['replace_ignore_null', 'patch', 'replace'] = 'replace_ignore_null') → Destination

DestinationsAPI.update(items: Sequence[cognite.client.data_classes.hosted_extractors.destinations.DestinationWrite | cognite.client.data_classes.hosted_extractors.destinations.DestinationUpdate], mode: Literal['replace_ignore_null', 'patch', 'replace'] = 'replace_ignore_null') → DestinationList

Update one or more destinations.

Parameters

items (DestinationWrite | DestinationUpdate | Sequence[DestinationWrite | DestinationUpdate]) – Destination(s) to update.
mode (Literal['replace_ignore_null', 'patch', 'replace']) – How to update data when a non-update object is given (DestinationWrite). If you use ‘replace_ignore_null’, only the fields you have set will be used to replace existing (default). Using ‘replace’ will additionally clear all the fields that are not specified by you. Last option, ‘patch’, will update only the fields you have set and for container-like fields such as metadata or labels, add the values to the existing. For more details, see Update and Upsert Mode Parameter.

Returns

Updated destination(s)

Return type

Destination | DestinationList

Examples

Update destination:

>>> from cognite.client import CogniteClient
>>> from cognite.client.data_classes.hosted_extractors import DestinationUpdate
>>> client = CogniteClient()
>>> destination = DestinationUpdate('my_dest').target_data_set_id.set(123)
>>> res = client.hosted_extractors.destinations.update(destination)

Mappings

Create new mappings

MappingsAPI.create(items: MappingWrite) → Mapping

MappingsAPI.create(items: Sequence[MappingWrite]) → MappingList

Create one or more mappings.

Parameters: items (MappingWrite | Sequence[MappingWrite]) – Mapping(s) to create.
Returns: Created mapping(s)
Return type: Mapping | MappingList

Examples

Create new mapping:

>>> from cognite.client import CogniteClient
>>> from cognite.client.data_classes.hosted_extractors import MappingWrite, CustomMapping
>>> client = CogniteClient()
>>> mapping = MappingWrite(external_id="my_mapping", mapping=CustomMapping("some expression"), published=True, input="json")
>>> res = client.hosted_extractors.mappings.create(mapping)

Delete mappings

MappingsAPI.delete(external_ids: Union[str, SequenceNotStr[str]], ignore_unknown_ids: bool = False, force: bool = False) → None

Delete one or more mappings

Parameters

external_ids (str | SequenceNotStr[str]) – The external ID provided by the client. Must be unique for the resource type.
ignore_unknown_ids (bool) – Ignore external IDs that are not found
force (bool) – Delete any jobs associated with each item.

Examples

Delete mappings by id:

>>> from cognite.client import CogniteClient
>>> client = CogniteClient()
>>> client.hosted_extractors.mappings.delete(["myMapping", "MyMapping2"])

List mappings

MappingsAPI.list(limit: int | None = 25) → MappingList

List mappings

Parameters: limit (int | None) – Maximum number of mappings to return. Defaults to 25. Set to -1, float(“inf”) or None to return all items.
Returns: List of requested mappings
Return type: MappingList

Examples

List mappings:

>>> from cognite.client import CogniteClient
>>> client = CogniteClient()
>>> mapping_list = client.hosted_extractors.mappings.list(limit=5)

Iterate over mappings:

>>> for mapping in client.hosted_extractors.mappings:
...     mapping # do something with the mapping

Iterate over chunks of mappings to reduce memory load:

>>> for mapping_list in client.hosted_extractors.mappings(chunk_size=25):
...     mapping_list # do something with the mappings

Retrieve mappings

MappingsAPI.retrieve(external_ids: str, ignore_unknown_ids: bool = False) → Mapping

MappingsAPI.retrieve(external_ids: SequenceNotStr[str], ignore_unknown_ids: bool = False) → MappingList

Retrieve one or more mappings.

Parameters

external_ids (str | SequenceNotStr[str]) – The external ID provided by the client. Must be unique for the resource type.
ignore_unknown_ids (bool) – Ignore external IDs that are not found

Returns

Requested mappings

Return type

Mapping | MappingList

Examples

>>> from cognite.client import CogniteClient
>>> client = CogniteClient()
>>> res = client.hosted_extractors.mappings.retrieve('myMapping')

Get multiple mappings by id:

>>> res = client.hosted_extractors.mappings.retrieve(["myMapping", "myMapping2"], ignore_unknown_ids=True)

Update mappings

MappingsAPI.update(items: cognite.client.data_classes.hosted_extractors.mappings.MappingWrite | cognite.client.data_classes.hosted_extractors.mappings.MappingUpdate) → Mapping

MappingsAPI.update(items: Sequence[cognite.client.data_classes.hosted_extractors.mappings.MappingWrite | cognite.client.data_classes.hosted_extractors.mappings.MappingUpdate]) → MappingList

Update one or more mappings.

Parameters: items (MappingWrite | MappingUpdate | Sequence[MappingWrite | MappingUpdate]) – Mapping(s) to update.
Returns: Updated mapping(s)
Return type: Mapping | MappingList

Examples

Update mapping:

>>> from cognite.client import CogniteClient
>>> from cognite.client.data_classes.hosted_extractors import MappingUpdate
>>> client = CogniteClient()
>>> mapping = MappingUpdate('my_mapping').published.set(False)
>>> res = client.hosted_extractors.mappings.update(mapping)