OpenTelemetryReportingTask

Description:

Reports metrics in OpenTelemetry format by exporting to an OTLP-compatible endpoint which can be used for external monitoring of the application. The reporting task exports a set of metrics regarding the Runtime Flow and JVM.

Tags:

reporting, opentelemetry, metrics, time series data

Properties:

In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values.

Display Name	API Name	Default Value	Allowable Values	Description
Export Endpoint	Export Endpoint	http://localhost:4317		The endpoint that will receive OTLP data exported from this reporting task.
Export Protocol	Export Protocol	GRPC	GRPC HTTP	The format to send metric data. Endpoints may support one or both.
Authentication Strategy	Authentication Strategy	None	Basic Authentication Bearer Token Client Certificate None	The strategy that the OTLP exporter will use to authenticate to the receiving endpoint.
SSL Context Service	SSL Context Service		Controller Service API: SSLContextService Implementations: StandardRestrictedSSLContextService StandardSSLContextService	The SSL Context Service used to provide client certificate information for TLS/SSL connections to the OTLP endpoint for metrics.
Basic Authentication Username	Basic Authentication Username			Username for Basic Authentication to OTLP Endpoint. This Property is only considered if the [Authentication Strategy] Property has a value of "Basic Authentication".
Basic Authentication Password	Basic Authentication Password			Password for Basic Authentication to OTLP Endpoint. Sensitive Property: true This Property is only considered if the [Authentication Strategy] Property has a value of "Basic Authentication".
Bearer Token	Bearer Token			Authorization Bearer Token for the OTLP Endpoint. Sensitive Property: true This Property is only considered if the [Authentication Strategy] Property has a value of "Bearer Token".
Resource Attributes	Resource Attributes			Comma-separated key=value pairs to use as Resource Attributes.
Export Retry Cache Size	Export Retry Cache Size	5		If an export fails or is blocked by an ongoing export, it will keep up to this number of exports in memory to retry during the next run of the reporting task. Exports are retained in an LRU cache, so the oldest exports are dropped if the cache fills up.
Metrics Included	Metrics Included	All	All Selected	Choose between exporting all metrics or some metrics.
Export Connection Metrics	Export Connection Metrics	true	true false	If set, export metrics from Connections. Set this to false to ignore metrics from Connections. This Property is only considered if the [Metrics Included] Property has a value of "Selected".
Export JVM Metrics	Export JVM Metrics	true	true false	If set, export metrics from the JVM. Set this to false to ignore metrics from the JVM. This Property is only considered if the [Metrics Included] Property has a value of "Selected".
Export Port Metrics	Export Port Metrics	true	true false	If set, export metrics from Ports. Set this to false to ignore metrics from Ports. This Property is only considered if the [Metrics Included] Property has a value of "Selected".
Export Processor Metrics	Export Processor Metrics	true	true false	If set, export metrics from Processors. Set this to false to ignore metrics from Processors. This Property is only considered if the [Metrics Included] Property has a value of "Selected".
Export Process Group Metrics	Export Process Group Metrics	true	true false	If set, export metrics from Process Groups. Set this to false to ignore metrics from Process Groups. This Property is only considered if the [Metrics Included] Property has a value of "Selected".
Export Remote Process Group Metrics	Export Remote Process Group Metrics	true	true false	If set, export metrics from Remote Process Groups. Set this to false to ignore metrics from Remote Process Groups. This Property is only considered if the [Metrics Included] Property has a value of "Selected".

State management:

This component does not store state.

Restricted:

This component is not restricted.

System Resource Considerations:

None specified

---

Want to monitor your data flow with the widely adopted, vendor-neutral industry standard?

This reporting task publishes OpenTelemetry metrics to an OTLP-compatible endpoint for external monitoring of the data flow.

Getting Started

To add an OpenTelemetry Reporting Task to your Datavolo Runtime:

1. Open Controller Settings under the top-right menu
2. Navigate to the Reporting Tasks tab
3. Use the right-hand [+] button to add a new reporting task
4. Search for and select OpenTelemetryReportingTask
5. Click the Add button
6. The new OpenTelemetryReportingTask instance now appears in the table of all configured reporting tasks.
7. Use the pencil icon to configure its properties.
8. Once it is valid, click the play icon to enable it.

Configuration

Export Protocol

Choose either GRPC or HTTP, depending on the OTLP receiver and the endpoint / port in use. Many OpenTelemetry Collectors will support both protocols, but on different ports.

GRPC will typically use port 4317, and HTTP will typically use port 4318.

Resource Attributes

The OpenTelemetry Reporting Task will use the comma-separated key=value pairs in this property to set Resource Attributes for the exported metrics. These attributes typically describe the Datavolo Runtime or data flow producing the telemetry.

For example:

Property	Value
Resource Attributes	service.name=datavolo-operations, service.version=1.0, deployment.environment=production

See https://opentelemetry.io/docs/concepts/resources for more details on OpenTelemetry resources.

Authentication

The OpenTelemetry Reporting Task provides 4 different authentication methods, covering a broad range of OTLP-compatible endpoints.

Basic Authentication

Choose Basic Authentication if the OTLP-compatible endpoint supports an Authorization header with a Username and Password.

Those properties will display once this option is selected, and will be required to enable the OpenTelemetry Reporting Task.

Bearer Token

Choose Bearer Token if the OTLP-compatible endpoint supports an Authorization header with a Bearer Token.

The OpenTelemetry Reporting Task will require that the Bearer Token property is also set.

Client Certificate

Choose Client Certificate for any OTLP-compatible endpoint that accepts certificate authentication.

The reporting task authenticates to the server with the key provided in the SSL Context Service's Keystore. As a result, the SSL Context Service property is required for this authentication mechanism.

This can be used, for example, to send to another Datavolo Runtime configured with a ListenOTLP processor.

None

The None authentication strategy provides the least overhead for co-located receivers, such as an OpenTelemetry Collector.

Warning: This authentication strategy is not recommended for exporting metrics outside of a local network.

Exported Metrics

The OpenTelemetry Reporting Task reports metrics for the following data flow components:

• Connections
• Input and Output Ports
• Process Groups
• Processors
• Remote Process Groups

It also exports Java Virtual Machine (JVM) metrics for analysis and tuning of flow configurations.

Connections

Metrics

Metric Name	Description	Unit
connection.input.bytes	Size of Items Input	bytes
connection.input.count	Count of Items Input	items
connection.output.bytes	Size of Items Output	bytes
connection.output.count	Count of Items Output	items
connection.queued.bytes	Size of Items Queued	bytes
connection.queued.bytes.max	Max Size of Items Queued	bytes
connection.queued.count	Count of Items Queued	items
connection.queued.count.max	Max Count of Items Queued	items
connection.queued.duration.total	Total Duration of Queued Items	milliseconds
connection.queued.duration.max	Max Duration of Queued Items	milliseconds
connection.backpressure.threshold.bytes	The maximum size of data in bytes that can be queued in this connection before it applies back pressure.	byte
connection.backpressure.threshold.objects	The configured maximum number of FlowFiles that can be queued in this connection before it applies back pressure.	items
connection.loadbalance.status.load_balance_not_configured	1 if the connection does not have a configured load balance setting. Otherwise, 0.	0 or 1
connection.loadbalance.status.load_balance_active	1 if the connection is load balancing across the cluster. Otherwise, 0.	0 or 1
connection.loadbalance.status.load_balance_inactive	1 if the connection is not load balancing across the cluster. Otherwise, 0.	0 or 1

Attributes

Each Connection's metrics includes the following attributes:

Attribute	Description
id	The unique identifier of the connection
name	The user-visible name of the connection
type	The fixed value of "connection"
source.id	The unique identifier of the component that is sending FlowFiles to this connection
source.name	The user-visible name of the component that is sending FlowFiles to this connection
destination.id	The unique identifier of the component that is receiving FlowFiles from this connection
destination.name	The user-visible name of the component that is receiving FlowFiles from this connection
group.id	The unique identifier of the Process Group that this Connection resides in

Input and Output Ports

Metrics

Metric Name	Description	Unit
port.thread.count.active	Number of Active Threads	threads
port.bytes.received	Number of Bytes Received	bytes
port.bytes.sent	Number of Bytes Sent	bytes
port.flowfiles.received	Number of FlowFiles Received	flowfiles
port.flowfiles.sent	Number of FlowFiles Sent	flowfiles
port.input.bytes	Size of Items Input	bytes
port.input.count	Count of Items Input	items
port.output.bytes	Size of Items Output	bytes
port.output.count	Count of Items Output	items

Attributes

Each Port's metrics includes the following attributes:

Attribute	Description
id	The unique identifier of the port
name	The user-visible name of the port
type	Either port-input or port-output
group.id	The unique identifier of the Process Group that this Port resides in

Process Groups

Metrics

Metric Name	Description	Unit
processgroup.thread.count.active	Number of Active Threads	threads
processgroup.thread.count.stateless	Number of Stateless Threads	threads
processgroup.thread.count.terminated	Number of Terminated Threads	threads
processgroup.bytes.read	Number of Bytes Read	bytes
processgroup.bytes.received	Number of Bytes Received	bytes
processgroup.bytes.transferred	Number of Bytes Transferred	bytes
processgroup.bytes.sent	Number of Bytes Sent	bytes
processgroup.bytes.written	Number of Bytes Written	bytes
processgroup.flowfiles.received	Number of FlowFiles Received	flowfiles
processgroup.flowfiles.sent	Number of FlowFiles Sent	flowfiles
processgroup.flowfiles.transferred	Number of FlowFiles Transferred	flowfiles
processgroup.input.count	Number of Items Input	items
processgroup.input.content.size	Size of Items Input	bytes
processgroup.output.count	Number of Items Output	items
processgroup.output.content.size	Size of Items Output	bytes
processgroup.queued.count	Number of Items Queued	items
processgroup.queued.content.size	Size of Items Queued	bytes
processgroup.time.processing	Time Spent Processing	nanoseconds

Attributes

Each Process Group's metrics includes the following attributes:

Attribute	Description
id	The unique identifier of the Process Group
name	The user-visible name of the Process Group
type	The fixed value process-group

Processors

Metrics

Metric Name	Description	Unit
processor.thread.count.active	Number of Active Threads	thread
processor.thread.count.terminated	Number of Terminated Threads	thread
processor.time.lineage.average	Average Lineage Duration	nanosecond
processor.invocations	Number of Invocations	invocations
processor.bytes.read	Number of Bytes Read	byte
processor.bytes.received	Number of Bytes Received	byte
processor.bytes.sent	Number of Bytes Sent	byte
processor.bytes.written	Number of Bytes Written	byte
processor.flowfiles.received	Number of FlowFiles Received	flowfiles
processor.flowfiles.removed	Number of FlowFiles Removed	flowfiles
processor.flowfiles.sent	Number of FlowFiles Sent	flowfiles
processor.input.count	Number of Items Input	item
processor.input.content.size	Size of Items Input	byte
processor.output.count	Number of Items Output	item
processor.output.content.size	Size of Items Output	byte
processor.time.processing	Time Spent Processing	nanosecond
processor.run.status.running	1 if running; 0 otherwise	binary
processor.run.status.stopped	1 if stopped; 0 otherwise	binary
processor.run.status.validating	1 if validating; 0 otherwise	binary
processor.run.status.invalid	1 if invalid; 0 otherwise	binary
processor.run.status.disabled	1 if disabled; 0 otherwise	binary
processor.counter	Value of the counter	count

Attributes

Each Processor's metrics includes the following attributes:

Attribute	Description
id	The unique identifier of the processor
name	The user-visible name of the Processor
type	The fixed value processor
execution.node	Either ALL or PRIMARY, depending on how this Processor is configured to run
group.id	The unique identifier of the Process Group that this Processor resides in

Additional Attributes for Counters

In addition to the standard Processor attributes above, processor.counter metrics include the following:

Attribute	Description
type	The fixed value counter
counter	The name of the counter

Remote Process Groups

Metrics

Metric Name	Description	Unit
remoteprocessgroup.thread.count.active	Number of Active Threads	threads
remoteprocessgroup.remote.port.count.active	Number of Active Remote Ports	ports
remoteprocessgroup.remote.port.count.inactive	Number of Inactive Remote Ports	ports
remoteprocessgroup.duration.lineage.average	Average Lineage Duration	nanoseconds
remoteprocessgroup.refresh.age	Time since last refresh	milliseconds
remoteprocessgroup.received.count	Number of Received Items	items
remoteprocessgroup.received.content.size	Size of Received Items	bytes
remoteprocessgroup.sent.count	Number of Sent Items	items
remoteprocessgroup.sent.content.size	Size of Sent Items	bytes
remoteprocessgroup.transmission.status.transmitting	1 if the Remote Process Group is transmitting. Otherwise, 0.	0 or 1
remoteprocessgroup.transmission.status.nottransmitting	0 if the Remote Process Group is transmitting. Otherwise, 1.	0 or 1

Attributes

Each Remote Process Group's metrics includes the following attributes:

Attribute	Description
id	The unique identifier of the remote process group
name	The user-visible name of the Remote Process Group
group.id	The unique identifier of the Process Group that this Remote Process Group resides in
authorization.issue	The Authorization used to access the Remote Process Group
target.uri	The URI of the Remote Process Group
type	The fixed value remote-process-group

JVM Metrics

Additionally, the OpenTelemetry Reporting Task exports the following JVM metrics:

JVM Metric Name	Description	Unit
jvm.memory.heap.used	JVM Heap Used	bytes
jvm.memory.heap.committed	JVM Heap Committed	bytes
jvm.memory.heap.max	JVM Heap Max	bytes
jvm.memory.heap.init	JVM Heap Init	bytes
jvm.memory.heap.usage	JVM Heap Usage	percentage
jvm.memory.non-heap.usage	JVM Non-Heap Usage	percentage
jvm.memory.total.init	JVM Memory Total Init	bytes
jvm.memory.total.used	JVM Memory Total Used	bytes
jvm.memory.total.max	JVM Memory Total Max	bytes
jvm.memory.total.committed	JVM Memory Total Committed	bytes
jvm.threads.count	Number of live threads	threads
jvm.threads.deadlocks	JVM Thread Deadlocks	threads
jvm.threads.daemon.count	Number of live daemon threads	threads
jvm.uptime	Number of seconds the JVM process has been running	seconds
jvm.file.descriptor.usage	Percentage of available file descriptors currently in use.	percentage
jvm.gc.G1-Concurrent-GC.runs	Total number of times that the G1 Concurrent Garbage Collection has run	runs
jvm.gc.G1-Concurrent-GC.time	Total amount of time that the G1 Concurrent Garbage Collection has been running	milliseconds
jvm.gc.G1-Young-Generation.runs	Total number of times that the G1 Young Generation has run	runs
jvm.gc.G1-Young-Generation.time	Total amount of time that the G1 Young Generation has been running	milliseconds
jvm.gc.G1-Old-Generation.runs	Total number of times that the G1 Old Generation has run	runs
jvm.gc.G1-Old-Generation.time	Total amount of time that the G1 Old Generation has been running	milliseconds

CPU Metrics

CPU Metric Name	Description	Unit
cores.available	The number of available cores for the Runtime	cores
cores.load	Either the system load average or -1 if it is not available	percentage

Attributes

Each CPU metric includes the following attributes:

Attribute	Description
id	The fixed value cpu
name	The name of the operating system
architecture	The architecture of the operating system
version	The version of the operating system

Storage Metrics

Storage Metric Name	Description	Unit
storage.free	The amount of free storage for a given repository	bytes
storage.used	The amount of used storage for a given repository	bytes

Attributes

Each storage metric includes the following attributes:

Attribute	Description
id	The unique identifier of the storage repository
name	Same as "id" and provided for consistency
storage.type	One of "flowfile", "content", or "provenance"