ControlRate
Description
Controls the rate at which data is transferred to follow-on processors. If you configure a very small Time Duration, then the accuracy of the throttle gets worse. You can improve this accuracy by decreasing the Yield Duration, at the expense of more Tasks given to the processor.
Tags
rate, rate control, throttle, throughput
Properties
In the list below required Properties are shown with an asterisk (*). Other properties are considered optional. The table also indicates any default values, and whether a property supports the NiFi Expression Language.
| Display Name | API Name | Default Value | Allowable Values | Description |
|---|---|---|---|---|
| Rate Control Criteria * | Rate Control Criteria | data rate |
| Indicates the criteria that is used to control the throughput rate. Changing this value resets the rate counters. |
| Time Duration * | Time Duration | 1 min | The amount of time to which the Maximum Rate pertains. Changing this value resets the rate counters. | |
| Maximum Rate | Maximum Rate | The maximum rate at which data should pass through this processor. The format of this property is expected to be a positive integer, or a Data Size (such as '1 MB') if Rate Control Criteria is set to 'data rate'. This property is only considered if:
| ||
| Maximum Data Rate | Maximum Data Rate | The maximum rate at which data should pass through this processor. The format of this property is expected to be a Data Size (such as '1 MB') representing bytes per Time Duration. This property is only considered if:
| ||
| Maximum FlowFile Rate | Maximum FlowFile Rate | The maximum rate at which FlowFiles should pass through this processor. The format of this property is expected to be a positive integer representing FlowFiles count per Time Duration This property is only considered if:
| ||
| Rate Exceeded Strategy * | Rate Exceeded Strategy | Hold FlowFile |
| Specifies how to handle an incoming FlowFile when the maximum data rate has been exceeded. |
| Rate Controlled Attribute | Rate Controlled Attribute | The name of an attribute whose values build toward the rate limit if Rate Control Criteria is set to 'attribute value'. The value of the attribute referenced by this property must be a positive long, or the FlowFile will be routed to failure. This value is ignored if Rate Control Criteria is not set to 'attribute value'. Changing this value resets the rate counters. This property is only considered if:
| ||
| Grouping Attribute | Grouping Attribute | By default, a single "throttle" is used for all FlowFiles. If this value is specified, a separate throttle is used for each value specified by the attribute with this name. Changing this value resets the rate counters. |
Dynamic Properties
This component does not support dynamic properties.
Relationships
| Name | Description |
|---|---|
| failure | FlowFiles will be routed to this relationship if they are missing a necessary Rate Controlled Attribute or the attribute is not in the expected format |
| success | FlowFiles are transferred to this relationship under normal conditions |
Reads Attributes
This processor does not read attributes.
Writes Attributes
This processor does not write attributes.
State Management
This component does not store state.
Restricted
This component is not restricted.
Input Requirement
This component requires an incoming relationship.
Example Use Cases
Use Case 1
Limit the rate at which data is sent to a downstream system with little to no bursts
Configuration
Set the "Rate Control Criteria" to data rate.
Set the "Time Duration" property to 1 sec.
Configure the "Maximum Rate" property to specify how much data should be allowed through each second.
For example, to allow through 8 MB per second, set "Maximum Rate" to 8 MB.
Use Case 2
Limit the rate at which FlowFiles are sent to a downstream system with little to no bursts
Configuration
Set the "Rate Control Criteria" to flowfile count.
Set the "Time Duration" property to 1 sec.
Configure the "Maximum Rate" property to specify how many FlowFiles should be allowed through each second.
For example, to allow through 100 FlowFiles per second, set "Maximum Rate" to 100.
Use Case 3
Reject requests that exceed a specific rate with little to no bursts
Configuration
Set the "Rate Control Criteria" to flowfile count.
Set the "Time Duration" property to 1 sec.
Set the "Rate Exceeded Strategy" property to Route to 'rate exceeded'.
Configure the "Maximum Rate" property to specify how many requests should be allowed through each second.
For example, to allow through 100 requests per second, set "Maximum Rate" to 100.
If more than 100 requests come in during any one second, the additional requests will be routed to rate exceeded instead of success.
Use Case 4
Reject requests that exceed a specific rate, allowing for bursts
Configuration
Set the "Rate Control Criteria" to flowfile count.
Set the "Time Duration" property to 1 min.
Set the "Rate Exceeded Strategy" property to Route to 'rate exceeded'.
Configure the "Maximum Rate" property to specify how many requests should be allowed through each minute.
For example, to allow through 100 requests per second, set "Maximum Rate" to 6000.
This will allow through 6,000 FlowFiles per minute, which averages to 100 FlowFiles per second. However, those 6,000 FlowFiles may come all within the first couple of
seconds, or they may come in over a period of 60 seconds. As a result, this gives us an average rate of 100 FlowFiles per second but allows for bursts of data.
If more than 6,000 requests come in during any one minute, the additional requests will be routed to rate exceeded instead of success.
System Resource Considerations
This component does not specify system resource considerations.