Skip to main content

DeduplicateRecord

Description​

This processor de-duplicates individual records within a record set. It can operate on a per-file basis using an in-memory hashset or bloom filter. When configured with a distributed map cache, it de-duplicates records across multiple files.

Tags​

change, dedupe, distinct, dupe, duplicate, filter, hash, modify, record, replace, text, unique, update

Properties​

In the list below required Properties are shown with an asterisk (*). Other properties are considered optional. The table also indicates any default values, and whether a property supports the NiFi Expression Language.

Display NameAPI NameDefault ValueAllowable ValuesDescription
Deduplication Strategy *deduplication-strategySingle File
  • Single File
  • Multiple Files
The strategy to use for detecting and routing duplicate records. The option for detecting duplicates across a single FlowFile operates in-memory, whereas detection spanning multiple FlowFiles utilises a distributed map cache.
Distributed Map Cache clientdistributed-map-cacheController Service:
DistributedMapCacheClient

Implementations:
HazelcastMapCacheClient
MapCacheClientService
RedisDistributedMapCacheClientService
SimpleRedisDistributedMapCacheClientService		This property is required when the deduplication strategy is set to 'multiple files.' The map cache will for each record, atomically check whether the cache key exists and if not, set it.

This property is only considered if:
  • the property Deduplication Strategy has a value of multiple
Cache Identifiercache-identifierAn optional expression language field that overrides the record's computed cache key. This field has an additional attribute available: ${record.hash.value}, which contains the cache key derived from dynamic properties (if set) or record fields.

Supports Expression Language, using FlowFile attributes and Environment variables.

This property is only considered if:
  • the property Deduplication Strategy has a value of multiple
Cache the Entry Identifier *put-cache-identifierfalse
  • true
  • false
For each record, check whether the cache identifier exists in the distributed map cache. If it doesn't exist and this property is true, put the identifier to the cache.

This property is only considered if:
  • the property Distributed Map Cache client has a value specified
Record Reader *record-readerController Service:
RecordReaderFactory

Implementations:
AvroReader
CEFReader
CSVReader
ExcelReader
GrokReader
JsonPathReader
JsonTreeReader
ReaderLookup
ScriptedReader
Syslog5424Reader
SyslogReader
WindowsEventLogReader
XMLReader
YamlTreeReader		Specifies the Controller Service to use for reading incoming data
Record Writer *record-writerController Service:
RecordSetWriterFactory

Implementations:
AvroRecordSetWriter
CSVRecordSetWriter
FreeFormTextRecordSetWriter
JsonRecordSetWriter
RecordSetWriterLookup
ScriptedRecordSetWriter
XMLRecordSetWriter		Specifies the Controller Service to use for writing out the records
Include Zero Record FlowFiles *include-zero-record-flowfilestrue
  • true
  • false
If a FlowFile sent to either the duplicate or non-duplicate relationships contains no records, a value of false in this property causes the FlowFile to be dropped. Otherwise, the empty FlowFile is emitted.
Record Hashing Algorithm *record-hashing-algorithmSHA-256
  • None
  • SHA-256
  • SHA-512
The algorithm used to hash the cache key.
Filter Type *filter-typeHashSet
  • HashSet
  • BloomFilter
The filter used to determine whether a record has been seen before based on the matching RecordPath criteria. If hash set is selected, a Java HashSet object will be used to deduplicate all encountered records. If the bloom filter option is selected, a bloom filter will be used. The bloom filter option is less memory intensive, but has a chance of having false positives.

This property is only considered if:
  • the property Deduplication Strategy has a value of single
Filter Capacity Hint *filter-capacity-hint25000An estimation of the total number of unique records to be processed. The more accurate this number is will lead to fewer false negatives on a BloomFilter.

This property is only considered if:
  • the property Filter Type has a value of bloom-filter
Bloom Filter Certaintybloom-filter-certainty0.10The desired false positive probability when using the BloomFilter type. Using a value of .05 for example, guarantees a five-percent probability that the result is a false positive. The closer to 1 this value is set, the more precise the result at the expense of more storage space utilization.

Dynamic Properties​

NameValueDescription
Name of the property.A valid RecordPath to the record field to be included in the cache key used for deduplication.A record's cache key is generated by combining the name of each dynamic property with its evaluated record value (as specified by the corresponding RecordPath).

Supports Expression Language: No

Relationships​

NameDescription
duplicateRecords detected as duplicates are routed to this relationship.
failureIf unable to communicate with the cache, the FlowFile will be penalized and routed to this relationship
non-duplicateRecords not found in the cache are routed to this relationship.
originalThe original input FlowFile is sent to this relationship unless a fatal error occurs.

Reads Attributes​

This processor does not read attributes.

Writes Attributes​

NameDescription
record.countNumber of records written to the destination FlowFile.

State Management​

This component does not store state.

Restricted​

This component is not restricted.

Input Requirement​

This component requires an incoming relationship.

System Resource Considerations​

ScopeDescription
CPUIf a more advanced hash algorithm is chosen, the amount of time required to hash any particular record could increase substantially.

See Also​

DistributedMapCacheClientService, DistributedMapCacheServer, DetectDuplicate