SearchElasticsearch

Description

A processor that allows the user to repeatedly run a paginated query (with aggregations) written with the Elasticsearch JSON DSL. Search After/Point in Time queries must include a valid "sort" field. The processor will retrieve multiple pages of results until either no more results are available or the Pagination Keep Alive expiration is reached, after which the query will restart with the first page of results being retrieved.

Tags

elasticsearch, elasticsearch5, elasticsearch6, elasticsearch7, elasticsearch8, json, page, query, scroll, search

Properties

In the list below required Properties are shown with an asterisk (*). Other properties are considered optional. The table also indicates any default values, and whether a property supports the NiFi Expression Language.

Display Name	API Name	Default Value	Allowable Values	Description
Query Definition Style *	el-rest-query-definition-style	FULL_QUERY	FULL_QUERY BUILD_QUERY	How the JSON Query will be defined for use by the processor.
Query	el-rest-query			A query in JSON syntax, not Lucene syntax. Ex: {"query":{"match":{"somefield":"somevalue"}}}. If the query is empty, a default JSON Object will be used, which will result in a "match_all" query in Elasticsearch. Supports Expression Language, using FlowFile attributes and Environment variables. This property is only considered if: the property Query Definition Style has a value of full
Query Clause	el-rest-query-clause			A "query" clause in JSON syntax, not Lucene syntax. Ex: {"match":{"somefield":"somevalue"}}. If the query is empty, a default JSON Object will be used, which will result in a "match_all" query in Elasticsearch. Supports Expression Language, using FlowFile attributes and Environment variables. This property is only considered if: the property Query Definition Style has a value of build
Size	es-rest-size			The maximum number of documents to retrieve in the query. If the query is paginated, this "size" applies to each page of the query, not the "size" of the entire result set. Supports Expression Language, using FlowFile attributes and Environment variables. This property is only considered if: the property Query Definition Style has a value of build
Sort	es-rest-query-sort			Sort results by one or more fields, in JSON syntax. Ex: [{"price" : {"order" : "asc", "mode" : "avg"}}, {"post_date" : {"format": "strict_date_optional_time_nanos"}}] Supports Expression Language, using FlowFile attributes and Environment variables. This property is only considered if: the property Query Definition Style has a value of build
Aggregations	es-rest-query-aggs			One or more query aggregations (or "aggs"), in JSON syntax. Ex: {"items": {"terms": {"field": "product", "size": 10}}} Supports Expression Language, using FlowFile attributes and Environment variables. This property is only considered if: the property Query Definition Style has a value of build
Fields	es-rest-query-fields			Fields of indexed documents to be retrieved, in JSON syntax. Ex: ["user.id", "http.response.*", {"field": "@timestamp", "format": "epoch_millis"}] Supports Expression Language, using FlowFile attributes and Environment variables. This property is only considered if: the property Query Definition Style has a value of build
Script Fields	es-rest-query-script-fields			Fields to created using script evaluation at query runtime, in JSON syntax. Ex: {"test1": {"script": {"lang": "painless", "source": "doc['price'].value * 2"}}, "test2": {"script": {"lang": "painless", "source": "doc['price'].value * params.factor", "params": {"factor": 2.0}}}} Supports Expression Language, using FlowFile attributes and Environment variables. This property is only considered if: the property Query Definition Style has a value of build
Query Attribute	el-query-attribute			If set, the executed query will be set on each result flowfile in the specified attribute. Supports Expression Language, using FlowFile attributes and Environment variables.
Index *	el-rest-fetch-index			The name of the index to use. Supports Expression Language, using FlowFile attributes and Environment variables.
Type	el-rest-type			The type of this document (used by Elasticsearch for indexing and searching). Supports Expression Language, using FlowFile attributes and Environment variables.
Max JSON Field String Length *	Max JSON Field String Length	20 MB		The maximum allowed length of a string value when parsing a JSON document or attribute.
Client Service *	el-rest-client-service		Controller Service: ElasticSearchClientService Implementations: ElasticSearchClientServiceImpl	An Elasticsearch client service to use for running queries.
Search Results Split *	el-rest-split-up-hits	PER_RESPONSE	PER_HIT PER_RESPONSE PER_QUERY	Output a flowfile containing all hits or one flowfile for each individual hit or one flowfile containing all hits from all paged responses.
Search Results Format *	el-rest-format-hits	FULL	FULL SOURCE_ONLY METADATA_ONLY	Format of Hits output.
Aggregation Results Split *	el-rest-split-up-aggregations	PER_RESPONSE	PER_HIT PER_RESPONSE	Output a flowfile containing all aggregations or one flowfile for each individual aggregation.
Aggregation Results Format *	el-rest-format-aggregations	FULL	FULL BUCKETS_ONLY METADATA_ONLY	Format of Aggregation output.
Output No Hits *	el-rest-output-no-hits	false	true false	Output a "hits" flowfile even if no hits found for query. If true, an empty "hits" flowfile will be output even if "aggregations" are output.
Pagination Type *	el-rest-pagination-type	SCROLL	SCROLL SEARCH_AFTER POINT_IN_TIME	Pagination method to use. Not all types are available for all Elasticsearch versions, check the Elasticsearch docs to confirm which are applicable and recommended for your service.
Pagination Keep Alive *	el-rest-pagination-keep-alive	10 mins		Pagination "keep_alive" period. Period Elasticsearch will keep the scroll/pit cursor alive in between requests (this is not the time expected for all pages to be returned, but the maximum allowed time for requests between page retrievals).

Dynamic Properties

Name	Value	Description
The name of a URL query parameter to add	The value of the URL query parameter	Adds the specified property name/value as a query parameter in the Elasticsearch URL used for processing. These parameters will override any matching parameters in the query request body. For SCROLL type queries, these parameters are only used in the initial (first page) query as the Elasticsearch Scroll API does not support the same query parameters for subsequent pages of data. Supports Expression Language: Yes, evaluated using Environment variables.

Relationships

Name	Description
aggregations	Aggregations are routed to this relationship.
hits	Search hits are routed to this relationship.

Reads Attributes

This processor does not read attributes.

Writes Attributes

Name	Description
aggregation.name	The name of the aggregation whose results are in the output flowfile
aggregation.number	The number of the aggregation whose results are in the output flowfile
elasticsearch.query.error	The error message provided by Elasticsearch if there is an error querying the index.
hit.count	The number of hits that are in the output flowfile
mime.type	application/json
page.number	The number of the page (request), starting from 1, in which the results were returned that are in the output flowfile

State Management

Scope	Description
LOCAL	The pagination state (scrollId, searchAfter, pitId, hitCount, pageCount, pageExpirationTimestamp) is retained in between invocations of this processor until the Scroll/PiT has expired (when the current time is later than the last query execution plus the Pagination Keep Alive interval).

Restricted

This component is not restricted.

Input Requirement

This component does not allow an incoming relationship.

System Resource Considerations

Scope	Description
MEMORY	Care should be taken on the size of each page because each response from Elasticsearch will be loaded into memory all at once and converted into the resulting flowfiles.

See Also

ConsumeElasticsearch, PaginatedJsonQueryElasticsearch