EvaluateRagRetrieval
Description
Calculates retrieval metrics (Precision@N, Recall@N, FScore@N, MAP@N, MRR) for a RAG system using an LLM as a judge. For each record, it uses both Precision and Recall prompts to evaluate the response, and adds the metrics as attributes to the FlowFile.
Tags
evaluation, fscore, llm, metrics, mrr, openai, precision, rag, recall, retrieval
Properties
In the list below required Properties are shown with an asterisk (*). Other properties are considered optional. The table also indicates any default values, and whether a property supports the NiFi Expression Language.
| Display Name | API Name | Default Value | Allowable Values | Description |
|---|---|---|---|---|
| Record Reader * | Record Reader | Controller Service: RecordReaderFactory Implementations: AvroReader CEFReader CSVReader ExcelReader GrokReader JsonPathReader JsonTreeReader ReaderLookup ScriptedReader Syslog5424Reader SyslogReader WindowsEventLogReader XMLReader YamlTreeReader | The Record Reader to use for reading the FlowFile. | |
| Record Writer * | Record Writer | Controller Service: RecordSetWriterFactory Implementations: AvroRecordSetWriter CSVRecordSetWriter FreeFormTextRecordSetWriter JsonRecordSetWriter RecordSetWriterLookup ScriptedRecordSetWriter XMLRecordSetWriter | The Record Writer to use for writing the results. | |
| Question Record Path * | Question Record Path | The RecordPath to the question field in the record. Supports Expression Language, using FlowFile attributes and Environment variables. | ||
| Context Record Path * | Context Record Path | The RecordPath to the array of contexts in the record. Supports Expression Language, using FlowFile attributes and Environment variables. | ||
| Context Identifier Record Path * | Context Identifier Record Path | The RecordPath to the array of contexts IDs in the record. Supports Expression Language, using FlowFile attributes and Environment variables. | ||
| Ground Truth Record Path * | Ground Truth Record Path | The RecordPath to the ground truth field in the record. Supports Expression Language, using FlowFile attributes and Environment variables. | ||
| Evaluation Results Record Path * | Evaluation Results Record Path | The RecordPath to write the results of the evaluation to. Supports Expression Language, using FlowFile attributes and Environment variables. | ||
| LLM Provider Service * | LLM Provider Service | Controller Service: LLMService Implementations: StandardAnthropicLLMService StandardOpenAILLMService | The provider service for sending evaluation prompts to LLM |
Dynamic Properties
This component does not support dynamic properties.
Relationships
| Name | Description |
|---|---|
| failure | FlowFiles that cannot be processed are routed to this relationship |
| success | FlowFiles that are successfully processed are routed to this relationship |
Reads Attributes
This processor does not read attributes.
Writes Attributes
| Name | Description |
|---|---|
| fscore.at.n | The average F-Score at N over all queries. |
| json.parse.failures | Number of JSON parse failures encountered. |
| mrr | The Mean Reciprocal Rank. |
| n | The average number of retrieved documents per query. |
| precision.at.n | The average precision at N over all queries. |
| recall.at.n | The average recall at N over all queries. |
| retrieval.eval.failures | Number of records where the eval could not be calculated. |
State Management
This component does not store state.
Restricted
This component is not restricted.
Input Requirement
This component requires an incoming relationship.
System Resource Considerations
This component does not specify system resource considerations.