EvaluateRagFaithfulness
Description
Evaluates the faithfulness of generated answers in a Retrieval-Augmented Generation (RAG) system by analyzing responses using an LLM (e.g., OpenAI's GPT). The processor enriches each FlowFile record with faithfulness metrics and detailed analysis.
Tags
ai, evaluation, faithfulness, llm, nlp, openai, rag
Properties
In the list below required Properties are shown with an asterisk (*). Other properties are considered optional. The table also indicates any default values, and whether a property supports the NiFi Expression Language.
| Display Name | API Name | Default Value | Allowable Values | Description |
|---|---|---|---|---|
| Record Reader * | Record Reader | Controller Service: RecordReaderFactory Implementations: AvroReader CEFReader CSVReader ExcelReader GrokReader JsonPathReader JsonTreeReader ReaderLookup ScriptedReader Syslog5424Reader SyslogReader WindowsEventLogReader XMLReader YamlTreeReader | The Record Reader to use for reading the FlowFile. | |
| Record Writer * | Record Writer | Controller Service: RecordSetWriterFactory Implementations: AvroRecordSetWriter CSVRecordSetWriter FreeFormTextRecordSetWriter JsonRecordSetWriter RecordSetWriterLookup ScriptedRecordSetWriter XMLRecordSetWriter | The Record Writer to use for writing the results. | |
| Question Record Path * | Question Record Path | The RecordPath to the question field in the record. Supports Expression Language, using FlowFile attributes and Environment variables. | ||
| Context Record Path * | Context Record Path | The RecordPath to the array of contexts in the record. Supports Expression Language, using FlowFile attributes and Environment variables. | ||
| Context Identifier Record Path * | Context Identifier Record Path | The RecordPath to the array of contexts IDs in the record. Supports Expression Language, using FlowFile attributes and Environment variables. | ||
| Generated Answer Record Path * | Generated Answer Record Path | /generated_answer | The path to the answer field in the record Supports Expression Language, using FlowFile attributes and Environment variables. | |
| Evaluation Results Record Path * | Evaluation Results Record Path | The RecordPath to write the results of the evaluation to. Supports Expression Language, using FlowFile attributes and Environment variables. | ||
| LLM Provider Service * | LLM Provider Service | Controller Service: LLMService Implementations: StandardAnthropicLLMService StandardOpenAILLMService | The provider service for sending evaluation prompts to LLM |
Dynamic Properties
This component does not support dynamic properties.
Relationships
| Name | Description |
|---|---|
| failure | FlowFiles that cannot be processed are routed to this relationship |
| success | FlowFiles that are successfully processed are routed to this relationship |
Reads Attributes
This processor does not read attributes.
Writes Attributes
| Name | Description |
|---|---|
| average.answer.faithfulness | The average faithfulness score computed over all records. |
| json.parse.failures | Number of JSON parse failures encountered. |
State Management
This component does not store state.
Restricted
This component is not restricted.
Input Requirement
This component requires an incoming relationship.
Example Use Cases
Use Case 1
Use this processor to assess the faithfulness of answers generated by an LLM compared to the provided context. It provides metrics that can be used for monitoring and improving the performance of RAG systems.
Configuration
Set the Record Reader and Record Writer to read and write records in the desired format.
Specify the Record Paths for the question, context, context IDs, answer, and where to store the evaluation results.
The processor will read each record, evaluate the faithfulness of the answer against the context, and write the enriched records to the output.
System Resource Considerations
This component does not specify system resource considerations.