CreateVertexAIEmbeddings
Description
Uses VertexAI to create embeddings for text. The input text can be provided as a single FlowFile or as a record-oriented FlowFile.
Tags
chatbot, cloud, embeddings, gcp, gen ai, generative ai, google, llm, nlp, text, vertex
Properties
In the list below required Properties are shown with an asterisk (*). Other properties are considered optional. The table also indicates any default values, and whether a property supports the NiFi Expression Language.
| Display Name | API Name | Default Value | Allowable Values | Description |
|---|---|---|---|---|
| GCP Credentials Service * | GCP Credentials Service | Controller Service: GCPCredentialsService Implementations: GCPCredentialsControllerService | The Controller Service used to obtain Google Cloud Platform credentials. | |
| GCP Location * | GCP Location | The location to configure the Vertex client with Supports Expression Language, using Environment variables. | ||
| GCP Project ID * | GCP Project ID | The project ID to configure the Vertex client with Supports Expression Language, using Environment variables. | ||
| Model Publisher * | Model Publisher | The publisher of the model Supports Expression Language, using FlowFile attributes and Environment variables. | ||
| Embeddings Model * | Embeddings Model | text-embedding-004 | The model to use for embeddings, available models are listed at https://cloud.google.com/vertex-ai/generative-ai/docs/learn/models#models Supports Expression Language, using FlowFile attributes and Environment variables. | |
| Task Type * | Task Type | RETRIEVAL_QUERY |
| Used to convey intended downstream application of embeddings to help the model tune embeddings for a specific purpose. |
| Auto Truncate * | Auto Truncate | true |
| If set to false, text that exceeds the token limit causes the request to fail. |
| Output Dimensionality | Output Dimensionality | Used to specify output embedding size. If set, output embeddings will be truncated to the size specified. | ||
| User | User | An identifier for the remote user on whose behalf the request is being made. Supports Expression Language, using FlowFile attributes and Environment variables. | ||
| Record Reader | Record Reader | Controller Service: RecordReaderFactory Implementations: AvroReader CEFReader CSVReader ExcelReader GrokReader JsonPathReader JsonTreeReader ReaderLookup ScriptedReader Syslog5424Reader SyslogReader WindowsEventLogReader XMLReader YamlTreeReader | The record reader to use for reading record-oriented data. If the incoming data is to be treated as plaintext, this property should be left unset. | |
| Record Writer * | Record Writer | Controller Service: RecordSetWriterFactory Implementations: AvroRecordSetWriter CSVRecordSetWriter FreeFormTextRecordSetWriter JsonRecordSetWriter RecordSetWriterLookup ScriptedRecordSetWriter XMLRecordSetWriter | The Record Writer to use for writing the output | |
| Text Record Path * | Text Record Path | /text | The path to the field in the record that contains the text to be embedded. If the incoming data is to be treated as plaintext, this property should be left unset. Supports Expression Language, using FlowFile attributes and Environment variables. This property is only considered if:
| |
| Embeddings Record Path * | Embeddings Record Path | /embeddings | The path to the field in the record where the embeddings are to be written. Supports Expression Language, using FlowFile attributes and Environment variables. This property is only considered if:
| |
| Max Batch Size * | Max Batch Size | 100 | The maximum number of records to include in each batch sent to VertexAI |
Dynamic Properties
This component does not support dynamic properties.
Relationships
| Name | Description |
|---|---|
| failure | The original FlowFile will be routed to this relationship if the embeddings could not be created |
| success | The embeddings will be routed to this relationship |
Reads Attributes
This processor does not read attributes.
Writes Attributes
| Name | Description |
|---|---|
| mime.type | The MIME type of the output data, based on the chosen Record Writer |
| record.count | The number of records written to the output |
State Management
This component does not store state.
Restricted
This component is not restricted.
Input Requirement
Input requirements are not specified for this component.
Example Use Cases
Use Case 1
Create embeddings for text using VertexAI's Embedding model
Configuration
Set "GCP Credentials Service" to a GCP service that is authenticated to GCP.
Set "GCP Project ID" to the GCP project with VertexAI API enabled.
Set "GCP Location" to one of the GCP locations enabled for the project ID.
Set "Embeddings Model" to the name of the GCP model to use for creating embeddings.
Set "Input Type" based on the desired use case for the vector embeddings.
Set "Record Writer" to a Record Writer that writes out data in the desired format, typically JSON.
If the incoming data is in a structured format such as JSON, set "Record Reader" to a Record Reader that can read the format of the incoming data and set "Text Record Path" to the path to the field that contains the text to be embedded.
Otherwise, if the incoming data is raw text, leave the "Text Record Path" and "Record Reader" properties unset.
System Resource Considerations
This component does not specify system resource considerations.