Tasks

Overview

Tasks are the smallest unit of work that can be executed by the platform. Tasks are grouped into flows. They can be either basic or composite.

Every task runs with the following:

• input: a set of input parameters such as a text for llm completion, the llm model parameters, etc.
• task_config: configuration for the run, such as the task name, retries and caching strategies, etc.
• context: environment variables, previous task outputs

and produces:

• output: the output of the task

Core Task Categories

Content Loaders

Load data into Document objects. Examples include:

• WebLoader: load a document from a url
• RemoteFilesLoader: load a document from uploaded files
• CombinedLoader: load a document from a directory
• More to be added

Text Splitters

Split texts into chunks that meet token limits of LLMs. Examples include:

• CharacterTextSplitter: split text based on separator characters
• RecursiveTextSplitter: split text recursively based on a set of separators
• TokenSplitter: split text based on a set token size per chunk
• More to be added

Embedding

Convert chunks into embeddings. Examples include:

• EmbeddingsModel: Use OpenAI embedding models to convert chunks into embeddings
• More to be added

Vector Store Builder

Store embeddings in a vector store. Examples include:

• QdrantBuilder: Use Qdrant to store embeddings in a local instance
• More to be added

Retriever

Retrieve documents from a vector store given an input vector. Examples include:

• QdrantRetriever: Use Qdrant to retrieve documents
• More to be added

LLM

Use an LLM for text-based completion such as generating answer, summarizing, classification. Examples include:

• LLM: An LLM with non-chat-based completion
• ChatLLM: An LLM with chat-based completion
• SummarizeLLM: An LLM for summarization