Skip to content

Upload files to LLM Stack

Motivation

For some tasks such as RemoteFilesLoader, you will need to upload files to LLM Stack. This guide will show you how to do it.

Using Files with LLM Stack

1. Get a Pre-signed S3 URL via LLM Stack

The Upload File API is meant for this.

Get a Pre-signed S3 URL 
# Pre-requisite: Set LLM_STACK_API_KEY environment variable to your API key
curl --location 'https://api.stack.govtext.gov.sg/v1/flows/upload-file-url?flow_id=My_Flow_ID&filename=myfile.ext' \
  --header "Authorization: Bearer ${LLM_STACK_API_KEY}" \

# Pre-requisite: Set LLM_STACK_API_KEY environment variable to your API key
import requests
import os

flow_id = "My_Flow_ID"
filename = "myfile.ext"

url = f"https://api.stack.govtext.gov.sg/v1/flows/upload-file-url?flow_id={flow_id}&filename={filename}"
LLM_STACK_API_KEY = os.environ["LLM_STACK_API_KEY"]

headers = {
    'Authorization': f'Bearer {LLM_STACK_API_KEY}'
}

response = requests.request("GET", url, headers=headers)

print(response.text)

// Pre-requisite: Set LLM_STACK_API_KEY environment variable to your API key

var flow_id = "My_Flow_ID";
var filename = "myfile.ext";
var url = "https://api.stack.govtext.gov.sg/v1/flows/upload-file-url?flow_id=" + flow_id + "&filename=" + filename;
var myHeaders = new Headers();
myHeaders.append("Authorization", "Bearer " + process.env.LLM_STACK_API_KEY);

var requestOptions = {
  method: 'GET',
  headers: myHeaders,
  redirect: 'follow'
};

fetch(url, requestOptions)
  .then(response => response.text())
  .then(result => console.log(result))
  .catch(error => console.log('error', error));

You'll get a response in this format, take note of url, s3_object_url and fields: 

{
  "status": "string",
  "data": {
    "url": "https://s3.ap-southeast-1.amazonaws.com/path",
    "s3_object_url": "s3://bucket/path",
    "fields": {
      "key": "string",
      "AWSAccessKeyId": "string",
      "policy": "string",
      "signature": "string"
    }
  }
}

2. Upload file to S3 using the pre-signed URL

You can use any of the following methods to upload the file to S3. Note that the url and fields returned in the previous step is used for the following methods.

Upload file to S3 
curl --location <url> \
    --form 'file=@"/path/to/myfile.ext"' \
    --form 'key="<key>"' \
    --form 'AWSAccessKeyId="<AWSAccessKeyId>"' \
    --form 'policy="<policy>"' \
    --form 'signature="<signature>"'

import requests
import os

url = "<url>"

files=[
    ('file',('myfile.ext', open('/path/to/myfile.ext','rb'), 'text/csv'))
]

payload = {
    'key': '<key>',
    'AWSAccessKeyId': '<AWSAccessKeyId>',
    'policy': '<policy>',
    'signature': '<signature>'
}

response = requests.request("POST", url, data=payload, files=files)

print(response.text)

// Pre-requisite: Set LLM_STACK_API_KEY environment variable to your API key

var flow_id = "My_Flow_ID";
var filename = "myfile.ext";

var url = "https://api.stack.govtext.gov.sg/v1/flows/upload-file-url?flow_id=" + flow_id + "&filename=" + filename;
var myHeaders = new Headers();
myHeaders.append("Authorization", "Bearer " + process.env.LLM_STACK_API_KEY);

var formdata = new FormData();
formdata.append("file", myfile, "myfile.ext");
formdata.append("key", "<key>");
formdata.append("AWSAccessKeyId", "<AWSAccessKeyId>");
formdata.append("policy", "<policy>");
formdata.append("signature", "<signature>");

var requestOptions = {
    method: 'POST',
    body: formdata,
    redirect: 'follow'
};

fetch(url, requestOptions)
  .then(response => response.text())
  .then(result => console.log(result))
  .catch(error => console.log('error', error));

3. Get a Pre-signed S3 URL via LLM Stack

Use the s3_object_url as input for a Flow using RemoteFilesLoader.

{
  "flow_id": "My_Flow_Using_RemoteFilesLoader",
  "inputs": [{
    "file_urls": ["<s3_object_url>"],
    "other_parameters": "..."
  }]
}