Global Files

Global Files API

Manage global files in the sapienstream-global bucket. Supports component files (GSDML, EDS, datasheets), Nelo knowledge base documents, and miscellaneous files. Includes deduplication, vector indexing, and Nelo AI processing.

12 EndpointsComponent FilesNelo KnowledgeDeduplication

Overview

File Categories

• component_file: Device descriptions, datasheets
• nelo_knowledge: AI training documents
• miscellaneous: General purpose files

Features

• SHA256 content deduplication
• Vector indexing for Nelo AI
• Public URLs for component files
• Bulk upload for archives

File Categories & Limits

Component Files

Device descriptions

Max Size: 50MB

Public: Yes

Extensions: .xml, .gsdml, .csp, .eds, .gds, .edsx, .dcf, .xdd, .xdc, .pdf, etc.

Nelo Knowledge

AI training data

Max Size: 100MB

Public: No

Extensions: .pdf, .md, .txt, .docx, .json, .csv

Miscellaneous

General files

Max Size: 200MB

Public: No

Extensions: All allowed

File Upload

Admin Only: General upload endpoint requires system administrator privileges.

POST/v1/global-files/uploadAuth Required

Upload Global File (Admin)

Upload a single file to global storage. Automatically detects content duplicates by SHA256 hash and reuses existing vectors. Admin only.

Request Body

Requestjson

// multipart/form-data
file: [binary data]                    // Required: File to upload
category: "component_file"             // Required: component_file, nelo_knowledge, or miscellaneous
subcategory: "datasheet"               // Optional: Subcategory within category
component_id: "comp_abc123"            // Required for component_file: Links to component
relative_path: "vendor/model"          // Optional: Folder path within category
description: "Component datasheet"     // Optional: File description
tags: "siemens,sensor,temperature"     // Optional: Comma-separated tags

Response

Responsejson

{
  "id": "gfile_abc123",
  "category": "component_file",
  "subcategory": "datasheet",
  "bucket_name": "sapienstream-global",
  "object_key": "component-files/comp_abc123/datasheet.pdf",
  "filename": "TE01_datasheet.pdf",
  "mime_type": "application/pdf",
  "file_size": 1234567,
  "sha256_hash": "a7ffc6f8bf1ed76651c14756a061d662...",
  "public_url": "https://minio.sapienstream.com/sapienstream-global/component-files/comp_abc123/datasheet.pdf",
  "component_id": "comp_abc123",
  "description": "Component datasheet",
  "tags": ["siemens", "sensor", "temperature"],
  "is_content_duplicate": false,
  "vector_indexed": false,
  "available_to_nelo": false,
  "created_at": "2024-08-26T15:30:00Z",
  "uploaded_by": {
    "id": "user_001",
    "email": "[email protected]"
  }
}

Try it out

cURLbash

curl -X POST "https://sapienstream.com/api/v1/global-files/upload" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json"

POST/v1/global-files/upload-component-fileAuth Required

Upload Component Datasheet (User)

Upload a component datasheet PDF file. Available to any authenticated user. Returns file ID to use in component creation.

Request Body

Requestjson

// multipart/form-data
file: [binary data]                    // Required: PDF file to upload
subcategory: "datasheet"               // Optional: Type (datasheet, description, manual)
description: "Temperature sensor specs" // Optional: File description

Response

Responsejson

{
  "id": "gfile_def456",
  "category": "component_file",
  "subcategory": "datasheet",
  "bucket_name": "sapienstream-global",
  "object_key": "component-files/unassigned/datasheet_abc123.pdf",
  "filename": "TE01_datasheet.pdf",
  "mime_type": "application/pdf",
  "file_size": 1234567,
  "sha256_hash": "a7ffc6f8bf1ed76651c14756a061d662...",
  "component_id": null,
  "description": "Temperature sensor specs",
  "created_at": "2024-08-26T15:30:00Z"
}

Try it out

cURLbash

curl -X POST "https://sapienstream.com/api/v1/global-files/upload-component-file" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json"

POST/v1/global-files/bulk-uploadAuth Required

Bulk Upload (Archive)

Upload a ZIP or TAR archive containing multiple files. Extracts and processes each file individually. Admin only.

Request Body

Requestjson

// multipart/form-data
file: [binary data]                    // Required: ZIP or TAR archive
category: "component_file"             // Required: Category for all files
subcategory: "vendor-files"            // Optional: Subcategory
component_id: "comp_abc123"            // Optional: Link all to component
description: "Vendor files package"    // Optional: Description for all
tags: "vendor,import"                  // Optional: Tags for all files

Response

Responsejson

{
  "message": "Bulk upload completed",
  "archive_type": "zip",
  "files_processed": 15,
  "files_succeeded": 14,
  "files_failed": 1,
  "duplicates_found": 3,
  "results": [
    {
      "filename": "device1.gsdml",
      "status": "uploaded",
      "file_id": "gfile_001"
    },
    {
      "filename": "device2.gsdml",
      "status": "duplicate",
      "source_id": "gfile_existing"
    },
    {
      "filename": "invalid.exe",
      "status": "failed",
      "error": "File extension .exe not allowed"
    }
  ]
}

Try it out

cURLbash

curl -X POST "https://sapienstream.com/api/v1/global-files/bulk-upload" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json"

File Listing

GET/v1/global-filesAuth Required

List Global Files

List global files with filtering and pagination. Filter by category, component, search term, etc.

Parameters

categorystring

Filter by category (component_file, nelo_knowledge, miscellaneous)

component_idstring

Filter by linked component

searchstring

Search in filename and description

vector_indexedboolean

Filter by indexing status

pageinteger

Page number (default: 1)

per_pageinteger

Items per page (default: 20)

Response

Responsejson

{
  "files": [
    {
      "id": "gfile_abc123",
      "category": "component_file",
      "filename": "TE01_datasheet.pdf",
      "file_size": 1234567,
      "mime_type": "application/pdf",
      "component_id": "comp_abc123",
      "vector_indexed": true,
      "available_to_nelo": true,
      "created_at": "2024-08-26T15:30:00Z"
    }
  ],
  "total": 125,
  "page": 1,
  "per_page": 20,
  "pages": 7
}

Try it out

cURLbash

curl -X GET "https://sapienstream.com/api/v1/global-files" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json"

GET/v1/global-files/components-with-filesAuth Required

List Components with Files

List all components that have associated global files. Useful for browsing file library by component.

Response

Responsejson

{
  "components": [
    {
      "id": "comp_abc123",
      "name": "Temperature Sensor TE01",
      "identifier": "6ES7511-1AK02-0AB0",
      "file_count": 3,
      "files": [
        {
          "id": "gfile_001",
          "filename": "datasheet.pdf",
          "subcategory": "datasheet"
        },
        {
          "id": "gfile_002",
          "filename": "description.gsdml",
          "subcategory": "description"
        }
      ]
    }
  ],
  "total_components": 45
}

Try it out

cURLbash

curl -X GET "https://sapienstream.com/api/v1/global-files/components-with-files" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json"

GET/v1/global-files/component/{component_id}Auth Required

Get Component Files

Get all files associated with a specific component.

Parameters

component_idstringRequired

Component ID

Response

Responsejson

{
  "component_id": "comp_abc123",
  "files": [
    {
      "id": "gfile_001",
      "filename": "datasheet.pdf",
      "subcategory": "datasheet",
      "file_size": 1234567,
      "vector_indexed": true,
      "public_url": "https://..."
    },
    {
      "id": "gfile_002",
      "filename": "description.gsdml",
      "subcategory": "description",
      "file_size": 45678,
      "vector_indexed": false
    }
  ],
  "total": 2
}

Try it out

cURLbash

curl -X GET "https://sapienstream.com/api/v1/global-files/component/{component_id}" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json"

File Operations

GET/v1/global-files/{file_id}Auth Required

Get File Metadata

Get detailed metadata for a specific global file.

Parameters

file_idstringRequired

Global file ID

Response

Responsejson

{
  "id": "gfile_abc123",
  "category": "component_file",
  "subcategory": "datasheet",
  "bucket_name": "sapienstream-global",
  "object_key": "component-files/comp_abc123/datasheet.pdf",
  "filename": "TE01_datasheet.pdf",
  "mime_type": "application/pdf",
  "file_size": 1234567,
  "sha256_hash": "a7ffc6f8bf1ed76651c14756a061d662...",
  "public_url": "https://minio.sapienstream.com/...",
  "component_id": "comp_abc123",
  "description": "Temperature sensor datasheet",
  "tags": ["siemens", "sensor"],
  "is_content_duplicate": false,
  "content_hash_source_id": null,
  "duplicate_count": 2,
  "vector_indexed": true,
  "available_to_nelo": true,
  "collection_name": "global_files",
  "chunk_count": 45,
  "indexed_at": "2024-08-26T16:00:00Z",
  "created_at": "2024-08-26T15:30:00Z",
  "updated_at": "2024-08-26T16:00:00Z",
  "uploaded_by": {
    "id": "user_001",
    "email": "[email protected]"
  }
}

Try it out

cURLbash

curl -X GET "https://sapienstream.com/api/v1/global-files/{file_id}" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json"

GET/v1/global-files/{file_id}/downloadAuth Required

Download File

Download the file content. Redirects to a presigned URL for the actual file.

Parameters

file_idstringRequired

Global file ID

Response

Responsejson

// 302 Redirect to presigned URL
Location: https://minio.sapienstream.com/sapienstream-global/component-files/comp_abc123/datasheet.pdf?X-Amz-Signature=...

Try it out

cURLbash

curl -X GET "https://sapienstream.com/api/v1/global-files/{file_id}/download" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json"

GET/v1/global-files/{file_id}/view

View File (Inline)

View the file inline in the browser. For PDFs and images, displays directly. For other types, triggers download.

Parameters

file_idstringRequired

Global file ID

Response

Responsejson

// Streaming response with inline Content-Disposition
Content-Type: application/pdf
Content-Disposition: inline; filename="datasheet.pdf"

[Binary file data]

Try it out

cURLbash

curl -X GET "https://sapienstream.com/api/v1/global-files/{file_id}/view" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json"

Nelo AI Processing

POST/v1/global-files/{file_id}/process-neloAuth Required

Process File for Nelo

Trigger vector indexing of a file for Nelo AI. Chunks the document and creates embeddings for semantic search. Returns immediately, processing happens asynchronously.

Parameters

file_idstringRequired

Global file ID to process

Response

Responsejson

{
  "message": "File processing started",
  "file_id": "gfile_abc123",
  "filename": "datasheet.pdf",
  "status": "processing",
  "task_id": "task_xyz789",
  "estimated_chunks": 45
}

Try it out

cURLbash

curl -X POST "https://sapienstream.com/api/v1/global-files/{file_id}/process-nelo" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json"

File Deletion

DELETE/v1/global-files/{file_id}Auth Required

Delete File (Soft Delete)

Soft delete a global file. The file is marked as deleted but not removed from storage.

Parameters

file_idstringRequired

Global file ID

Response

Responsejson

{
  "message": "File deleted successfully",
  "file_id": "gfile_abc123",
  "filename": "datasheet.pdf"
}

Try it out

cURLbash

curl -X DELETE "https://sapienstream.com/api/v1/global-files/{file_id}" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json"

DELETE/v1/global-files/components/{component_id}/permanentAuth Required

Permanently Delete Component Files

Permanently delete all files associated with a component from both database and MinIO storage. Admin only. Use with caution.

Parameters

component_idstringRequired

Component ID

Response

Responsejson

{
  "message": "All files for component permanently deleted",
  "component_id": "comp_abc123",
  "files_deleted": 3,
  "storage_freed_bytes": 3456789
}

Try it out

cURLbash

curl -X DELETE "https://sapienstream.com/api/v1/global-files/components/{component_id}/permanent" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json"

Upload and Process Example

// 1. Upload component datasheet (any authenticated user)
const formData = new FormData();
formData.append('file', datasheetFile);
formData.append('subcategory', 'datasheet');
formData.append('description', 'Siemens S7-1500 CPU datasheet');

const uploadResponse = await fetch('/v1/global-files/upload-component-file', {
  method: 'POST',
  headers: {
    'Authorization': 'Bearer ' + token
  },
  body: formData
});

const { id: fileId } = await uploadResponse.json();
console.log('File uploaded:', fileId);

// 2. Process for Nelo AI (make searchable)
const processResponse = await fetch(`/v1/global-files/${fileId}/process-nelo`, {
  method: 'POST',
  headers: {
    'Authorization': 'Bearer ' + token
  }
});

const { task_id } = await processResponse.json();
console.log('Processing started:', task_id);

// 3. Check file status
const statusResponse = await fetch(`/v1/global-files/${fileId}`, {
  headers: {
    'Authorization': 'Bearer ' + token
  }
});

const fileInfo = await statusResponse.json();
console.log('Vector indexed:', fileInfo.vector_indexed);
console.log('Available to Nelo:', fileInfo.available_to_nelo);

Bulk Upload Example

// Upload ZIP archive of component files (admin only)
const zipFormData = new FormData();
zipFormData.append('file', zipArchive);
zipFormData.append('category', 'component_file');
zipFormData.append('subcategory', 'vendor-package');
zipFormData.append('component_id', 'comp_abc123');
zipFormData.append('tags', 'siemens,import,2024');

const bulkResponse = await fetch('/v1/global-files/bulk-upload', {
  method: 'POST',
  headers: {
    'Authorization': 'Bearer ' + adminToken
  },
  body: zipFormData
});

const result = await bulkResponse.json();
console.log('Files processed:', result.files_processed);
console.log('Succeeded:', result.files_succeeded);
console.log('Duplicates:', result.duplicates_found);

// Check individual results
result.results.forEach(r => {
  if (r.status === 'failed') {
    console.error(`Failed: ${r.filename} - ${r.error}`);
  } else if (r.status === 'duplicate') {
    console.log(`Duplicate: ${r.filename} (source: ${r.source_id})`);
  } else {
    console.log(`Uploaded: ${r.filename} (id: ${r.file_id})`);
  }
});