GitLab now enforces expiry dates on tokens that originally had no set expiration date. Those tokens were given an expiration date of one year later. Please review your personal access tokens, project access tokens, and group access tokens to ensure you are aware of upcoming expirations. Administrators of GitLab can find more information on how to identify and mitigate interruption in our documentation.

Dung Phan requested to merge feature/indexing-and-sparse-embeddings into develop Apr 25, 2025

Indexing and Cloud-based Sparse Embedding

Changes Overview

This PR introduces the following improvements and adjustments related to data indexing and embedding handling:

Add support for indexing data directly to cloud
Remove local sparse embedding logic

Detailed Changes

1. Cloud Integration

Implemented functionality to index data directly into vector database
Established connection handling and necessary API calls for uploading vectors

2. Remove Local Sparse Embedding

Deleted local sparse embedding logic and related files/modules
Transitioned embedding responsibilities fully to cloud services

Testing

Verified successful data indexing to cloud
Tested cloud embedding integration with sample data
Ran full pipeline to validate end-to-end indexing

Cleanup Verification

Ensured all removed embedding code is unused and unreferenced
Verified new cloud indexing logic works as intended
Checked logs and cloud responses for consistency and errors