feat: rework the reindex feature #31853

This request should actually be named to rework the reindex feature and not to "allow changing the vector dimensions" :)

The reindex doesn't reindex everything. To my knowledge it's limited to only knowledge bases today so what is needed before implementing any collection dropping would be to rework the reindex to reindex absolutely everything.

Secondly

It is valuable that deletions happens KB by KB as RAG can be still used during reindexing (which can take hours or days) and only the current re-indexed KB is affected for a short time. That is already very helpful and keeps operations of RAG up as good as possible.

I am not sure how that would be even possible.

In a non-multitenancy setup you CAN drop collections one by one (where one collection == one file) and thus do partial reindexing so that RAG stays partially usable during reindexing -- BUT only on the files that were already reindexed as the new embedding model with the new dimensions won't be able to match any of the existing, not-yet-reindexed data. If the new embedding model matches the old one in dimensions then it will technically work but you'll get absolutely garbage results from your RAG and RAG definitely won't be usable. So again i am not sure how you envision this to work.

Worse: On multitenancy setups (which you are on) like qdrant- and milvus-multitenancy that would be strictly technically impossible as there is one large collection for all files ever uploaded to a knowledge base for example. So in this scenario you can't even keep the RAG Running in the meantime.

So this is another reason why first the reindex has to be massively refactored.

More importantly though, neither Qdrant (what you use) nor most other vector backends are even officially supported. Open WebUI only officially supports chromadb and pgvector. All other backends are community-added and get no priority support if something breaks there.

Finally a subjective subtle disagreement:

The issue is simple: you start with an embedding with e.g. dimension x and then switch to an embedding with dimension y. That is a real use case as embedding models get better and switching from time to time is normal.

It is not normal at all to change embedding Models just because there's a new model that came out that is slightly better.

Even if that new model seems to be much better in all embedding benchmarks and appears better, in reality you'll not notice much of any IF ANY quality improvement (unless you jump from a sentence transformer L6 to something like gemini embedding 002). But assuming you were already using something half decent then a new model isn't gonna net you any gains in practice.

The typical lifecycles of embedding models are super slow and very long until deprecation because reindexing everything is slow painful and costs money and the providers also know that.

What IS typical is to try to hold the embedding model you chose as long as possible until it gets deprecated and then switch - or if you self host it, just using it as long as possible.

Newer models even if so much better on benchmarks typically don't net a noticable improvement in practice.

And constantly switching embedding models (inferred usecase from how you wrote it: new one comes out so you switch) is in fact extremely atypical. All of the vector dbs are designed to never change vector dimensions. Which is why you have to drop the entire collection (the database) to even be able to finally change it. They were all designed around the principle that under normal circumstances you never change the vector dimensions or the embedding model (or at least not frequently change it).

Now that all of this is said this is a feature request i can understand in principle but this is super low priority as for the complexity of implementing it. Every vector Backend needs different implementation here but even preceding that you'd first need to refactor the reindex feature to even reindex ALL files instead of just files in knowledge bases and only when these two things are done can you even think about also dropping collections and recreating them with the different vector dimension.

If you need this today better write a script to do it or event function. This is very low on the feature request list due to the complex nature of implementing it and being a very nieche request in reality even, if for you it might seem like a sensible thing to do, most don't want to swap embedding models often at all