it paragraphs, or to separate them into two collections. If we choose to store redundantly, the main text will be stored redundantly as many times as the number of split paragraphs, making the size of the collection inefficiently large. Therefore, in the previous work, we separated the collections. When collections are separated, it looks like the following. One collection (`demo.law`) stores the main text and other supplementary information.

And another collection (`demo.law.content`) stores the split paragraphs of the main text.

However, although this method is more efficient than redundant storage, it increases complexity because we have to manage separate storage and join two collections during queries. Also, there are more considerations, such as whether to prioritize the score of the retrieved paragraph or other fields, whether to merge the scores of both, and where to set the criterion for the number of documents. Unfortunately, this method is common when using storage that supports vector search. This problem arises because the storage that indexes/searches cannot fully keep up with the limitations and characteristics of the embedding models. Here, we think about ways to store vector data in a structured manner, like the [nested fields supported by Cognica](https://docs.cognica.io/user-guide/documentdb/collection/index-fts#nested-fields). # Supporting Two-Dimensional Vectors To address this, we modified the code that stores, indexes, and searches vector data so that **a field in a document can store one or more vector data, that is, store two-dimensional vectors**. We will explain the detailed implementation in the next article. Now, we no longer need to separate two or more collections to consider the limitations of embedding models. We store a single collection and store the embedding data of the split paragraphs (`passage_embed`) as a two-dimensional array. demo.law.v2

As a result, the commands to merge the two collections are also removed, and the query has become simpler. route

## Execution Time and Data Size Tests were conducted on a MacBook Pro M3 Max with 36GB of RAM, and the execution times are as follows. The execution time was not significantly different from storing in separate collections. - Data Insertion: 14m 24.3s - Indexing: 33m 44s Data storage slightly reduced the size of the Full-Text Search index as the text data of the split paragraphs was excluded.

	Existing	Two-Dimensional Vector
Original Data	3.2 GiB (965.2 MiB, Paragraphs: 2.3 GiB)	3 GiB
Index	11.7 GiB (3.7 GiB, Paragraphs: 8 GiB)	9.8 GiB

# Conclusion We looked into what constraints vector embedding models have and what problems can occur in the storage and indexing process when splitting and storing long documents considering these constraints. This is something that can be easily encountered in the field, and we can reduce complexity through Cognica. # References - [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://arxiv.org/pdf/1908.10084) # Getting to Know Cognica If you'd like to explore Cognica further, you can easily [install it with Docker](https://docs.cognica.io/getting-started/q

Cognica Blog

Why Did We Store Two-Dimensional Vectors for Vector Search?

Read Post