Sparse Vectors

Elastic Stack Elasticsearch
1

Is there a way to create index with sparse vectors and insert sparse vectors

2

From Elastic Search to Elasticsearch

3

Removed docker

4

Welcome!

Yes. See Sparse vector field type | Elasticsearch Guide [8.15] | Elastic

5

I checked the documentation you specified but I am using elastic using docker and I dont want use the ELSER model without that can I index sparse vectors as I tried to mention field type : "sparse_vectors" then I got the error that sparse_vectors are not supported

6

Which version of Elasticsearch are you using? Vector search is an area that has been and is moving quickly, so I would recommend you ensure you are on the latest released version.

7

18.5.1

8

Is that 8.5.1 or 8.15.1? If it is 8.5.1 you should upgarde to 8.15.1.

9

Sorry I wrote it wrong in a hurry it is already 8.15.1

10

Did you try the example?

PUT my-index-000001
{
  "mappings": {
    "properties": {
      "text": {
        "type": "text",
        "analyzer": "standard"
      },
      "impact": {
        "type": "sparse_vector"
      },
      "positive": {
        "type": "sparse_vector"
      },
      "negative": {
        "type": "sparse_vector"
      }
    }
  }
}

POST my-index-000001/_doc
{
    "text": "I had some terribly delicious carrots.",
    "impact": [{"I": 0.55, "had": 0.4, "some": 0.28, "terribly": 0.01, "delicious": 1.2, "carrots": 0.8},
               {"I": 0.54, "had": 0.4, "some": 0.28, "terribly": 2.01, "delicious": 0.02, "carrots": 0.4}],
    "positive": {"I": 0.55, "had": 0.4, "some": 0.28, "terribly": 0.01, "delicious": 1.2, "carrots": 0.8},
    "negative": {"I": 0.54, "had": 0.4, "some": 0.28, "terribly": 2.01, "delicious": 0.02, "carrots": 0.4}
}

GET my-index-000001/_search
{
  "query": {
    "term": {
      "impact": {
         "value": "delicious"
      }
    }
  }
}

There's no ELSER involved here.

1 Like
11

I tried it like this :

mappings = {
        "properties": {
            "title": {
                "type": "sparse_vector",
            },
            "content": {
                "type": "dense_vector",
                "dims": 384
            }
        }
    }

    # Create index with mapping
es_client.indices.create(index=index_name, mappings = mappings)

On executing this I got the error that sparse_vector is not supported

12

I just tested the following on a new 8.16.0-SNAPSHOT cluster and it's all good:

PUT my-index-000001
{
  "mappings": {
    "properties": {
      "text": {
        "type": "text",
        "analyzer": "standard"
      },
      "impact": {
        "type": "sparse_vector"
      },
      "positive": {
        "type": "sparse_vector"
      },
      "negative": {
        "type": "sparse_vector"
      }
    }
  }
}

So please first, test this in Kibana so we can know where to search.

Thanks

13

I will try in Kibana and let you know. I tried implementing it using Elasticsearch Python client like this :

from elasticsearch import Elasticsearch

index_name = 'temp1'
    # Define mappings for dense and sparse vectors
mappings = {
    "properties": {
      "text": {
        "type": "text",
        "analyzer": "standard"
      },
      "impact": {
        "type": "sparse_vector"
      },
      "positive": {
        "type": "sparse_vector"
      },
      "negative": {
        "type": "sparse_vector"
      }
    }
  }

    # Create index with mapping
es_client.indices.create(index=index_name, mappings = mappings)

And it gave the following error :

BadRequestError: BadRequestError(400, 'mapper_parsing_exception', 'Failed to parse mapping: The [sparse_vector] field type is no longer supported.')

14

What is the version of the Python client?

15

Version: 8.15.1

16

I ran that on my side and got:

ObjectApiResponse({'acknowledged': True, 'shards_acknowledged': True, 'index': 'temp1'})

So I think there's something wrong in your code somewhere... Or may be you don't hit the right endpoint.

What is the output of:

es_client.info()
17 
ObjectApiResponse({
    'name': 'bfd82a576095',
    'cluster_name': 'docker-cluster',
    'version': {
        'number': '8.8.0',
        'build_flavor': 'default',
        'build_date': '2023-05-23T17:16:07.179039820Z',
        'lucene_version': '9.6.0',
        'minimum_wire_compatibility_version': '7.17.0',
        'minimum_index_compatibility_version': '7.0.0'
    },
    'tagline': 'You Know, for Search'
})
18

It's now obvious that you are not using Elasticsearch 8.15.1 but 8.8.0.

19

I checked that 8.8.0 has support for sparse vectors

20

I checked that 8.8.0 has support for sparse vectors

Where? Because if you read the documentation I linked (Sparse vector field type | Elasticsearch Guide [8.15] | Elastic), you will read:

sparse_vector fields can not be included in indices that were created on Elasticsearch versions between 8.0 and 8.10

sparse_vector has been introduced in 8.11 AFAIK.

1 Like