Is there a way to create index with sparse vectors and insert sparse vectors

From Elastic Search to Elasticsearch

Removed docker

Welcome!

Yes. See Sparse vector field type | Elasticsearch Guide [8.15] | Elastic

I checked the documentation you specified but I am using elastic using docker and I dont want use the ELSER model without that can I index sparse vectors as I tried to mention field type : "sparse_vectors" then I got the error that sparse_vectors are not supported

Which version of Elasticsearch are you using? Vector search is an area that has been and is moving quickly, so I would recommend you ensure you are on the latest released version.

18.5.1

Is that 8.5.1 or 8.15.1? If it is 8.5.1 you should upgarde to 8.15.1.

Sorry I wrote it wrong in a hurry it is already 8.15.1

Did you try the example?

```
PUT my-index-000001
{
"mappings": {
"properties": {
"text": {
"type": "text",
"analyzer": "standard"
},
"impact": {
"type": "sparse_vector"
},
"positive": {
"type": "sparse_vector"
},
"negative": {
"type": "sparse_vector"
}
}
}
}
POST my-index-000001/_doc
{
"text": "I had some terribly delicious carrots.",
"impact": [{"I": 0.55, "had": 0.4, "some": 0.28, "terribly": 0.01, "delicious": 1.2, "carrots": 0.8},
{"I": 0.54, "had": 0.4, "some": 0.28, "terribly": 2.01, "delicious": 0.02, "carrots": 0.4}],
"positive": {"I": 0.55, "had": 0.4, "some": 0.28, "terribly": 0.01, "delicious": 1.2, "carrots": 0.8},
"negative": {"I": 0.54, "had": 0.4, "some": 0.28, "terribly": 2.01, "delicious": 0.02, "carrots": 0.4}
}
GET my-index-000001/_search
{
"query": {
"term": {
"impact": {
"value": "delicious"
}
}
}
}
```

There's no ELSER involved here.

I tried it like this :

```
mappings = {
"properties": {
"title": {
"type": "sparse_vector",
},
"content": {
"type": "dense_vector",
"dims": 384
}
}
}
# Create index with mapping
es_client.indices.create(index=index_name, mappings = mappings)
```

On executing this I got the error that sparse_vector is not supported

I just tested the following on a new 8.16.0-SNAPSHOT cluster and it's all good:

```
PUT my-index-000001
{
"mappings": {
"properties": {
"text": {
"type": "text",
"analyzer": "standard"
},
"impact": {
"type": "sparse_vector"
},
"positive": {
"type": "sparse_vector"
},
"negative": {
"type": "sparse_vector"
}
}
}
}
```

So please first, test this in Kibana so we can know where to search.

Thanks

I will try in Kibana and let you know. I tried implementing it using Elasticsearch Python client like this :

```
from elasticsearch import Elasticsearch
index_name = 'temp1'
# Define mappings for dense and sparse vectors
mappings = {
"properties": {
"text": {
"type": "text",
"analyzer": "standard"
},
"impact": {
"type": "sparse_vector"
},
"positive": {
"type": "sparse_vector"
},
"negative": {
"type": "sparse_vector"
}
}
}
# Create index with mapping
es_client.indices.create(index=index_name, mappings = mappings)
```

And it gave the following error :

BadRequestError: BadRequestError(400, 'mapper_parsing_exception', 'Failed to parse mapping: The [sparse_vector] field type is no longer supported.')

What is the version of the Python client?

Version: 8.15.1

I ran that on my side and got:

```
ObjectApiResponse({'acknowledged': True, 'shards_acknowledged': True, 'index': 'temp1'})
```

So I think there's something wrong in your code somewhere... Or may be you don't hit the right endpoint.

What is the output of:

```
es_client.info()
```

```
ObjectApiResponse({
'name': 'bfd82a576095',
'cluster_name': 'docker-cluster',
'version': {
'number': '8.8.0',
'build_flavor': 'default',
'build_date': '2023-05-23T17:16:07.179039820Z',
'lucene_version': '9.6.0',
'minimum_wire_compatibility_version': '7.17.0',
'minimum_index_compatibility_version': '7.0.0'
},
'tagline': 'You Know, for Search'
})
```

It's now obvious that you are not using Elasticsearch 8.15.1 but 8.8.0.

I checked that 8.8.0 has support for sparse vectors

I checked that 8.8.0 has support for sparse vectors

Where? Because if you read the documentation I linked (Sparse vector field type | Elasticsearch Guide [8.15] | Elastic), you will read:

`sparse_vector`

fields can not be included in indices that werecreatedon Elasticsearch versions between 8.0 and 8.10

`sparse_vector`

has been introduced in 8.11 AFAIK.