How to index only some of the fields of the entire document?

Elastic Search
1

Hi everyone,

I have an indexing question related to EES app searching. Btw we are on the cloud offering v8.4.

We are using the API to push documents into the search engine and we are actually sending to EES the whole document we want to be returned by our custom API. But only a couple of the properties/fields of the indexed document are relevant to our search results.
I know there is an option to specify the fields EES should search into, but that doesn't seem to yield the results we think it should and the scoring is really weird.

My questions:

  • are all the fields being indexed by EES?
  • is there a way to tell EES which fields to index or not to index? because I'm assuming when it does index all of them then the generated score might be incorrect and that could affect the search result

Thanks,
Lucian

2

Hello @lnaie , welcome to the community !
If I understand the requirement correctly, you only want to store some fields from your logs in ES, the best approach for this would be to adjust the logging source (API) which is logging those messages/ audit fields and remove the unwanted fields.

If that's not a viable option, you can update your index template and set "index": false for the fields you don't want to index.

For your questions:

  1. By default, yes all the fields are indexed by ES.
  2. Using "index": false mapping parameter for the fields that should not be indexed or available for search, but are still stored.
3

Thanks for taking the time to answer.

I have to work with the 2nd option. Is there a documentation page or a sample about how to set "index": false ?

4

This setting need to be in the mapping and is generally defined in a template

1 Like
5

@stephenb does this mean index | Elasticsearch Guide [8.6] | Elastic will be deprecated in upcoming releases ?

6

Hi @lnaie and @Ayush_Mathur

Apologies, I was not careful or clear.

There are 2 settings that are similar both are valid but accomplish 2 slightly different things.

  1. "index": false This will still parse the fields and store it in the fields structure but it will not be searchable or aggregatable, but it is retrievable as part of fields which can be faster than _source, but it does take some storage.

  2. "enabled": false This does not attempt to parse nor does it store the field in the fields structure it is left alone and just remains in the _source it is neither searchable, aggregatable and can not be retrieved from the fields

Hope that helps

Example

DELETE discuss-test

PUT discuss-test
{
  "mappings": {
    "properties": {
      "normal_field": {
        "type": "keyword"
      },
      "not_indexed": {
        "type": "keyword",
        "index": false
      },
      "not_enabled": {
        "type": "object",
        "enabled": false
      }
    }
  }
}


POST discuss-test/_doc
{
  "normal_field" : "myindexedkeyword",
  "not_indexed": "notindexedkeyword",
  "not_enabled" : "notenabled"
}


GET discuss-test/_search
{
  "fields": ["*"]
}

# Results
{
  "took": 396,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 1,
      "relation": "eq"
    },
    "max_score": 1,
    "hits": [
      {
        "_index": "discuss-test",
        "_id": "iInN7oUBWYz4cMSr1FCA",
        "_score": 1,
        "_source": {
          "normal_field": "myindexedkeyword",
          "not_indexed": "notindexedkeyword",
          "not_enabled": "notenabled"
        },
        "fields": {
          "normal_field": [
            "myindexedkeyword"
          ],
          "not_indexed": [
            "notindexedkeyword"
          ]
        }
      }
    ]
  }
}
2 Likes
7

Thank you @stephenb for the explanation and example, really improved my knowledge :slight_smile:

8

Hi,

I think that's about ES docs but we are using EES and those docs are here:
App Search Engine API:

App Search Schema API:

App Search Search API:

I did not see anything similar to what you have indicated there and wondered if there is something I'm missing or if it's just not supported.

Thanks

9

ESS and APP Search are related but not the same thing ...

ESS is the Elasticsearch Service which includes Elasticsearch and Elasticsearch App Search which is a Solution that "Sits On Top Of" Elasticsearch.

App Search is built to simplify the App Search experience and so not every feature of the underlying Elasticsearch is supported.

I do not think the same feature is available in App Search.

If you wanted to not index fields you will need to remove them from the incoming / source JSON

(I am double checking this, but pretty sure)

10

I'm aware of the differences but more ES features should be implemented in the EES.
I know that removing the fields is the default solution but it's not always the needed one. There should be a way to tell in the schema what fields should be indexed or not or some variation on this idea.

Thanks

11

Btw, is there a place to request this feature from the EES team?

12

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.