How can i increase text field limit on aggregations?

Elastic Stack Elasticsearch
1

Hello,

I have a SearchGroupedErrorByCriteria method for take data on Elasticsearch and view it on a admin panel table.

I'm using Elasticsearch .NET Client. I have a search descriptor:

var searchDescriptor = new SearchRequestDescriptor<ServiceLogMonitoringFieldModel>(indexName);

I add a query like this to my search descriptor and it has some aggregations:

  searchDescriptor
                .Size(0)
                .Query(q => q
                .Bool(b => b
                 .Filter(queryDescriptorsFilter.Any() ? queryDescriptorsFilter.ToArray() : null)
                 .Must(queryDescriptorsMust.Any() ? queryDescriptorsMust.ToArray() : null)
                 .MustNot(queryDescriptorsMustNot.Any() ? queryDescriptorsMustNot.ToArray() : null)
                 ))
                .Sort(sortOptionsDescriptor.Any() ? sortOptionsDescriptor.ToArray() : null)
                .Aggregations(a => a
                    .Terms("group_IcCode", avg => avg
                    .Size(10000)
                        .Field(p => p.IcCode)
                        .Aggregations(aa => aa
                           .Terms("group_BmvProductNo", t2 => t2
                               .Size(10000)
                               .Field(f => f.BmvProductNo)
                               .Aggregations(aaa => aaa
                                   .Terms("group_ErrorMessage", t3 => t3
                                   .Field(f => f.ErrMessage.Suffix("keyword"))
                                   .Size(10000)
                                   .Aggregations(aaaa => aaaa
                                           .Max("max_log_date", m => m
                                               .Field(f => f.LogDateUtc)
                                           )
                                       )
                                   )
                               )
                           )
                        )
                    )
                );

When I made a request with this query, I realized that I could not see some values in the date range. 5-6 pieces of data were missing, and the data that was missing was data whose text field named ErrMessage was larger than 256 characters. I don't encounter the same problem without using aggregation. Could you please help about this?

Thanks.

2

Welcome!

Check the mapping. I think you are using the default one.

3

Hello again,

Thanks for your quick response. I changed the default value by making a request via devtools. This is what the mappings look like now. But still nothing changed.

resim
resim883×642 17.6 KB

4

Please don't post images of text as they are hard to read, may not display correctly for everyone, and are not searchable.

Instead, paste the text and format it with </> icon or pairs of triple backticks (```), and check the preview window to make sure it's properly formatted before posting it. This makes it more likely that your question will receive a useful answer.

Did you reindex your data?

Could you provide a full recreation script as described in About the Elasticsearch category. It will help to better understand what you are doing. Please, try to keep the example as simple as possible.

A full reproduction script is something anyone can copy and paste in Kibana dev console, click on the run button to reproduce your use case. It will help readers to understand, reproduce and if needed fix your problem. It will also most likely help to get a faster answer.

Have a look at the Elastic Stack and Solutions Help · Forums and Slack | Elastic page. It contains also lot of useful information on how to ask for help.

1 Like
5

I didn't reindex my data after change the ignore_above field.

There is a code about we wrote for change the ignore_above value.

PUT /service_logs_*/_mapping
{
  "properties": {
    "ErrMessage": {
      "type": "text",
      "fields": {
        "keyword": {
          "type": "keyword",
          "ignore_above": 2048
        }
      }
    }
  }
}

We used this for change the limit for ignore_above field from settings.

PUT service_logs_*/_settings
{
  "index.mapping.field_name_length.limit": 2048
}

This is the mapping request:

GET service_logs_20220308

This is the mapping response:

{
  "service_logs_20220308": {
    "aliases": {},
    "mappings": {
      "properties": {
        "BmvProductNo": {
          "type": "long"
        },
        "ClientIp": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "ErrMessage": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 2048
            }
          }
        },

(result goes on...)
6

That change will only apply to documents indexed from this point on and does not change any documents already indexed. To make this apply to all documents in the index you need to update/reindex.

1 Like
7

Thanks for your help. :pray:

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.