How to set the "ignore_above" on elasticsearch / using logstash and kibana

Elastic Stack Kibana
1

Hi,
can anybody tell me where I can find the config, displayed in kibana to set ignore_above to a higher value? I'm using Ubuntu.

The current 256 chars don't fit as we often get longer messages (even after parsing). Currently I've got to use the truncate filter which often cuts off relevant informations.

2

Hi @Christian_Lorenz,

Try to do a mapping on your index, see: https://www.elastic.co/guide/en/elasticsearch/reference/current/ignore-above.html

Thanks,
Liza

3

Well, yes @LizaD this was my plan. But where? We‘re using Elasticsearch 2 times: One time as a search engine, the second time for log storage. On the searching one I know where to put it. But on the 2nd installation which is basically install Elasticsearch, Logstash and Kibana and let it go I‘ve got troubles where to change it as just using Elasticsearch at the Logstash output pipelines config. So I need the config file - where it’s usually located to change this param.

4

Hi @Christian_Lorenz,

You can use curl request or Kibana Devtools console to send the API requests on the index, I am not sure ignore_above goes into the config file itself. I will see if I can get someone to give more details.

Thanks,
Liza

5

I found @talevy who can give more detail on some different ways to do this.

6

Hi @Christian_Lorenz,

Just as @LizaD recommended, I would suggest updating the ignore_above[1] configuration for strings in your index templates associated with your Logstash indices. This can either be done by editing the existing default logstash mapping, or by applying a new template that takes precedence [2] over the existing ones.

Without knowing exactly how your Logstash mappings are determined, I will try and share a snippet of a new template that would define the behavior I believe you are hoping to modify.

PUT _template/ignore_above
{
  "index_patterns": [
    "logstash*"
  ],
  "mappings": {
    "dynamic_templates": [
      {
        "strings_as_keyword": {
          "match_mapping_type": "string",
          "mapping": {
            "type": "text",
            "fields": {
              "keyword": {
                "ignore_above": 512,
                "type": "keyword"
              }
            }
          }
        }
      }
    ]
  }
}

You can find the default template for Logstash and ES 7.x here [3] You will notice that there is no explicit mention of the ignore_above: 256 there. That is because this is a global Elasticsearch default on all dynamic fields that are recognized as strings. Here is a blog post that explains this. For more information about Logstash template management you can check out the docs

Does that help?

7

Hi, thanks! Yes it worked (but strangely only after the 4th attempt) - at least on my local machine :slight_smile:

8

great!

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.