How to remove restriction from the value of a field in a document such that the field doesn't get marked as _ignored due to it's long length

Elastic Stack Kibana
1

PROLOGUE:

We are using Elastic Fleet and Elastic Agents.

Using filebeat and ingest pipeline we are fetching logs from a custom log file kept on an elastic agent.

These logs are then indexed in Elastic Search and can be viewed on Kibana.

There is a field in each document named LOGS.

PROBLEM :

When the data is indexed, in some documents LOGS field is marked as "_ignored" which can be seen in the Discover view.

When trying to display LOGS in the Dashboard using Top Hit aggregation, the documents which had their LOGS field marked _ignored are not being shown.

After some digging I found out that there is a limit to how long the value of a field can be before it becomes ignored.

I want to know if there is a way to allow very long strings to get indexed without getting marked as _ignored, and also is this restriction applied by elasticsearch or kibana ?

2

Generally you would manage this in a template that creates the index mapping.

What is the output of GET INDEXNAME/_mapping and which field is the one with issues?

3 
{
  ".ds-logs-server-wbw-2023.02.16-000001": {
    "mappings": {
      "_data_stream_timestamp": {
        "enabled": true
      },
      "dynamic_templates": [
        {
          "match_ip": {
            "match": "ip",
            "match_mapping_type": "string",
            "mapping": {
              "type": "ip"
            }
          }
        },
        {
          "match_message": {
            "match": "message",
            "match_mapping_type": "string",
            "mapping": {
              "type": "match_only_text"
            }
          }
        },
        {
          "strings_as_keyword": {
            "match_mapping_type": "string",
            "mapping": {
              "ignore_above": 1024,
              "type": "keyword"
            }
          }
        }
      ],
      "date_detection": false,
      "properties": {
        "@timestamp": {
          "type": "date"
        },
        "HASH": {
          "type": "keyword",
          "ignore_above": 1024
        },
        "LOGS": {
          "type": "keyword",
          "ignore_above": 1024
        },
        "data_stream": {
          "properties": {
            "dataset": {
              "type": "constant_keyword"
            },
            "namespace": {
              "type": "constant_keyword"
            },
            "type": {
              "type": "constant_keyword",
              "value": "logs"
            }
          }
        },
        "ecs": {
          "properties": {
            "version": {
              "type": "keyword",
              "ignore_above": 1024
            }
          }
        },
        "host": {
          "type": "object"
        },
        "message": {
          "type": "match_only_text"
        }
      }
    }
  }
}

Above is the output for GET INDEXNAME/_mapping.
Here you can see that LOGS property has a property "ignore_above" : 1024

I want to increase this limit before creating the index and data stream.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.