Impact of ignore_above property in ES mapping

Elastic Stack Elasticsearch
1

We've configured logstash in combination with elasticsearch and Kibana to
centralize our server logs.
In Kibana I want to setup a table which groups all error messages so we can
create a top 10 of the most occuring errors.

We tried to setup a terms table grouped on a specific field (which contains
the error message).
However there is a record "missing fields" in the table with a count of
16997.

The configuration of the terms table results in the following elasticsearch
query (posting a part of the filter):

"facets": {
"terms": {
"terms": {
"field": "information.information.raw",
"size": 100000000,
"order": "count",
"exclude": []
}
[...]

In the result a missing property indicates quite a lot of missing results
as we've noticed in the terms table itself:

"facets": {
"terms": {
"_type": "terms",
"missing": 16997,
"total": 584,
"other": 0,
[...]

I've noticed that in the mapping defined by logstash the following is added:

"information.raw": {
"type": "string",
"index": "not_analyzed",
"ignore_above": 256
}

I assume that ignore_above means that any message longer then 256
characters will not be taken into account when doing queries?
I tried to find some documentation about it but the only thing I can find
is at the Core Types documentation
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-core-types.html
of elasticsearch and I'm not entirely sure if I understand it correctly.

If I remove this property in the mapping, what could be the impact of this?
Slower queries because it needs to take larger strings into account? Is if
safe to remove this property?

Thanks in advance!

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/3a5f188f-e77e-458a-ab34-eaf913f8b291%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

2

Bump? I actually removed the "ignore_above" option from the template and
disabled logstash from creating a template. So my mapping now looks like:

      "error-message" : {
        "type" : "string",
        "norms" : {
          "enabled" : false
        },
        "fields" : {
          "raw" : {
            "type" : "string",
            "index" : "not_analyzed"
          }
        }
      },

But I'm still not seeing any results in kibana. Anyone have any ideas on
this?

--Ajay

On Friday, November 14, 2014 at 12:24:25 AM UTC-8, thomasvdb wrote:

We've configured logstash in combination with elasticsearch and Kibana to
centralize our server logs.
In Kibana I want to setup a table which groups all error messages so we
can create a top 10 of the most occuring errors.

We tried to setup a terms table grouped on a specific field (which
contains the error message).
However there is a record "missing fields" in the table with a count of
16997.

The configuration of the terms table results in the following
elasticsearch query (posting a part of the filter):

"facets": {
"terms": {
"terms": {
"field": "information.information.raw",
"size": 100000000,
"order": "count",
"exclude":
}
[...]

In the result a missing property indicates quite a lot of missing results
as we've noticed in the terms table itself:

"facets": {
"terms": {
"_type": "terms",
"missing": 16997,
"total": 584,
"other": 0,
[...]

I've noticed that in the mapping defined by logstash the following is
added:

"information.raw": {
"type": "string",
"index": "not_analyzed",
"ignore_above": 256
}

I assume that ignore_above means that any message longer then 256
characters will not be taken into account when doing queries?
I tried to find some documentation about it but the only thing I can find
is at the Core Types documentation
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-core-types.html
of elasticsearch and I'm not entirely sure if I understand it correctly.

If I remove this property in the mapping, what could be the impact of this?
Slower queries because it needs to take larger strings into account? Is if
safe to remove this property?

Thanks in advance!

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/ffa0a7d3-2b95-4d0c-a86d-58f77ec01011%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.