Getting illegal_state_exception error while pushing logs to elasticsearch

Elastic Stack Logstash
2

In elasticsearch a field can be either an object or a value, it cannot be an object in some documents and a value in others.

For example, suppose you have both a tcp input and a beats input sending events to the same pipeline. The tcp input will (unless ECS is enabled) add a [host] field to the event that contains a text string. Events from the beats input will have a [host][name] field, so [host] is an object.

If a template sets the mapping of the [host] field to be text, then all of the beats events will get rejected with an error like

"mapper_parsing_exception", "reason"=>"failed to parse field [host] of type [text] in document ... "reason"=>"Can't get text on a START_OBJECT at 1:10"}}}}}

If the template sets the mapping of [host] to be an object, then all of the tcp events will get rejected with an error like

"type"=>"mapper_parsing_exception", "reason"=>"object mapping for [host] tried to parse field [host] as object, but found a concrete value"

If there is no template then the mapping will be determined by whatever is in the first event indexed. If the tcp input sends an event before the beats input then it will be text, if the beats input is first then it will be an object. This means that the type (and error) can change every time the index rolls over.

To find out what mapping is currently configured for the index use the Mapping API.

One option is to rename the [host] field if it is a string.

if ! [host][name] { mutate { rename { "host" => "hostname" } }

or

if ! [host][name] { mutate { rename { "host" => "[host][name]" } }

Another is to rename the [host][name] field

if [host][name] { mutate { rename { "[host][name]" => "host" } }

but this loses all the other fields in the [host] object. Another option is to convert the [host] object to a string

    if [host][hostname] {
        ruby { code => 'event.set("host", event.get("host").to_s)' }
    }

will produce

      "host" => "{\"hostname\"=>\"bar\", \"os\"=>\"windows\"}"

As would

    ruby {
         code => '
            host = event.get("host")
            if host.is_a? Hash
                event.set("host", host.to_s)
            end
        '
    }
2 Likes