org.elasticsearch.index.mapper.MapperParsingException: object mapping for [message] tried to parse field [message] as object, but found a concrete value

Elastic Stack Logstash
1

I upload a template, and try to parse a column, but it recognized as a string, how to make it as a object, please help...

Here is my template:

{
  "index_patterns": "springboot*",
  "order": 0,
  "settings": {
    "number_of_shards": 1,
    "number_of_replicas": "0"
  },
  "aliases": {
    "alias_1": {}
  },
  "mappings": {
    "properties": {
      "@timestamp": {
        "type": "date"
      },
      "@version": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      },
      "host": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      },
      "level": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      },
      "level_value": {
        "type": "long"
      },
      "logger_name": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      },
      "message": {
       "type":"object", 
        
        "properties": {
          "appName": {
            "type": "text"
          },
          "className": {
            "type": "text"
            
          },
          "methodName": {
            "type": "text"
            
          }
        }
      },
      "port": {
        "type": "long"
      },
      "tags": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      },
      "thread_name": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      }
    }
  }
}
2

I think either you have an existing index before you have added the Template or the object is not recognized as an object in logstash.

So if it is a json object you can use the json filter. After that Elasticsearch can Index it as an object.

3 
filter {
  json {
    source => "message"
  }
}

I add the json filter to my logstash conf, but it does not work.

https://www.elastic.co/guide/en/logstash/current/plugins-filters-json.html

4

What do you get if you use

output { stdout { codec => rubydebug } }

What does logstash log?

5

Here is the output:

[2019-05-24T21:11:17,880][DEBUG][o.e.a.b.TransportShardBulkAction] [VM_0_17_centos] [springboot-logstash-2019.05.24][0] failed to execute bulk item (index) index {[springboot-logstash-2019.05.24][_doc][mV_36WoBWEbX7YrrlP54], source[{"@timestamp":"2019-05-24T13:12:57.691Z","thread_name":"http-nio-8080-exec-1","@version":"1","logger_name":"com.example.demo.MyLog","message":"{\"appName\":\"demo\",\"className\":\"HelloController\",\"methodName\":\"hello\"}","appName":"demo","port":19333,"level":"INFO","host":"111.203.45.2","className":"HelloController","methodName":"hello","level_value":20000}]}
org.elasticsearch.index.mapper.MapperParsingException: object mapping for [message] tried to parse field [message] as object, but found a concrete value
        at org.elasticsearch.index.mapper.DocumentParser.parseObjectOrNested(DocumentParser.java:363) ~[elasticsearch-7.0.0.jar:7.0.0]
        at org.elasticsearch.index.mapper.DocumentParser.parseObjectOrField(DocumentParser.java:465) ~[elasticsearch-7.0.0.jar:7.0.0]
        at org.elasticsearch.index.mapper.DocumentParser.parseValue(DocumentParser.java:596) ~[elasticsearch-7.0.0.jar:7.0.0]
        at org.elasticsearch.index.mapper.DocumentParser.innerParseObject(DocumentParser.java:407) ~[elasticsearch-7.0.0.jar:7.0.0]
        at org.elasticsearch.index.mapper.DocumentParser.parseObjectOrNested(DocumentParser.java:381) ~[elasticsearch-7.0.0.jar:7.0.0]
        at org.elasticsearch.index.mapper.DocumentParser.internalParseDocument(DocumentParser.java:98) ~[elasticsearch-7.0.0.jar:7.0.0]
        at org.elasticsearch.index.mapper.DocumentParser.parseDocument(DocumentParser.java:71) ~[elasticsearch-7.0.0.jar:7.0.0]
        at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:267) ~[elasticsearch-7.0.0.jar:7.0.0]
        at org.elasticsearch.index.shard.IndexShard.prepareIndex(IndexShard.java:770) ~[elasticsearch-7.0.0.jar:7.0.0]
        at org.elasticsearch.index.shard.IndexShard.applyIndexOperation(IndexShard.java:747) ~[elasticsearch-7.0.0.jar:
6

How does the "message" look like? not the logstash log.

7

Here is the "message":

"message":"{\"appName\":\"demo\",\"className\":\"HelloController\",\"methodName\":\"hello\"}"

I want to parse it as a json object.

8

ah ok, this is not an object.

with this you get "appName" and "className"... as fields on their own.
so they won´t be nested under "message"

this means:
you will have the "message" field and all of those fields extracted with the json filter as fields on top level.

if you want to get something like [message][appName] and [message][className]...
then you can try the json filter with the option "target" => "message"

BUT if you try to put other logs inside this index which only have a "message" field with a normal value then it wont be logged.

9

Thanx a lot!!!

So, I should make it like this:

filter {
  json {
    source => "message"
    target => "message"
  }
}

Am I right?

10

yes this should delete the original message which is "key":"value" and create a "object":{"key":"value","key":value"}

I havent tried it but this should work.
or If you want to be more safe you can:
mutate { rename => { "message" => "msg" } }
json { source => "msg" target => "message" }

then you will have the msg field with the original event and the message should be an object

11

Thanks a loooot.

I will have a try.

12

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.