Blank message field

Elastic Stack Logstash
1

Trying to create a simple, non-filtered pipeline, but Kibana shows the message field as blank
image

Pipeline conf is

input {
  tcp {
    port => 9600
    codec => json
    mode => server
  }
}
output {
  elasticsearch {
    hosts => ["localhost:9200"]
    index => "myindex"
  }
}

When starting Logstash the following message is displayed

[WARN ] 2020-08-10 16:33:02.698 [[main]>worker0] elasticsearch - Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"myindex", :routing=>nil, :_type=>"_doc"}, #<LogStash::Event:0xbba272>], :response=>{"index"=>{"_index"=>"myindex", "_type"=>"_doc", "_id"=>"oJs32XMBjyfMsZpZ2EjQ", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"object mapping for [message] tried to parse field [message] as object, but found a concrete value"}}}}

Other fields are being mapped, but I would like to see the full message as wel...

2

Does this help?

3

This is the mapping of the message field:

      },
        "message" : {
          "properties" : {
            "someField" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            }
          }
        },

Tried to change/add this mapping with

{
  "properties": {
    "message": {
      "type": "object"
    }
  }
}

My point is - I am missing some information on the fields ES is detecting, so I would like to see the full message. Am I in the right direction?

Thank you

4

If I am reading that correctly then elasticsearch expects the field

[message][someField]

to exist. That makes [message] an object, not a string. You could try

if ![message][someField] { mutate { rename => { "message" => "originalMessage" } } }
5

Thanks @Badger after applying the filter you suggested, the messages are being displayed on the doc.

However, every line of this message generates a new doc, instead of showing the full message in the same doc. Is there any way we can group these lines into a single message?

** Edit: These docs share a common field, i.e. execution.execid - Is there a way to aggregate all messages into the same doc, based on this ID?

Thanks

6

I have no idea why that could be happening, so no suggestions on how to fix it.

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.