Logstash Adding Filebeat Fields

Hi Team,

The design we used to read log files - Filebeat -> Kafka Topic -> Logstash -> Elasticsearch

When the message is arriving in Logstash it is in the following format


"message":"[2018-09-19 19:23:46,612] INFO Incrementing log start offset of partition _confluent-metrics-5 to 8644625 in dir /data/kafka/kafka-logs (kafka.log.Log)",

And I m using the following filter in Logstash

filter {

 mutate {

    add_field => {
      "source" => "%{[message][source]}"
      "kafkaTopic" => "%{[@metadata][kafka][topic]}"

 grok {
  match => { "inputjson" => "\[%{TIMESTAMP_ISO8601:logTime}\] %{LOGLEVEL:severity} %{GREEDYDATA:logMessage} \(%{JAVACLASS:loggerName}\)"}


  date {
         match => [ "logTime", "yyyy-MM-dd HH:mm:ss,SSS" ]
         target => "logTime"

if "_grokparsefailure" in [tags] {
         mutate {
                   remove_field => [ "@version","path","type","host" ]
 } else {
        mutate {
                   remove_field => [ "message","@version","path","type","host" ]


And the source filed is coming as string in the ES

"source" : "%{[message][source]}"

How to access the nested fields in the incoming message? Any help/directions would help me to fix the issue.Thanks

Logeswaran Radhakrishnan

The reason I want to extract the fields like beat name, beat hostname and source because it is being dropped by the Logstash.

I have added the json codec and the fields are getting indexed properly. Thanks

This topic can be closed.

How to access the nested fields in the incoming message?

Which nested fields? source isn't nested and the Kafka topic field is clearly named [@metadata][topic].

I was trying to accessing fields like beat name and beat hostname. The above code representation was just representing the source .

The patterns like %{[message][beat][name]} , %{[message][source]} nothing was working.

None of those fields are nested under message. Use [beat][name], source, and so on.

Working fine. Thanks @magnusbaeck

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.