Logstash Adding Filebeat Fields

(Logeswaran) #1

Hi Team,

The design we used to read log files - Filebeat -> Kafka Topic -> Logstash -> Elasticsearch

When the message is arriving in Logstash it is in the following format


"message":"[2018-09-19 19:23:46,612] INFO Incrementing log start offset of partition _confluent-metrics-5 to 8644625 in dir /data/kafka/kafka-logs (kafka.log.Log)",

And I m using the following filter in Logstash

filter {

 mutate {

    add_field => {
      "source" => "%{[message][source]}"
      "kafkaTopic" => "%{[@metadata][kafka][topic]}"

 grok {
  match => { "inputjson" => "\[%{TIMESTAMP_ISO8601:logTime}\] %{LOGLEVEL:severity} %{GREEDYDATA:logMessage} \(%{JAVACLASS:loggerName}\)"}


  date {
         match => [ "logTime", "yyyy-MM-dd HH:mm:ss,SSS" ]
         target => "logTime"

if "_grokparsefailure" in [tags] {
         mutate {
                   remove_field => [ "@version","path","type","host" ]
 } else {
        mutate {
                   remove_field => [ "message","@version","path","type","host" ]


And the source filed is coming as string in the ES

"source" : "%{[message][source]}"

How to access the nested fields in the incoming message? Any help/directions would help me to fix the issue.Thanks

Logeswaran Radhakrishnan

(Logeswaran) #2

The reason I want to extract the fields like beat name, beat hostname and source because it is being dropped by the Logstash.

I have added the json codec and the fields are getting indexed properly. Thanks

This topic can be closed.

(Magnus Bäck) #3

How to access the nested fields in the incoming message?

Which nested fields? source isn't nested and the Kafka topic field is clearly named [@metadata][topic].

(Logeswaran) #4

I was trying to accessing fields like beat name and beat hostname. The above code representation was just representing the source .

The patterns like %{[message][beat][name]} , %{[message][source]} nothing was working.

(Magnus Bäck) #5

None of those fields are nested under message. Use [beat][name], source, and so on.

(Logeswaran) #6

Working fine. Thanks @magnusbaeck

(system) #7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.