Logstash. Get fields by position number


(Paul Letski) #1

Background

I have the scheme: logs from my app go through rsyslog to central log server, then to Logstash and Elasticsearch. Logs from app is a pure JSON, but rsyslog adds to log "timestamp", "app name" and "server name" fileds. And log becomes to this:

timestamp app-name server-name [JSON]

Question

How can I remove first three fields with Logstash filters? Can I get fields by position numbers (like in awk) and do something like:

filter {
  somefilter_name {
      remove_field => $1, $2, $3 
  }
}

Or maybe my vision is totally wrong and I must do this in another way?

Thank you!


(Mark Walkom) #2

Do you want to remove them completely or just from the message part?

You can use mutate to do either.


(Paul Letski) #3

I want to remove them completely.
I got an advice on stackoverflow to use filter like this:

grok {
     pattern => "%{TIMESTAMP_ISO8601:timestamp} %{WORD:app} %{WORD:server} %{GREEDYDATA:message}"
     overwrite => [ "message" ]
}

json {
    source => "message"
}

But now I'm getting error in log about "Trouble parsing json".


(Magnus B├Ąck) #4

Well, what does the message field that you're trying to parse as JSON look like?


(system) #5