How to Parse App Log from Apache nifi


(John T) #1

We are trying to get the first 3 fields from a text app log file

2017-07-28 11:40:00,256 WARN [Timer-Driven Process Thread-7] o.a.h.c.protocol.ResponseProcessCookies Invalid cookie header: "set-cookie: guest_id=v1%3A150125640009836952; Expires=Sun, 28 Jul 2019 15:40:00 UTC; Path=/; Domain=.twitter.com". Invalid 'expires' attribute: Sun, 28 Jul 2019 15:40:00 UTC
2017-07-28 11:59:46,909 INFO [Provenance Maintenance Thread-1] o.a.n.p.PersistentProvenanceRepository Created new Provenance Event Writers for events starting with ID 65697100` 

We want the first 3 fields to be set and the rest to show up a a regular message. Here is the example of our logstash conf file.

input {

    tcp {
            port => 10000
    }
}
filter{

grok{
 match => {"message" => "%{WORD:date} %{WORD:time} %{WORD:EventType}"}
}

}
output {
  elasticsearch { hosts => ["localhost:9200"] }
}

any ideas?

Thank you!!!


(Leandro Maciel) #2

You can use something like this:

%{DATE:date} %{TIME:time} %{WORD:EventType} %{GREEDYDATA:EventText}

You will have a field with the date in the format yy-mm-dd, a field with the time, a field with the event type and a field with the rest of the message.

LIke this:

{
  "date": [
    "17-07-28"
  ],
  "time": [
    "11:40:00,256"
  ],
  "EventType": [
    "WARN"
  ],
  "EventText": [
    "[Timer-Driven Process Thread-7] o.a.h.c.protocol.ResponseProcessCookies Invalid cookie header: "set-cookie: guest_id=v1%3A150125640009836952; Expires=Sun, 28 Jul 2019 15:40:00 UTC; Path=/; Domain=.twitter.com". Invalid 'expires' attribute: Sun, 28 Jul 2019 15:40:00 UTC"
  ]
}

You can test grok patterns using grok debug or if you have Kibana 5.5 with X-Pack, it's a new feature.


(system) #3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.