john5634
(John T)
July 28, 2017, 9:09pm
1
We are trying to get the first 3 fields from a text app log file
2017-07-28 11:40:00,256 WARN [Timer-Driven Process Thread-7] o.a.h.c.protocol.ResponseProcessCookies Invalid cookie header: "set-cookie: guest_id=v1%3A150125640009836952; Expires=Sun, 28 Jul 2019 15:40:00 UTC; Path=/; Domain=.twitter.com". Invalid 'expires' attribute: Sun, 28 Jul 2019 15:40:00 UTC
2017-07-28 11:59:46,909 INFO [Provenance Maintenance Thread-1] o.a.n.p.PersistentProvenanceRepository Created new Provenance Event Writers for events starting with ID 65697100`
We want the first 3 fields to be set and the rest to show up a a regular message. Here is the example of our logstash conf file.
input {
tcp {
port => 10000
}
}
filter{
grok{
match => {"message" => "%{WORD:date} %{WORD:time} %{WORD:EventType}"}
}
}
output {
elasticsearch { hosts => ["localhost:9200"] }
}
any ideas?
Thank you!!!
1 Like
leandrojmp
(Leandro Pereira)
July 28, 2017, 10:54pm
2
You can use something like this:
%{DATE:date} %{TIME:time} %{WORD:EventType} %{GREEDYDATA:EventText}
You will have a field with the date in the format yy-mm-dd, a field with the time, a field with the event type and a field with the rest of the message.
LIke this:
{
"date": [
"17-07-28"
],
"time": [
"11:40:00,256"
],
"EventType": [
"WARN"
],
"EventText": [
"[Timer-Driven Process Thread-7] o.a.h.c.protocol.ResponseProcessCookies Invalid cookie header: "set-cookie: guest_id=v1%3A150125640009836952; Expires=Sun, 28 Jul 2019 15:40:00 UTC; Path=/; Domain=.twitter.com". Invalid 'expires' attribute: Sun, 28 Jul 2019 15:40:00 UTC"
]
}
You can test grok patterns using grok debug or if you have Kibana 5.5 with X-Pack, it's a new feature.
system
(system)
Closed
August 25, 2017, 10:54pm
3
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.