Grok Pattern to get response_time and app_id from Pivotal CloudFoundry Loggregator

Elastic Stack Logstash
1

Hi All,

I am trying to get few fields from Logs of my app on Pivotal Cloud Foundry to analyze them using ELK but not able to get all the fields ,below is log from my APP

2015-08-03T09:51:15.000+00:00 [RTR] OUT mm1-spring-music.example.kpit.com - [03/08/2015:09:51:15 +0000] "GET /assets/templates/albums.html HTTP/1.1" 200 1307 "http://mm1-spring-music.pcfdev.example.com/" "Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/44.0.2403.125 Safari/537.36" XX.XX.XXX.XX:XXX 1x_forwarded_for:"XXX.XX.2XX.XXX, XX.XX.XXX.XX" vcap_request_id:7962cc63-e589-409a-59c5-3d14a640fa74 response_time:0.002220537 app_id:08be9fc8-c7a3-4613-bf12-1a9c7d98cc27

using below pattern i am able to get fields like AppName,TimeStamp,Process,MsgType(Error or Warn) but i want to get Agent_used,Response_Time and App_id if present in logs like in above case

Pattern i have used is

filter {
if [type] == "syslog" {
    grok {
      match => { "message"=>"(?:%{TIMESTAMP_ISO8601:timestamp}|-) +(?:%{SYSLOG5424SD:process}|-) +(?:%{WORD:msgtype}|-) +(?:%{NOTSPACE:appname}|-) +(?:%{GREEDYDATA:foundry_message}|-)"
}

  add_field => [ "received_at", "%{@timestamp}" ]
  add_field => [ "received_from", "%{host}" ]

}
syslog_pri { }
date {
   match => [ "syslog_ts", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
}
  }

}

Kindly suggest how to get the remaining fields.

2

Probably cause that last greedydata will grab them.

Check out http://grokdebug.herokuapp.com/ and see if you can build something.

1 Like