Can you help me to discover if this is possible? Filter a mixed log

[Takitani (Alex Takitani)]

I receive logs from a heroku drain, like this:

295 <158>1 2017-03-06T18:54:32.485983+00:00 host heroku router - at=info method=GET path="/admin" host=myhost.com request_id=0ba313f9-9e88-449e-b4b6-7e34205cf3fe fwd="2a01:0e34:ed36:0190:6504:0424:3e4c:6533,108.162.229.41" dyno=web.1 connect=0ms service=89ms status=200 bytes=7563

1409 <190>1 2017-03-06T18:49:38.405766+00:00 host app web.1 - App 135 stdout: {"method":"GET","path":"/things/choice","format":"html","controller":"Controller","action":"choice","status":200,"duration":18.43,"view":9.34,"db":3.73,"exception":null,"exception_object":null,"user_id":10496,"remote_ip":"108.162.229.200","params":{},"sql_queries":"'User Load (0.71) SELECT \"users\".* FROM \"users\" WHERE \"users\".\"id\" = $1 ORDER BY \"users\".\"id\" ASC LIMIT 1\nProfileFeature Exists (0.62) SELECT 1 AS one FROM \"profile_features\" WHERE \"profile_features\".\"user_id\" = $1 LIMIT 1","@timestamp":"2017-03-06T18:49:38.405Z","@version":"1","message":"[200] GET /things/choice (ThingsController#choice)"}

139 <190>1 2017-03-06T18:33:20.288809+00:00 host app worker.1 - 4 TID-otu2rv610 SaveJob JID-68a22876a6fb0e2cf6f6a4c8 INFO: done: 0.301 sec

They come from router, app and worker all on the same file.

I've got the router working on a grok filter.

What do I do now? a IF checking for the existence of worker / router / app and different filters?

Thanks!

[magnusbaeck (Magnus Bäck)]

What do I do now? a IF checking for the existence of worker / router / app and different filters?

Yes, something like that. You can use a grok filter to extract the set of common prefix fields, then use a conditional to select which set of filters to apply to the rest.