# Logstash - selectively applying filter **URL:** \ **Category:** Logstash\ **Created:** [September 29, 2017, 8:25am UTC](https://discuss.elastic.co/t/logstash-selectively-applying-filter/102226 "2017-09-29T08:25:56Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![rs\_ela](https://avatars.discourse-cdn.com/v4/letter/r/f6c823/32.png) [@rs\_ela](https://discuss.elastic.co/u/rs_ela)\ **Post date:** [September 29, 2017, 8:25am UTC](https://discuss.elastic.co/t/logstash-selectively-applying-filter/102226/1 "2017-09-29T08:25:57Z")
Hi there I am sending logs from my nginx instance to ELK via Filebeat. The problem is that nginx access logs are json, but error logs are not and Logstash reports a lot of parsing errors which then swamp the log file on the disk. In order to avoid that I temporarily upped Logstash log level to critical, but I am not happy with that solution. The goal: - accept both error and access logs from nginx - avoid getting parsing errors - nginx access logs are in json, must be - nginx error logs are plaintext and it's fine (but could be json as well, no matter) Solutions? 1. configure nginx so that both access and error logs are json - but I believe it's currently not possible with reasonable amount of work 2. configure Logstash so that it uses json codec on access logs and plaintext codec on error logs - but how do I do that? My current Logstash config is as below. Tried configuring Logstash according to the 2nd solution, but failed. It seems that someone has already faced such issue ([Can I use both json and plain logs in a common filebeat input?](https://discuss.elastic.co/t/can-i-use-both-json-and-plain-logs-in-a-common-filebeat-input/36879)), but I am not sure how do I "selectively apply a json filter if the line looks like JSON"? Could someone help? Many thanks in advance for any suggestions. ``` input { beats { port => 1234 codec => "json" } } output { elasticsearch { hosts => ["localhost:9200"] } } ``` ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [September 29, 2017, 8:48am UTC](https://discuss.elastic.co/t/logstash-selectively-applying-filter/102226/2 "2017-09-29T08:48:17Z")
Can't you do the JSON parsing on the Filebeat side instead? Or at least assign a distinct type (with `document_type`) to each kind of log so that you can conditionally apply a json filter on the Logstash side? ---
**Author:** ![rs\_ela](https://avatars.discourse-cdn.com/v4/letter/r/f6c823/32.png) [@rs\_ela](https://discuss.elastic.co/u/rs_ela)\ **Post date:** [September 29, 2017, 10:08am UTC](https://discuss.elastic.co/t/logstash-selectively-applying-filter/102226/3 "2017-09-29T10:08:38Z")
I would like to use as little computing power on the nginx side as possible. So even if json parsing does not seem like a cpu-hungry operation I'd rather stick with option no 2. So this would be config on Filebeat side: ``` - input_type: log paths: - /var/log/nginx/*/access.log document_type: metricsets - input_type: log paths: - /var/log/nginx/*/error.log document_type: nginx_error_log ``` Then how do I approach conditionally appling json filter in Logstash input config? ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [September 29, 2017, 11:20am UTC](https://discuss.elastic.co/t/logstash-selectively-applying-filter/102226/4 "2017-09-29T11:20:23Z")
> Then how do I approach conditionally appling json filter in Logstash input config? > **[Accessing event data and fields | Logstash Reference \[8.11\] | Elastic](https://www.elastic.co/guide/en/logstash/current/event-dependent-configuration.html)** ---
**Author:** ![rs\_ela](https://avatars.discourse-cdn.com/v4/letter/r/f6c823/32.png) [@rs\_ela](https://discuss.elastic.co/u/rs_ela)\ **Post date:** [September 29, 2017, 1:16pm UTC](https://discuss.elastic.co/t/logstash-selectively-applying-filter/102226/5 "2017-09-29T13:16:10Z")
So we've got something like this: ``` input { beats { port => 1234 } } output { if [type] == "nginx_error_log" { elasticsearch { codec => plain hosts => ["localhost:9200"] } } else { elasticsearch { codec => json hosts => ["localhost:9200"] } } } ``` Thought that might do the job, but by the time the data reaches output section, the codec (plaintext) has already been applied by the beats input plugin. Should I apply the filter in the input section instead? Or maybe should I try and mutate the codec in the output section? Don't know if this idea is a good one? ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [September 29, 2017, 1:55pm UTC](https://discuss.elastic.co/t/logstash-selectively-applying-filter/102226/6 "2017-09-29T13:55:50Z")
Don't set the codec option for the elasticsearch output. Use a json filter instead and wrap it in a conditional. ---
**Author:** ![rs\_ela](https://avatars.discourse-cdn.com/v4/letter/r/f6c823/32.png) [@rs\_ela](https://discuss.elastic.co/u/rs_ela)\ **Post date:** [September 30, 2017, 1:19pm UTC](https://discuss.elastic.co/t/logstash-selectively-applying-filter/102226/7 "2017-09-30T13:19:29Z")
Right. Managed to come up with working config. Filebeat: ``` - input_type: log paths: - /var/log/nginx/*/access.log document_type: nginx_access_log - input_type: log paths: - /var/log/nginx/*/error.log document_type: metricsets ``` Logstash: ``` input { beats { port => 1234 } } filter { if [type] == "nginx_access_log" { json { source => "message" } } } output { elasticsearch { hosts => ["localhost:9200"] } } ``` Will let it work for a couple of days and see what happens, but so far it's delivering. Thank you for your help! ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [October 28, 2017, 1:19pm UTC](https://discuss.elastic.co/t/logstash-selectively-applying-filter/102226/8 "2017-10-28T13:19:32Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.