# Filtering question
**URL:** https://discuss.elastic.co/t/filtering-question/163241
**Category:** Logstash
**Created:** [January 7, 2019, 4:39pm UTC](https://discuss.elastic.co/t/filtering-question/163241 "2019-01-07T16:39:17Z")
**Posts on this page:** 12
**Page:** 1
### Author:  [@Diggy](https://discuss.elastic.co/u/Diggy)
#### Post date: [January 7, 2019, 4:39pm UTC](https://discuss.elastic.co/t/filtering-question/163241/1 "2019-01-07T16:39:17Z")
I have the following output, as viewed in Kibana:
> {
> "\_index": "filebeat-6.5.4-2019.01.07",
> "\_type": "doc",
> "\_id": "KoIlKWgBCwfgisRVoVxm",
> "\_version": 1,
> "\_score": null,
> "\_source": {
> "offset": 159112,
> "prospector": {
> "type": "log"
> },
> "source": "/var/log/syslog",
> "fileset": {
> "module": "system",
> "name": "syslog"
> },
> "input": {
> "type": "log"
> },
> "@timestamp": "2019-01-07T16:29:11.000Z",
> "system": {
> "syslog": {
> "hostname": "sentry01",
> "pid": "9886",
> "program": "ntpd",
> "message": "Soliciting pool server 91.189.89.199",
> "timestamp": "Jan 7 16:29:11"
> }
> },
> "beat": {
> "hostname": "sentry01.netatlantic.com.backend",
> "name": "sentry01.netatlantic.com.backend",
> "version": "6.5.4"
> },
> "host": {
> "name": "sentry01.netatlantic.com.backend"
> }
> },
> "fields": {
> "@timestamp": [
> "2019-01-07T16:29:11.000Z"
> ]
> },
> "highlight": {
> "system.syslog.hostname": [
> "@kibana-highlighted-field@sentry01@/kibana-highlighted-field@"
> ]
> },
> "sort": [
> 1546878551000
> ]
> }
I want to filter this out, but despite efforts, can't seem to do so. Here's an attempt at it:
> if [source] == "/var/log/syslog" and "Soliciting pool server" in [system][syslog][message] {
> drop {
> }
> }
Can someone help me out?
Many thanks.
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [January 7, 2019, 9:02pm UTC](https://discuss.elastic.co/t/filtering-question/163241/2 "2019-01-07T21:02:08Z")
Are you OK with using
```
if [source] == "/var/log/syslog" and [system][syslog][message] =~ "Soliciting pool server"
```
---
### Author:  [@Diggy](https://discuss.elastic.co/u/Diggy)
#### Post date: [January 7, 2019, 9:55pm UTC](https://discuss.elastic.co/t/filtering-question/163241/3 "2019-01-07T21:55:10Z")
> [@Badger](#):
>
> [source] == "/var/log/syslog" and [system][syslog][message] =~ "Soliciting pool server"
Thanks for the response. I have no trouble using the above. Unfortunately, though, it doesn't work. The stuff I'm trying to filter out is still getting through. I seem to be having a problem filtering out anything using fields with the pattern x.y.z
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [January 7, 2019, 10:19pm UTC](https://discuss.elastic.co/t/filtering-question/163241/4 "2019-01-07T22:19:16Z")
Using periods in field names is unsupported. It works most of the time, but some things will break. So having a field called system.syslog.hostname is not good.
Are you certain the system object has been parsed at the point where you are testing it?
Does the following work for you?
```
input { generator { count => 1 message => '{ "system": { "syslog": { "hostname": "sentry01", "pid": "9886", "program": "ntpd", "message": "Soliciting pool server 91.189.89.199", "timestamp": "Jan 7 16:29:11" } } }' } }
filter { json { source => "message" } }
filter { if [system][syslog][message] =~ "Soliciting pool server" { mutate { add_tag => ["ItMatched"] } } }
output { stdout { codec => rubydebug } }
```
---
### Author:  [@Diggy](https://discuss.elastic.co/u/Diggy)
#### Post date: [January 7, 2019, 10:30pm UTC](https://discuss.elastic.co/t/filtering-question/163241/5 "2019-01-07T22:30:32Z")
Badger,
This is exactly how the field looks in Kibana: system.syslog.message (there are others of the same format). I guess that's the way Filebeat presents the field.
Apologies,but, where/how do I run the code you just provided?
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [January 7, 2019, 10:48pm UTC](https://discuss.elastic.co/t/filtering-question/163241/6 "2019-01-07T22:48:36Z")
> [@Diggy](#):
>
> Apologies,but, where/how do I run the code you just provided?
Put it into /tmp/simple.conf and run logstash using something like
```
/usr/share/logstash/bin/logstash -f /tmp/simple.conf --path.settings /etc/logstash
```
---
### Author:  [@Diggy](https://discuss.elastic.co/u/Diggy)
#### Post date: [January 8, 2019, 1:30pm UTC](https://discuss.elastic.co/t/filtering-question/163241/8 "2019-01-08T13:30:14Z")
> [@Badger](#):
>
> /usr/share/logstash/bin/logstash -f /tmp/simple.conf --path.settings /etc/logstash
Apparently, it does work:
> root@elk01:/etc/logstash/conf.d# /usr/share/logstash/bin/logstash -f /tmp/simple.conf --path.settings /etc/logstash
> Sending Logstash logs to /var/log/logstash which is now configured via log4j2.properties
> [2019-01-08T08:25:13,764][WARN][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
> [2019-01-08T08:25:13,776][INFO][logstash.runner] Starting Logstash {"logstash.version"=\>"6.5.4"}
> [2019-01-08T08:25:14,570][INFO][logstash.pipeline] Starting pipeline {:pipeline\_id=\>"main", "pipeline.workers"=\>8, "pipeline.batch.size"=\>125, "pipeline.batch.delay"=\>50}
> [2019-01-08T08:25:14,603][INFO][logstash.pipeline] Pipeline started successfully {:pipeline\_id=\>"main", :thread=\>"#\"}
> [2019-01-08T08:25:14,616][INFO][logstash.agent] Pipelines running {:count=\>1, :running\_pipelines=\>[:main], :non\_running\_pipelines=\>}
> [2019-01-08T08:25:14,673][INFO][logstash.agent] Successfully started Logstash API endpoint {:port=\>9600}
> {
> "@timestamp" =\> 2019-01-08T13:25:14.611Z,
> "host" =\> "[elk01.mydomain.com](http://elk01.mydomain.com)",
> "tags" =\> [
> [0] "ItMatched"
> ],
> "message" =\> "{ "system": { "syslog": { "hostname": "sentry01", "pid": "9886", "program": "ntpd", "message": "Soliciting pool server 91.189.89.199", "timestamp": "Jan 7 16:29:11" } } }",
> "system" =\> {
> "syslog" =\> {
> "program" =\> "ntpd",
> "pid" =\> "9886",
> "hostname" =\> "sentry01",
> "message" =\> "Soliciting pool server 91.189.89.199",
> "timestamp" =\> "Jan 7 16:29:11"
> }
> },
> "sequence" =\> 0,
> "@version" =\> "1"
> }
> [2019-01-08T08:25:14,822][INFO][logstash.pipeline] Pipeline has terminated {:pipeline\_id=\>"main", :thread=\>"#\"}
But now, having gone through this, what does it mean in terms of getting my filter to work?
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [January 8, 2019, 1:33pm UTC](https://discuss.elastic.co/t/filtering-question/163241/9 "2019-01-08T13:33:10Z")
Can you show us your logstash config? The "if" works in your software version, but does not match the data in kibana. That suggests the data in logstash at the point where the "if" occurs does not match the data in kibana. i.e., there is some transformation in the logstash configuration that matters.
Something else that might help would be to show us the output of your data using "output { stdout { codec =\> rubydebug } }". That would make it clear if one of the structures in [system][syslog][message] is actually an array. I cannot remember if that shows up clearly in kibana.
---
### Author:  [@Diggy](https://discuss.elastic.co/u/Diggy)
#### Post date: [January 8, 2019, 2:04pm UTC](https://discuss.elastic.co/t/filtering-question/163241/10 "2019-01-08T14:04:17Z")
Sure. I have three config files - input, filter, and output. You can find them here: [https://pastebin.com/cZa8Wwth](https://pastebin.com/cZa8Wwth) . A warning that the filter config is very long. And, thanks for sticking with this! I very much appreciate it.
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [January 8, 2019, 3:29pm UTC](https://discuss.elastic.co/t/filtering-question/163241/11 "2019-01-08T15:29:56Z")
```
1. if [source] == "/var/log/syslog" and [system][syslog][message] =~ "Soliciting pool server" {
2. drop {
3. }
4. }
5. }
```
5 closes the filter. 3 closes the if. What does 4 close? I can't figure it out, but it seems to be inside another if, possibly an unintended one.
---
### Author:  [@Diggy](https://discuss.elastic.co/u/Diggy)
#### Post date: [January 8, 2019, 4:07pm UTC](https://discuss.elastic.co/t/filtering-question/163241/12 "2019-01-08T16:07:23Z")
Yeah, the filter config is a bit messy, but it does work (for the most part 🙂 ). That last } closes the first filter. I rejiggered the file to add } to the first filter (used for geoip), and removed it from the very end of the file (after the filter that I'm trying to make work). Alas, it still doesn't work. It's making me absolutely crazy!
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [February 5, 2019, 4:07pm UTC](https://discuss.elastic.co/t/filtering-question/163241/13 "2019-02-05T16:07:28Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.