# Filltering messages by tags problem with some messages

**URL:** <https://discuss.elastic.co/t/filltering-messages-by-tags-problem-with-some-messages/151856>\
**Category:** Logstash\
**Created:** [October 10, 2018, 1:43pm UTC](https://discuss.elastic.co/t/filltering-messages-by-tags-problem-with-some-messages/151856 "2018-10-10T13:43:33Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![Pietia](https://avatars.discourse-cdn.com/v4/letter/p/d07c76/32.png) [@Pietia](https://discuss.elastic.co/u/Pietia)\
**Post date:** [October 10, 2018, 1:43pm UTC](https://discuss.elastic.co/t/filltering-messages-by-tags-problem-with-some-messages/151856/1 "2018-10-10T13:43:33Z")

</div>

So I have a problem with logstash:  
some messages are being sent to default index:(below is the message that went into 'default' index.

> ```
> {
> "_index": "default-logs-2018.10.10",
> "_type": "doc",
> "_id": "MioyXmYBMkEJhrYr2PRg",
> "_score": 1,
> "_source": {
> "source": "/var/log/messages",
> "host": {
> "name": "serverx"
> },
> "beat": {
> "version": "6.4.2",
> "name": "serverx",
> "hostname": "serverx"
> },
> "@version": "1",
> "offset": 2767233,
> "fields": {
> "document_type": "syslog"
> },
> "@timestamp": "2018-10-10T13:37:56.000Z",
> "system": {
> "syslog": {
> "program": "root",
> "timestamp": "Oct 10 15:37:56",
> "hostname": "serverx",
> "message": "172.30.11.18 - - [10/Oct/2018:15:37:56 +0200] \"GET /something HTTP/1.1\" 200 3839 \"-\" \"Java/1.8.0_152\""
> }
> },
> "tags": [
> "syslog",
> "prod",
> "beats_input_codec_plain_applied"
> ]
> },
> "fields": {
> "@timestamp": [
> "2018-10-10T13:37:56.000Z"
> ]
> }
> }
> 
> ```

It happens only with syslog logs.  
This is relevant part of logstash

> ```
> if "syslog" in [tags] {
> grok {
> match => { "message" => ["%{SYSLOGTIMESTAMP:[system][syslog][timestamp]} %{SYSLOGHOST:[system][syslog][hostname]} %{DATA:[system][syslog][program]}(?:\[%{POSINT:[system][syslog][pid]}\])?: %{GREEDYMULTILINE:[system][syslog][message]}"] }
> pattern_definitions => { "GREEDYMULTILINE" => "(.|\n)*" }
> remove_field => "message"
> }
> date {
> match => ["[system][syslog][timestamp]", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]
> }
> }
> 
> output {
> if "syslog" in [tags] {
> elasticsearch {
> user => logstash
> password => logstash
> ssl => true
> hosts => ["https://localhost:9200"]
> cacert => '/etc/logstash/esnode.pem'
> sniffing => false
> manage_template => false
> index => "system-logs-%{+YYYY.MM.dd}"
> document_type => "%{[@metadata][type]}"
> }
> }
> else {
> elasticsearch {
> user => logstash
> password => logstash
> ssl => true
> hosts => ["https://localhost:9200"]
> cacert => '/etc/logstash/esnode.pem'
> sniffing => false
> manage_template => false
> index => "default-logs-%{+YYYY.MM.dd}"
> document_type => "%{[@metadata][type]}"
> }
> }
> 
> ```

This is filebeats conf:

> ```
> - input_type: log
> paths:
> - /var/log/messages*
> - /var/log/syslog*
> fields:
> document_type: syslog
> multiline.pattern: "^\\s"
> multiline.match: after
> exclude_files: [".gz$"]
> tags: ["syslog", "prod"]
> 
> ```

Same amount of messages goes to defaults-logs index and system-logs index.

---

<div class="post-metadata">

**Author:** ![Pietia](https://avatars.discourse-cdn.com/v4/letter/p/d07c76/32.png) [@Pietia](https://discuss.elastic.co/u/Pietia)\
**Post date:** [October 11, 2018, 11:11am UTC](https://discuss.elastic.co/t/filltering-messages-by-tags-problem-with-some-messages/151856/2 "2018-10-11T11:11:37Z")

</div>

Actually it seems like other messages are also affected not only system logs...

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [November 8, 2018, 11:11am UTC](https://discuss.elastic.co/t/filltering-messages-by-tags-problem-with-some-messages/151856/3 "2018-11-08T11:11:37Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.