# Beats and logstash

**URL:** <https://discuss.elastic.co/t/beats-and-logstash/171402>\
**Category:** Logstash\
**Tags:** elastic-stack-monitoring\
**Created:** [March 7, 2019, 11:42pm UTC](https://discuss.elastic.co/t/beats-and-logstash/171402 "2019-03-07T23:42:16Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![Farhan\_Umer](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/farhan_umer/32/51423_2.png) [@Farhan\_Umer](https://discuss.elastic.co/u/Farhan_Umer)\
**Post date:** [March 7, 2019, 11:42pm UTC](https://discuss.elastic.co/t/beats-and-logstash/171402/1 "2019-03-07T23:42:16Z")

</div>

Hi,

I have 2 logstash files i want to create indexes with specific names such as IIS-Log-\<Today's Date\> and windows-eventlogs-\<Today's Date\>. I am using logstash with filebeat and winlogbeat. Right now everything is going to all the indexes. json file logs and all other logs are getting into both the indexes. both are using beats to send files. I don't want to use metadata because of required index names.

`Preformatted text` beats {  
port =\> 5000  
codec =\> "json\_lines"  
type =\> "iis"  
}  
}  
output {  
elasticsearch {  
hosts =\> ["localhost:9200"]  
index =\> "wazuh-alerts-3.x-%{+YYYY.MM.dd}"  
document\_type =\> "wazuh"  
}  
}

==========

input {

beats {  
port =\> 5044  
type =\> "iis"  
}

}

filter {  
....  
}

output {  
#if [type] == "iis" {  
elasticsearch {  
hosts =\> ["localhost:9200"]  
index =\> "iis-logs-%{+YYYY.MM.dd}"  
document\_type =\> "iis"  
}  
}

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [March 8, 2019, 12:32am UTC](https://discuss.elastic.co/t/beats-and-logstash/171402/2 "2019-03-08T00:32:28Z")

</div>

If you use -f (or path.config) to point to a directory, then all of the files in that directory are concatenated. Events are read from all of the inputs, sent through the filters, and then written (unless there are conditionals) to all of the outputs.

If you want each configuration file to be independent then you would have to use multiple [pipelines](https://www.elastic.co/guide/en/logstash/6.6/multiple-pipelines.html).

---

<div class="post-metadata">

**Author:** ![Farhan\_Umer](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/farhan_umer/32/51423_2.png) [@Farhan\_Umer](https://discuss.elastic.co/u/Farhan_Umer)\
**Post date:** [March 8, 2019, 10:14pm UTC](https://discuss.elastic.co/t/beats-and-logstash/171402/3 "2019-03-08T22:14:41Z")

</div>

Here is my configuration: Can you please see if i am doing anything wrong, I am getting no data at all. Indexes are there but no data in them.

input {  
beats {  
port =\> 5000  
codec =\> "json\_lines"  
}  
}

input {

beats {  
port =\> 5044  
}

}

filter {  
if [type] == "iis" {  
grok {  
match =\> ["message","%{TIMESTAMP\_ISO8601:log\_timestamp} "%{WORD:S\_SiteName}" "%{NOTSPACE:S\_ComputerName}" %{IPORHOST:S\_IP} %{WORD:CS\_Method} (?:-|%{URIPATH:CS\_URI\_Stem}) (?:-|%{GREEDYDATA:CS\_URI\_Query}) %{NUMBER:S\_Port} (?:-|%{NOTSPACE:CS\_Username}) (?:-|"%{IPORHOST:C\_IP}") "%{NOTSPACE:CS\_Version}" (?:-|%{QUOTEDSTRING:User\_Agent}) (?:-|"%{NOTSPACE:CS\_Referrer}") (?:-|"%{NOTSPACE:CS\_Host}") %{NUMBER:response} %{NUMBER:subresponse} %{NUMBER:scstatus} %{NUMBER:SC\_Bytes} %{NUMBER:CS\_Bytes} %{NUMBER:time\_taken} (?:-|%{IPORHOST:client\_ip}) (?:-|"%{GREEDYDATA:cust\_response\_log}") (?:-|"%{NUMBER:request\_count}") (?:-|"%{NOTSPACE:olx\_user}") (?:-|"%{TIMESTAMP\_ISO8601:request\_start\_time}") (?:-|"%{TIMESTAMP\_ISO8601:request\_end\_time}") (?:-|"%{NUMBER:dotnet\_time\_taken}")"]  
}  
}  
}

output {  
if [type] == "iis" {  
elasticsearch {  
hosts =\> ["localhost:9200"]  
index =\> "iis-logs-%{+YYYY.MM.dd}"  
document\_type =\> "iis"  
}  
}  
else if [type] == "winlogbeat"  
{  
elasticsearch {  
hosts =\> ["localhost:9200"]  
index =\> "windows-events-%{+YYYY.MM.dd}"  
document\_type =\> "winlogbeat"  
}  
}  
else if [type] == "wazuh"  
{  
elasticsearch {  
hosts =\> ["localhost:9200"]  
index =\> "wazuh-alerts-%{+YYYY.MM.dd}"  
document\_type =\> "wazuh"  
}  
}  
}

---

<div class="post-metadata">

**Author:** ![Farhan\_Umer](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/farhan_umer/32/51423_2.png) [@Farhan\_Umer](https://discuss.elastic.co/u/Farhan_Umer)\
**Post date:** [March 8, 2019, 10:31pm UTC](https://discuss.elastic.co/t/beats-and-logstash/171402/4 "2019-03-08T22:31:29Z")

</div>

Actually only IIS one is working.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [March 8, 2019, 11:32pm UTC](https://discuss.elastic.co/t/beats-and-logstash/171402/5 "2019-03-08T23:32:33Z")

</div>

If you use

```
output { stdout { codec => rubydebug } }

```

what do the winlogbeat and wazuh events look like?

---

<div class="post-metadata">

**Author:** ![Farhan\_Umer](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/farhan_umer/32/51423_2.png) [@Farhan\_Umer](https://discuss.elastic.co/u/Farhan_Umer)\
**Post date:** [March 9, 2019, 12:37am UTC](https://discuss.elastic.co/t/beats-and-logstash/171402/6 "2019-03-09T00:37:08Z")

</div>

> [@Badger](#):
>
> stdout

here it is stdout for the wazuh

{  
"timestamp" =\> "2019-03-08T19:33:53.192-0500",  
"id" =\> "1552091633.2035411",  
"location" =\> "netstat listening ports",  
"@version" =\> "1",  
"previous\_output" =\> "Previous output:\nossec: output: 'netstat listening ports':\ntcp 0.0.0.0:22 0.0.0.0:\* 6816/sshd\ntcp6 :::22 :::\* 6816/sshd\ntcp 0.0.0.0:25 0.0.0.0:\* 6641/master\ntcp6 :::25 :::\* 6641/master\ntcp 0.0.0.0:80 0.0.0.0:\* 9256/apache2\ntcp 0.0.0.0:111 0.0.0.0:\* 449/rpcbind\ntcp6 :::111 :::\* 449/rpcbind\nudp 0.0.0.0:111 0.0.0.0:\* 449/rpcbind\nudp6 :::111 :::\* 449/rpcbind\nudp 0.0.0.0:123 0.0.0.0:\* 7763/ntpd\nudp 10.60.2.125:123 0.0.0.0:\* 7763/ntpd\nudp 127.0.0.1:123 0.0.0.0:\* 7763/ntpd\nudp6 ::1:123 :::\* 7763/ntpd\nudp6 :::123 :::\* 7763/ntpd\nudp6 fe80::250:56ff:fe01:123 :::\* 7763/ntpd\ntcp 0.0.0.0:443 0.0.0.0:\* 9256/apache2\nudp 0.0.0.0:625 0.0.0.0:\* 449/rpcbind\nudp6 :::625 :::\* 449/rpcbind\ntcp 0.0.0.0:3306 0.0.0.0:\* 27718/mysqld\ntcp 0.0.0.0:5666 0.0.0.0:\* 579/nrpe\ntcp6 :::5666 :::\* 579/nrpe",  
"agent" =\> {  
"ip" =\> "10.60.2.125",  
"id" =\> "006",  
"name" =\> "Support"  
},  
"manager" =\> {  
"name" =\> "wazuh"  
},  
"beat" =\> {  
"name" =\> "wazuh",  
"hostname" =\> "wazuh",  
"version" =\> "6.6.0"  
},  
"host" =\> {  
"name" =\> "wazuh"  
},  
"previous\_log" =\> "ossec: output: 'netstat listening ports':\ntcp 0.0.0.0:22 0.0.0.0:\* 6816/sshd\ntcp6 :::22 :::\* 6816/sshd\ntcp 0.0.0.0:25 0.0.0.0:\* 6641/master\ntcp6 :::25 :::\* 6641/master\ntcp 0.0.0.0:80 0.0.0.0:\* 9256/apache2\ntcp 0.0.0.0:111 0.0.0.0:\* 449/rpcbind\ntcp6 :::111 :::\* 449/rpcbind\nudp 0.0.0.0:111 0.0.0.0:\* 449/rpcbind\nudp6 :::111 :::\* 449/rpcbind\nudp 0.0.0.0:123 0.0.0.0:\* 7763/ntpd\nudp 10.60.2.125:123 0.0.0.0:\* 7763/ntpd\nudp 127.0.0.1:123 0.0.0.0:\* 7763/ntpd\nudp6 ::1:123 :::\* 7763/ntpd\nudp6 :::123 :::\* 7763/ntpd\nudp6 fe80::250:56ff:fe01:123 :::\* 7763/ntpd\ntcp 0.0.0.0:443 0.0.0.0:\* 9256/apache2\nudp 0.0.0.0:625 0.0.0.0:\* 449/rpcbind\nudp6 :::625 :::\* 449/rpcbind\ntcp 0.0.0.0:3306 0.0.0.0:\* 27718/mysqld\ntcp 0.0.0.0:5666 0.0.0.0:\* 579/nrpe\ntcp6 :::5666 :::\* 579/nrpe",  
"source" =\> "/var/ossec/logs/alerts/alerts.json",  
"full\_log" =\> "ossec: output: 'netstat listening ports':\ntcp 0.0.0.0:22 0.0.0.0:\* 6816/sshd\ntcp6 :::22 :::\* 6816/sshd\ntcp 0.0.0.0:25 0.0.0.0:\* 6641/master\ntcp6 :::25 :::\* 6641/master\ntcp 0.0.0.0:80 0.0.0.0:\* 10050/apache2\ntcp 0.0.0.0:111 0.0.0.0:\* 449/rpcbind\ntcp6 :::111 :::\* 449/rpcbind\nudp 0.0.0.0:111 0.0.0.0:\* 449/rpcbind\nudp6 :::111 :::\* 449/rpcbind\nudp 0.0.0.0:123 0.0.0.0:\* 7763/ntpd\nudp 10.60.2.125:123 0.0.0.0:\* 7763/ntpd\nudp 127.0.0.1:123 0.0.0.0:\* 7763/ntpd\nudp6 ::1:123 :::\* 7763/ntpd\nudp6 :::123 :::\* 7763/ntpd\nudp6 fe80::250:56ff:fe01:123 :::\* 7763/ntpd\ntcp 0.0.0.0:443 0.0.0.0:\* 10050/apache2\nudp 0.0.0.0:625 0.0.0.0:\* 449/rpcbind\nudp6 :::625 :::\* 449/rpcbind\ntcp 0.0.0.0:3306 0.0.0.0:\* 27718/mysqld\ntcp 0.0.0.0:5666 0.0.0.0:\* 579/nrpe\ntcp6 :::5666 :::\* 579/nrpe",  
"@timestamp" =\> 2019-03-09T00:33:56.772Z,  
"rule" =\> {  
"id" =\> "533",  
"mail" =\> true,  
"gdpr" =\> [  
[0] "IV\_35.7.d"  
],  
"gpg13" =\> [  
[0] "10.1"  
],  
"groups" =\> [  
[0] "ossec"  
],  
"firedtimes" =\> 2,  
"pci\_dss" =\> [  
[0] "10.2.7",  
[1] "10.6.1"  
],  
"level" =\> 7,  
"description" =\> "Listened ports status (netstat) changed (new port opened or closed)."  
},  
"decoder" =\> {  
"name" =\> "ossec"  
},  
"offset" =\> 2693593,  
"log" =\> "",  
"tags" =\> [  
[0] "beats\_input\_raw\_event"  
]  
}

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [March 9, 2019, 12:54am UTC](https://discuss.elastic.co/t/beats-and-logstash/171402/7 "2019-03-09T00:54:05Z")

</div>

That does not have a [type] field containing wazuh. The only fields containing that are [manager][name], [beat][name], [beat][hostname], and [host][name].

I suggest you go back and check the beat config (and whether you are adding fields\_under\_root).

---

<div class="post-metadata">

**Author:** ![Farhan\_Umer](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/farhan_umer/32/51423_2.png) [@Farhan\_Umer](https://discuss.elastic.co/u/Farhan_Umer)\
**Post date:** [March 9, 2019, 2:16am UTC](https://discuss.elastic.co/t/beats-and-logstash/171402/8 "2019-03-09T02:16:19Z")

</div>

Pointed in the right direction. You saved the day. Respect ++

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 6, 2019, 2:16am UTC](https://discuss.elastic.co/t/beats-and-logstash/171402/9 "2019-04-06T02:16:21Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.