Beats and logstash

Elastic Stack Logstash
1

Hi,

I have 2 logstash files i want to create indexes with specific names such as IIS-Log-<Today's Date> and windows-eventlogs-<Today's Date>. I am using logstash with filebeat and winlogbeat. Right now everything is going to all the indexes. json file logs and all other logs are getting into both the indexes. both are using beats to send files. I don't want to use metadata because of required index names.

Preformatted text beats {
port => 5000
codec => "json_lines"
type => "iis"
}
}
output {
elasticsearch {
hosts => ["localhost:9200"]
index => "wazuh-alerts-3.x-%{+YYYY.MM.dd}"
document_type => "wazuh"
}
}

==========

input {

beats {
port => 5044
type => "iis"
}

}

filter {
....
}

output {
#if [type] == "iis" {
elasticsearch {
hosts => ["localhost:9200"]
index => "iis-logs-%{+YYYY.MM.dd}"
document_type => "iis"
}
}

2

If you use -f (or path.config) to point to a directory, then all of the files in that directory are concatenated. Events are read from all of the inputs, sent through the filters, and then written (unless there are conditionals) to all of the outputs.

If you want each configuration file to be independent then you would have to use multiple pipelines.

3

Here is my configuration: Can you please see if i am doing anything wrong, I am getting no data at all. Indexes are there but no data in them.

input {
beats {
port => 5000
codec => "json_lines"
}
}

input {

beats {
port => 5044
}

}

filter {
if [type] == "iis" {
grok {
match => ["message","%{TIMESTAMP_ISO8601:log_timestamp} "%{WORD:S_SiteName}" "%{NOTSPACE:S_ComputerName}" %{IPORHOST:S_IP} %{WORD:CS_Method} (?:-|%{URIPATH:CS_URI_Stem}) (?:-|%{GREEDYDATA:CS_URI_Query}) %{NUMBER:S_Port} (?:-|%{NOTSPACE:CS_Username}) (?:-|"%{IPORHOST:C_IP}") "%{NOTSPACE:CS_Version}" (?:-|%{QUOTEDSTRING:User_Agent}) (?:-|"%{NOTSPACE:CS_Referrer}") (?:-|"%{NOTSPACE:CS_Host}") %{NUMBER:response} %{NUMBER:subresponse} %{NUMBER:scstatus} %{NUMBER:SC_Bytes} %{NUMBER:CS_Bytes} %{NUMBER:time_taken} (?:-|%{IPORHOST:client_ip}) (?:-|"%{GREEDYDATA:cust_response_log}") (?:-|"%{NUMBER:request_count}") (?:-|"%{NOTSPACE:olx_user}") (?:-|"%{TIMESTAMP_ISO8601:request_start_time}") (?:-|"%{TIMESTAMP_ISO8601:request_end_time}") (?:-|"%{NUMBER:dotnet_time_taken}")"]
}
}
}

output {
if [type] == "iis" {
elasticsearch {
hosts => ["localhost:9200"]
index => "iis-logs-%{+YYYY.MM.dd}"
document_type => "iis"
}
}
else if [type] == "winlogbeat"
{
elasticsearch {
hosts => ["localhost:9200"]
index => "windows-events-%{+YYYY.MM.dd}"
document_type => "winlogbeat"
}
}
else if [type] == "wazuh"
{
elasticsearch {
hosts => ["localhost:9200"]
index => "wazuh-alerts-%{+YYYY.MM.dd}"
document_type => "wazuh"
}
}
}

4

Actually only IIS one is working.

5

If you use

output { stdout { codec => rubydebug } }

what do the winlogbeat and wazuh events look like?

6

here it is stdout for the wazuh

{
"timestamp" => "2019-03-08T19:33:53.192-0500",
"id" => "1552091633.2035411",
"location" => "netstat listening ports",
"@version" => "1",
"previous_output" => "Previous output:\nossec: output: 'netstat listening ports':\ntcp 0.0.0.0:22 0.0.0.0:* 6816/sshd\ntcp6 :::22 :::* 6816/sshd\ntcp 0.0.0.0:25 0.0.0.0:* 6641/master\ntcp6 :::25 :::* 6641/master\ntcp 0.0.0.0:80 0.0.0.0:* 9256/apache2\ntcp 0.0.0.0:111 0.0.0.0:* 449/rpcbind\ntcp6 :::111 :::* 449/rpcbind\nudp 0.0.0.0:111 0.0.0.0:* 449/rpcbind\nudp6 :::111 :::* 449/rpcbind\nudp 0.0.0.0:123 0.0.0.0:* 7763/ntpd\nudp 10.60.2.125:123 0.0.0.0:* 7763/ntpd\nudp 127.0.0.1:123 0.0.0.0:* 7763/ntpd\nudp6 ::1:123 :::* 7763/ntpd\nudp6 :::123 :::* 7763/ntpd\nudp6 fe80::250:56ff:fe01:123 :::* 7763/ntpd\ntcp 0.0.0.0:443 0.0.0.0:* 9256/apache2\nudp 0.0.0.0:625 0.0.0.0:* 449/rpcbind\nudp6 :::625 :::* 449/rpcbind\ntcp 0.0.0.0:3306 0.0.0.0:* 27718/mysqld\ntcp 0.0.0.0:5666 0.0.0.0:* 579/nrpe\ntcp6 :::5666 :::* 579/nrpe",
"agent" => {
"ip" => "10.60.2.125",
"id" => "006",
"name" => "Support"
},
"manager" => {
"name" => "wazuh"
},
"beat" => {
"name" => "wazuh",
"hostname" => "wazuh",
"version" => "6.6.0"
},
"host" => {
"name" => "wazuh"
},
"previous_log" => "ossec: output: 'netstat listening ports':\ntcp 0.0.0.0:22 0.0.0.0:* 6816/sshd\ntcp6 :::22 :::* 6816/sshd\ntcp 0.0.0.0:25 0.0.0.0:* 6641/master\ntcp6 :::25 :::* 6641/master\ntcp 0.0.0.0:80 0.0.0.0:* 9256/apache2\ntcp 0.0.0.0:111 0.0.0.0:* 449/rpcbind\ntcp6 :::111 :::* 449/rpcbind\nudp 0.0.0.0:111 0.0.0.0:* 449/rpcbind\nudp6 :::111 :::* 449/rpcbind\nudp 0.0.0.0:123 0.0.0.0:* 7763/ntpd\nudp 10.60.2.125:123 0.0.0.0:* 7763/ntpd\nudp 127.0.0.1:123 0.0.0.0:* 7763/ntpd\nudp6 ::1:123 :::* 7763/ntpd\nudp6 :::123 :::* 7763/ntpd\nudp6 fe80::250:56ff:fe01:123 :::* 7763/ntpd\ntcp 0.0.0.0:443 0.0.0.0:* 9256/apache2\nudp 0.0.0.0:625 0.0.0.0:* 449/rpcbind\nudp6 :::625 :::* 449/rpcbind\ntcp 0.0.0.0:3306 0.0.0.0:* 27718/mysqld\ntcp 0.0.0.0:5666 0.0.0.0:* 579/nrpe\ntcp6 :::5666 :::* 579/nrpe",
"source" => "/var/ossec/logs/alerts/alerts.json",
"full_log" => "ossec: output: 'netstat listening ports':\ntcp 0.0.0.0:22 0.0.0.0:* 6816/sshd\ntcp6 :::22 :::* 6816/sshd\ntcp 0.0.0.0:25 0.0.0.0:* 6641/master\ntcp6 :::25 :::* 6641/master\ntcp 0.0.0.0:80 0.0.0.0:* 10050/apache2\ntcp 0.0.0.0:111 0.0.0.0:* 449/rpcbind\ntcp6 :::111 :::* 449/rpcbind\nudp 0.0.0.0:111 0.0.0.0:* 449/rpcbind\nudp6 :::111 :::* 449/rpcbind\nudp 0.0.0.0:123 0.0.0.0:* 7763/ntpd\nudp 10.60.2.125:123 0.0.0.0:* 7763/ntpd\nudp 127.0.0.1:123 0.0.0.0:* 7763/ntpd\nudp6 ::1:123 :::* 7763/ntpd\nudp6 :::123 :::* 7763/ntpd\nudp6 fe80::250:56ff:fe01:123 :::* 7763/ntpd\ntcp 0.0.0.0:443 0.0.0.0:* 10050/apache2\nudp 0.0.0.0:625 0.0.0.0:* 449/rpcbind\nudp6 :::625 :::* 449/rpcbind\ntcp 0.0.0.0:3306 0.0.0.0:* 27718/mysqld\ntcp 0.0.0.0:5666 0.0.0.0:* 579/nrpe\ntcp6 :::5666 :::* 579/nrpe",
"@timestamp" => 2019-03-09T00:33:56.772Z,
"rule" => {
"id" => "533",
"mail" => true,
"gdpr" => [
[0] "IV_35.7.d"
],
"gpg13" => [
[0] "10.1"
],
"groups" => [
[0] "ossec"
],
"firedtimes" => 2,
"pci_dss" => [
[0] "10.2.7",
[1] "10.6.1"
],
"level" => 7,
"description" => "Listened ports status (netstat) changed (new port opened or closed)."
},
"decoder" => {
"name" => "ossec"
},
"offset" => 2693593,
"log" => "",
"tags" => [
[0] "beats_input_raw_event"
]
}

7

That does not have a [type] field containing wazuh. The only fields containing that are [manager][name], [beat][name], [beat][hostname], and [host][name].

I suggest you go back and check the beat config (and whether you are adding fields_under_root).

8

Pointed in the right direction. You saved the day. Respect ++

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.