Is there a mistake?


(Bakkali Amine) #1

is there something wrong with this config :
input {
file {
type => "wazuh-alerts"
path => "/var/ossec/logs/alerts/alerts.json"
codec => "json"

}
beats {

    port => 5044
    type => "apache"

}
}
filter {
if [data][srcip] {
mutate {
add_field => [ "@src_ip", "%{[data][srcip]}" ]
}
}
if [data][aws][sourceIPAddress] {
mutate {
add_field => [ "@src_ip", "%{[data][aws][sourceIPAddress]}" ]
}
}
geoip {
source => "@src_ip"
target => "GeoLocation"
fields => ["city_name", "continent_code", "country_code2", "country_name", "region_name", "location"]
}
date {
match => ["timestamp", "ISO8601"]
target => "@timestamp"
}
mutate {
remove_field => [ "timestamp", "beat", "input_type", "tags", "count", "@version", "log", "offset", "type","@src_ip"]
}
}
output {
if [type] == "apache" {
elasticsearch {
hosts => ["localhost:9200"]
index => "apache.x-%{+YYYY.MM.dd}"
}
}
else {
elasticsearch {
hosts => ["localhost:9200"]
index => "wazuh-alerts-3.x-%{+YYYY.MM.dd}"
document_type => "wazuh"

}
}
}


(Magnus Bäck) #2

What makes you think there's a problem with the configuration?


(Bakkali Amine) #3

I can't generate index on elasticsearch with that config, if I remove the "if" I can but I get all my logs mixed up.


(Magnus Bäck) #4

Have you looked in the Logstash log for clues? If it has problems sending to ES it'll tell you about it.


(Bakkali Amine) #5

It gives any Erro just the pipeline started


(Bakkali Amine) #6


(Magnus Bäck) #7

Okay. And how do you know Logstash is getting any events to process and eventually send to ES?


(Bakkali Amine) #8

I'm using Filebeat in a web server to get access.log, and ossec agent to get File integrety monitoring. When I use the normal config without the "if" I get all the logs and event but all in the index "apache", but when I add the "if" it stops.


(Magnus Bäck) #9

But you're getting the events in the other index, right? What does an Apache event look like? Copy/paste from Kibana's JSON tab.


(Bakkali Amine) #10

I got OSSEC logs in apache index :
{
"_index": "apache-2018.03.02",
"_type": "doc",
"_id": "sEYj5mEBaNtzX4YdlGze",
"_version": 1,
"_score": null,
"_source": {
"decoder": {
"name": "ossec"
},
"location": "netstat listening ports",
"path": "/var/ossec/logs/alerts/alerts.json",
"full_log": "ossec: output: 'netstat listening ports':\ntcp 0.0.0.0:22 0.0.0.0:* 875/sshd\ntcp6 :::22 :::* 875/sshd\nudp 0.0.0.0:68 0.0.0.0:* 5211/dhclient\nudp 0.0.0.0:1514 0.0.0.0:* 1223/ossec-remoted\ntcp6 :::5044 :::* 5704/java\ntcp 0.0.0.0:5601 0.0.0.0:* 643/node\nudp6 :::8236 :::* 5211/dhclient\ntcp6 127.0.0.1:9200 :::* 5478/java\ntcp6 ::1:9200 :::* 5478/java\ntcp6 127.0.0.1:9300 :::* 5478/java\ntcp6 ::1:9300 :::* 5478/java\ntcp6 127.0.0.1:9600 :::* 5704/java\nudp 0.0.0.0:20256 0.0.0.0:* 5211/dhclient\ntcp6 :::55000 :::* 870/node",
"previous_log": "ossec: output: 'netstat listening ports':\ntcp 0.0.0.0:22 0.0.0.0:* 875/sshd\ntcp6 :::22 :::* 875/sshd\nudp 0.0.0.0:68 0.0.0.0:* 5211/dhclient\nudp 0.0.0.0:1514 0.0.0.0:* 1223/ossec-remoted\ntcp6 :::5044 :::* 640/java\ntcp 0.0.0.0:5601 0.0.0.0:* 643/node\nudp6 :::8236 :::* 5211/dhclient\ntcp6 127.0.0.1:9200 :::* 868/java\ntcp6 ::1:9200 :::* 868/java\ntcp6 127.0.0.1:9300 :::* 868/java\ntcp6 ::1:9300 :::* 868/java\ntcp6 127.0.0.1:9600 :::* 640/java\nudp 0.0.0.0:20256 0.0.0.0:* 5211/dhclient\ntcp6 :::55000 :::* 870/node",
"rule": {
"description": "Listened ports status (netstat) changed (new port opened or closed).",
"id": "533",
"groups": [
"ossec",
"gpg13_10.1"
],
"mail": false,
"level": 7,
"firedtimes": 4,
"pci_dss": [
"10.2.7",
"10.6.1"
]
},
"id": "1519984545.30906",
"manager": {
"name": "localhost.localdomain"
},
"host": "localhost.localdomain",
"agent": {
"name": "localhost.localdomain",
"id": "000"
},
"previous_output": "ossec: output: 'netstat listening ports':\ntcp 0.0.0.0:22 0.0.0.0:* 875/sshd\ntcp6 :::22 :::* 875/sshd\nudp 0.0.0.0:68 0.0.0.0:* 5211/dhclient\nudp 0.0.0.0:1514 0.0.0.0:* 1223/ossec-remoted\ntcp6 :::5044 :::* 640/java\ntcp 0.0.0.0:5601 0.0.0.0:* 643/node\nudp6 :::8236 :::* 5211/dhclient\ntcp6 127.0.0.1:9200 :::* 868/java\ntcp6 ::1:9200 :::* 868/java\ntcp6 127.0.0.1:9300 :::* 868/java\ntcp6 ::1:9300 :::* 868/java\ntcp6 127.0.0.1:9600 :::* 640/java\nudp 0.0.0.0:20256 0.0.0.0:* 5211/dhclient\ntcp6 :::55000 :::* 870/node",
"predecoder": {
"hostname": "localhost"
},
"@timestamp": "2018-03-02T09:55:45.000Z"
},
"fields": {
"@timestamp": [
"2018-03-02T09:55:45.000Z"
]
},
"sort": [
1519984545000
]
}


(Magnus Bäck) #11

When you comment out the if conditional? Yes, that's expected. I'm more interested in what happens with the conditional in place.


(Bakkali Amine) #12

when I use the "if" I can't get any data


(Magnus Bäck) #13

I think you're getting all events to the wazuh-alerts events. It's not likely that the presence of the conditional mean you get nothing. Again, what does an Apache event look like? Does it really have type set to "apache"


(Bakkali Amine) #14

{
"_index": "wazuh-alerts-3.x-2018.03.02",
"_type": "wazuh",
"_id": "zEZS5mEBaNtzX4YdFWxK",
"_version": 1,
"_score": null,
"_source": {
"host": "localhost.localdomain",
"source": "/var/log/httpd/access_log",
"message": "192.168.112.1 - - [02/Mar/2018:10:46:46 +0000] "GET /noindex/css/fonts/Bold/OpenSans-Bold.ttf HTTP/1.1" 404 238 "http://192.168.112.176/noindex/css/open-sans.css" "Mozilla/5.0 (Windows NT 10.0; WOW64; rv:58.0) Gecko/20100101 Firefox/58.0"",
"prospector": {
"type": "log"
},
"@timestamp": "2018-03-02T10:46:52.141Z"
},
"fields": {
"@timestamp": [
"2018-03-02T10:46:52.141Z"
]
},
"sort": [
1519987612141
]
}
They have type Wazuh


(Magnus Bäck) #15

The type field contains "log" so the

if [type] == "apache" {

conditional is never true.


(Bakkali Amine) #16

Even if I used type => "apache" in the input ?


(Magnus Bäck) #17

Apparently. Perhaps Filebeat sets it to "log" and the beats input doesn't overwrite the value? Since you might want to send other kinds of logs to the beats input you shouldn't set the type there anyway.


(Bakkali Amine) #18

so the solution is to use "tags" ? but even if I put them in the input field I can't find them in the Json Table.


(Magnus Bäck) #19

You should set the kind of events in the Filebeat configuration. You can either use tags or use fields (e.g. type) by using the fields option. In the latter case you should also make sure fields_under_root is set to true.


(Bakkali Amine) #20

Ok thank you a lot, I used if [fields][log_type] == "access" and it works .