is there something wrong with this config :
input {
file {
type => "wazuh-alerts"
path => "/var/ossec/logs/alerts/alerts.json"
codec => "json"
}
beats {
port => 5044
type => "apache"
}
}
filter {
if [data][srcip] {
mutate {
add_field => [ "@src_ip", "%{[data][srcip]}" ]
}
}
if [data][aws][sourceIPAddress] {
mutate {
add_field => [ "@src_ip", "%{[data][aws][sourceIPAddress]}" ]
}
}
geoip {
source => "@src_ip"
target => "GeoLocation"
fields => ["city_name", "continent_code", "country_code2", "country_name", "region_name", "location"]
}
date {
match => ["timestamp", "ISO8601"]
target => "@timestamp"
}
mutate {
remove_field => [ "timestamp", "beat", "input_type", "tags", "count", "@version", "log", "offset", "type","@src_ip"]
}
}
output {
if [type] == "apache" {
elasticsearch {
hosts => ["localhost:9200"]
index => "apache.x-%{+YYYY.MM.dd}"
}
}
else {
elasticsearch {
hosts => ["localhost:9200"]
index => "wazuh-alerts-3.x-%{+YYYY.MM.dd}"
document_type => "wazuh"
}
}
}
What makes you think there's a problem with the configuration?
I can't generate index on elasticsearch with that config, if I remove the "if" I can but I get all my logs mixed up.
Have you looked in the Logstash log for clues? If it has problems sending to ES it'll tell you about it.
It gives any Erro just the pipeline started
Okay. And how do you know Logstash is getting any events to process and eventually send to ES?
I'm using Filebeat in a web server to get access.log, and ossec agent to get File integrety monitoring. When I use the normal config without the "if" I get all the logs and event but all in the index "apache", but when I add the "if" it stops.
But you're getting the events in the other index, right? What does an Apache event look like? Copy/paste from Kibana's JSON tab.
I got OSSEC logs in apache index :
{
"_index": "apache-2018.03.02",
"_type": "doc",
"_id": "sEYj5mEBaNtzX4YdlGze",
"_version": 1,
"_score": null,
"_source": {
"decoder": {
"name": "ossec"
},
"location": "netstat listening ports",
"path": "/var/ossec/logs/alerts/alerts.json",
"full_log": "ossec: output: 'netstat listening ports':\ntcp 0.0.0.0:22 0.0.0.0:* 875/sshd\ntcp6 :::22 :::* 875/sshd\nudp 0.0.0.0:68 0.0.0.0:* 5211/dhclient\nudp 0.0.0.0:1514 0.0.0.0:* 1223/ossec-remoted\ntcp6 :::5044 :::* 5704/java\ntcp 0.0.0.0:5601 0.0.0.0:* 643/node\nudp6 :::8236 :::* 5211/dhclient\ntcp6 127.0.0.1:9200 :::* 5478/java\ntcp6 ::1:9200 :::* 5478/java\ntcp6 127.0.0.1:9300 :::* 5478/java\ntcp6 ::1:9300 :::* 5478/java\ntcp6 127.0.0.1:9600 :::* 5704/java\nudp 0.0.0.0:20256 0.0.0.0:* 5211/dhclient\ntcp6 :::55000 :::* 870/node",
"previous_log": "ossec: output: 'netstat listening ports':\ntcp 0.0.0.0:22 0.0.0.0:* 875/sshd\ntcp6 :::22 :::* 875/sshd\nudp 0.0.0.0:68 0.0.0.0:* 5211/dhclient\nudp 0.0.0.0:1514 0.0.0.0:* 1223/ossec-remoted\ntcp6 :::5044 :::* 640/java\ntcp 0.0.0.0:5601 0.0.0.0:* 643/node\nudp6 :::8236 :::* 5211/dhclient\ntcp6 127.0.0.1:9200 :::* 868/java\ntcp6 ::1:9200 :::* 868/java\ntcp6 127.0.0.1:9300 :::* 868/java\ntcp6 ::1:9300 :::* 868/java\ntcp6 127.0.0.1:9600 :::* 640/java\nudp 0.0.0.0:20256 0.0.0.0:* 5211/dhclient\ntcp6 :::55000 :::* 870/node",
"rule": {
"description": "Listened ports status (netstat) changed (new port opened or closed).",
"id": "533",
"groups": [
"ossec",
"gpg13_10.1"
],
"mail": false,
"level": 7,
"firedtimes": 4,
"pci_dss": [
"10.2.7",
"10.6.1"
]
},
"id": "1519984545.30906",
"manager": {
"name": "localhost.localdomain"
},
"host": "localhost.localdomain",
"agent": {
"name": "localhost.localdomain",
"id": "000"
},
"previous_output": "ossec: output: 'netstat listening ports':\ntcp 0.0.0.0:22 0.0.0.0:* 875/sshd\ntcp6 :::22 :::* 875/sshd\nudp 0.0.0.0:68 0.0.0.0:* 5211/dhclient\nudp 0.0.0.0:1514 0.0.0.0:* 1223/ossec-remoted\ntcp6 :::5044 :::* 640/java\ntcp 0.0.0.0:5601 0.0.0.0:* 643/node\nudp6 :::8236 :::* 5211/dhclient\ntcp6 127.0.0.1:9200 :::* 868/java\ntcp6 ::1:9200 :::* 868/java\ntcp6 127.0.0.1:9300 :::* 868/java\ntcp6 ::1:9300 :::* 868/java\ntcp6 127.0.0.1:9600 :::* 640/java\nudp 0.0.0.0:20256 0.0.0.0:* 5211/dhclient\ntcp6 :::55000 :::* 870/node",
"predecoder": {
"hostname": "localhost"
},
"@timestamp": "2018-03-02T09:55:45.000Z"
},
"fields": {
"@timestamp": [
"2018-03-02T09:55:45.000Z"
]
},
"sort": [
1519984545000
]
}
When you comment out the if conditional? Yes, that's expected. I'm more interested in what happens with the conditional in place.
when I use the "if" I can't get any data
I think you're getting all events to the wazuh-alerts events. It's not likely that the presence of the conditional mean you get nothing. Again, what does an Apache event look like? Does it really have type set to "apache"
{
"_index": "wazuh-alerts-3.x-2018.03.02",
"_type": "wazuh",
"_id": "zEZS5mEBaNtzX4YdFWxK",
"_version": 1,
"_score": null,
"_source": {
"host": "localhost.localdomain",
"source": "/var/log/httpd/access_log",
"message": "192.168.112.1 - - [02/Mar/2018:10:46:46 +0000] "GET /noindex/css/fonts/Bold/OpenSans-Bold.ttf HTTP/1.1" 404 238 "http://192.168.112.176/noindex/css/open-sans.css" "Mozilla/5.0 (Windows NT 10.0; WOW64; rv:58.0) Gecko/20100101 Firefox/58.0"",
"prospector": {
"type": "log"
},
"@timestamp": "2018-03-02T10:46:52.141Z"
},
"fields": {
"@timestamp": [
"2018-03-02T10:46:52.141Z"
]
},
"sort": [
1519987612141
]
}
They have type Wazuh
The type field contains "log" so the
if [type] == "apache" {
conditional is never true.
Even if I used type => "apache" in the input ?
Apparently. Perhaps Filebeat sets it to "log" and the beats input doesn't overwrite the value? Since you might want to send other kinds of logs to the beats input you shouldn't set the type there anyway.
so the solution is to use "tags" ? but even if I put them in the input field I can't find them in the Json Table.
You should set the kind of events in the Filebeat configuration. You can either use tags or use fields (e.g. type) by using the fields option. In the latter case you should also make sure fields_under_root is set to true.
Ok thank you a lot, I used if [fields][log_type] == "access" and it works .
