Hi there guys,
I've been having this issue Packetbeat index not created and no info from Kibana , I cannot ingest the data sent by packetbeat into elasticsearch through logstash, I configure the plugin to connect directly to ES and it works, so I need you assistance.
Apart from that, maybe I'm hitting my head against the wall unnecessary, is there any advantage configuring the plugin to connect directly to ES instead of LS?
I've configured filebeat and topbeat and are working through LS, this is my configuration:
00-log.conf ( for sensu , was trying to configure metrics, still no luck )
input {
tcp {
port => 5514
codec => "json"
type => "sensu-logs"
}
}
01-beats-input.conf ( for the beats )
input {
beats {
port => 5044
congestion_threshold => "60"
ssl => true
ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt"
ssl_key => "/etc/pki/tls/private/logstash-forwarder.key"
}
}
10-syslog-filter.conf
filter {
if [type] == "syslog" {
grok {
match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:[%{POSINT:syslog_pid}])?: %{GREEDYDATA:syslog_message}" }
add_field => [ "received_at", "%{@timestamp}" ]
add_field => [ "received_from", "%{host}" ]
}
syslog_pri { }
date {
match => [ "syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]
}
}
}
11-sensu-filter.conf
filter {
if [type] == "sensu" {
date {
match => ["[check][issued]", "UNIX" ]
}
mutate {
remove_field => [ "host", "[client][handlers]", "[check][handlers]", "[check][history]", "[client][keepalive][handler]", "[client][keepalive][refresh]", "[client][keepalive][thresholds][critical]", "[client][keepalive][thresholds][warning]", "[client][subscriptions]", "[client][address]" ]
}
}
}
filter {
mutate {
add_field => { "event_id" => "%{[client][name]}%{[check][name]}%{[check][status]}" }
}
throttle {
after_count => 1
period => 86400
key => "%{event_id}"
add_tag => "throttled"
}
}
filter {
grok {
match => { "message" => "%{DATA:metric} %{DATA:value} %{INT:unixtime}" }
}
}
20-packetbeat-output.conf
output {
For debugging, remove later.
stdout { codec => rubydebug { metadata => true } }
If you need a conditional on the output you could use a tag. Don't use
type because it will be set to dns or http.
if "packetbeat" in [tags] {
elasticsearch {
hosts => "localhost:9200"
manage_template => false
index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
document_type => "%{[@metadata][type]}"
}
}
}
30-elasticsearch-output.conf
output {
elasticsearch {
hosts => "localhost:9200"
sniffing => true
manage_template => false
index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
document_type => "%{[@metadata][type]}"
}
}
Version numbers:
logstash-2.2.4-1.noarch
elasticsearch-2.4.0-1.noarch
Any help/advice really appreciated!!
Thanks for your time and support
Best regards