Test on beat version doesn't work

Elastic Stack Logstash
1

Hello
Currently i have 2 versions of beat on my infra 5.5 and 6.4, my logstash version is 5.6 so compatible with beat client 5.5 and 6.4.
I try to detect the beat version of the document and redirect it to correct output. My document in 6.4 doesn't index in elasticsearch but 5.5 works .
if [beat][version] == "6.4.2" {
elasticsearch {
hosts => ["172.18.3.192:9200", "172.18.3.191:9200"]
ssl => true
ssl_certificate_verification => false
index => "logstash-syslog-hp-v6-%{+YYYY.MM}"
user => "xxxx"
password => "xxxx"
}
}
else {
elasticsearch {
hosts => ["172.18.3.192:9200", "172.18.3.191:9200"]
ssl => true
ssl_certificate_verification => false
index => "logstash-syslog-hp-%{+YYYY.MM}"
user => "xxxxx"
password => "xxxxx"
}
}

    }

Thanks in advance for your help i tried everything .

2

Try this

if [beat][version] == 6.4.2 {
3

Hello
Thanks for your fast reply, I tried but doesn't work. I wonder if the [beat][version] is correct, on elastic site they say [@metadata] [version] to access to beat version but it doesn't work too. I tried [@metadata][version] or [@metadata][beat][version] or [beat][version] , nothing work :frowning:

4

Can you try this:

if "6.4.2" in [beat][version] {
5

same issue .
bellow the output received by logstash :
[2018-11-26T12:17:07,378][DEBUG][logstash.pipeline ] output received {"event"=>{"appli_hostname"=>"FRCCEISEPT01", "syslog_severity_code"=>5, "offset"=>217996292, "syslog_facility"=>"user-level", "project"=>"infra", "syslog_facility_code"=>1, "source"=>"/var/log/messages", "message"=>"Nov 26 12:16:33 FRCCEISEPT01 journal: E1126 11:16:33.174102 1 authentication.go:62] Unable to authenticate the request due to an error: [x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kubernetes"), x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kubernetes")]", "env"=>"TST", "type"=>"log", "syslog_severity"=>"notice", "tags"=>["beats_input_codec_plain_applied", "v6"], "appli_timestamp"=>"Nov 26 12:16:33", "received_from"=>"{"name":"FRCCEISEPT01"}", "@timestamp"=>2018-11-26T11:16:33.000Z, "int1"=>1, "appli"=>["syslog", "syslog"], "appli_message"=>"E1126 11:16:33.174102 1 authentication.go:62] Unable to authenticate the request due to an error: [x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kubernetes"), x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kubernetes")]", "received_at"=>"2018-11-26T11:16:34.101Z", "@version"=>"1", "beat"=>{"name"=>"FRCCEISEPT01", "hostname"=>"FRCCEISEPT01", "version"=>"6.4.2"}, "host"=>{"name"=>"FRCCEISEPT01"}, "appli_program"=>"journal"}}

6

That's not the same issue, please read the error message, it does not like your certificate!

7

Hi in fact the the part about ssl certificates is the content of the messages field received by the client :

"message"=>;"Nov 26 12:16:33 FRCCEISEPT01 journal: E1126 11:16:33.174102 1 authentication.go:62] Unable to authenticate the request due to an error: [x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kubernetes"), x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kubernetes"

8

My apologies!

Can you please post your full config?

9

Sorry for late reply,
- filebeat client 6.4
filebeat.yml :
filebeat:
registry_file: /var/lib/filebeat/registry
config_dir: /etc/filebeat/conf.d
prospectors:
-
paths:
- /var/log/messages
- /var/log/secure
input_type: log
fields_under_root: true
fields:
project: infra
env: TST
appli: syslog
document_type: syslog
force_close_files: true

output:
logstash:
hosts: ["XXXXX:5044", "XXXXXXXX:5044"]
loadbalance: false
enable: true
ssl:
certificate_authorities: ["/etc/filebeat/logstash.crt"]
verification_mode: "none"

shipper:
logging:
level: error
to_syslog: false
to_files: true
files:
path: /var/log/filebeat
name: filebeat.log
rotateeverybytes: 99999999999
keepfiles: 2

**Logstash server 5.6 **
- config file :
input {
beats {
port => 5044
type => "log"
ssl => true
ssl_certificate => "/etc/logstash/logstash.crt"
ssl_key => "/etc/logstash/logstash.key"
ssl_verify_mode => "none"
}
}

filter {
grok {
match => { "message" => "%{SYSLOGTIMESTAMP:appli_timestamp} %{SYSLOGHOST:appli_hostname} %{DATA:appli_program}(?:[%{POSINT:appli_pid}])?: %{GREEDYDATA:appli_message}" }
add_field => [ "received_at", "%{@timestamp}" ]
add_field => [ "received_from", "%{host}" ]
tag_on_failure => ["soucis-grok"]
}
}
output {
if [@metadata][version] == "6.4.2" {
elasticsearch {
hosts => ["XXXXX:9200", "XXXX:9200"]
ssl => true
ssl_certificate_verification => false
index => "system-hp-%{+YYYY.MM}"
user => "XXXX"
password => "XXXX"
}
}
else {
elasticsearch {
hosts => ["172.18.3.192:9200", "172.18.3.191:9200"]
ssl => true
ssl_certificate_verification => false
index => "logstash-syslog-hp-%{+YYYY.MM}"
user => "XXX"
password => "XXX"
}
}
}
template file on elastic (5.6)
{
"system-hp": {
"order": 0,
"version": 50001,
"template": "system-hp-",
"settings": {
"index": {
"number_of_shards": "3",
"refresh_interval": "5s"
}
},
"mappings": {
"default": {
"dynamic_templates": [
{
"message_field": {
"path_match": "message",
"match_mapping_type": "string",
"mapping": {
"type": "text",
"norms": false
}
}
},
{
"string_fields": {
"match": "",
"match_mapping_type": "string",
"mapping": {
"type": "text",
"norms": false,
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
}
}
}
],
"properties": {
"@timestamp": {
"type": "date"
},
"@version": {
"type": "keyword"
}
}
}
},
"aliases": {}
}
}

The document that should go in logstash-syslog-hp-* are fine and works, but the index system-hp-* where filebeat client 6.4 should be indexed doesn't work no index created no document .

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.