Malformed logstash message

Elastic Stack Logstash
1

Hello Everyone,

I'm working in a new logstash plateform built from kafka output, archtiecture is as shown below:

" filebeat -> kafka -> logstash ( first site SSL encryption) -> logstash ( second site ssl decryption) -> elasticsearch -> kibana"

Issue, is shown between ( logstash and the second logstash) instance.

you can find below two différent messages for same "input" from kafka:

logstash (1)

"message": " [Other: 0.5 ms]",

logstash (2)

"caa-bloc": "%{[fields][caa-bloc]}",
"message": "2018-07-02T10:07:56.620Z {name=vl-a-rxx-56} [Other: 0.5 ms]"

and complete JSON message:

logstash (1)

{
"_index": "logs.caa.devrct.applications_59_2018.07.02",
"_type": "doc",
"_id": "FMSAWmQBF5mYg5ij5l8q",
"_version": 1,
"_score": 17.41422,
"_source": {
"caa-bloc": "%{[fields][caa-bloc]}",
"caa-type": "application",
"input": {
"type": "log"
},
"host": {
"name": "vl-a-rxx-56"
},
"offset": 8543392,
"message": " [Other: 0.5 ms]",
"tags": [
"_grokparsefailure"
],
"prospector": {
"type": "log"
},
"caa-allocid": "%{[fields][caa-allocid]}",
"caa-srvip": "10.108.99.222",
"caa-env": "horsprod",
"source": "/apps/kafka/confluent-4.0.0/logs/kafkaServer-gc.log.0.current",
"@version": "1",
"beat": {
"name": "vl-a-rxx-56",
"version": "6.3.0",
"hostname": "vl-a-rxx-56"
},
"log_topic": "logs.caa.devrct.applications",
"caa-module": "kafka",
"caa-image": "%{[fields][caa-image]}",
"topic": "logs.caa.devrct.applications",
"fields": {
"caa-type": "application",
"caa-env": "horsprod",
"log_topic": "logs.caa.devrct.applications",
"caa-module": "kafka",
"caa-srvip": "10.108.99.222",
"caa-host": "vl-a-rxx-56"
},
"timestamp": "%{year}-%{month}-%{day} %{time}",
"caa-type2": "%{[fields][caa-type2]}",
"@timestamp": "2018-07-02T10:19:12.060Z",
"caa-host": "vl-a-rxx-56"
},
"fields": {
"@timestamp": [
"2018-07-02T10:19:12.060Z"
]
},
"highlight": {
"beat.hostname": [
"@kibana-highlighted-field@vl@/kibana-highlighted-field@-@kibana-highlighted-field@a@/kibana-highlighted-field@-@kibana-highlighted-field@rxx@/kibana-highlighted-field@-@kibana-highlighted-field@56@/kibana-highlighted-field@"
],
"message": [
"[@kibana-highlighted-field@Other@/kibana-highlighted-field@: @kibana-highlighted-field@0.5@/kibana-highlighted-field@ @kibana-highlighted-field@ms@/kibana-highlighted-field@]"
]
}
}

logstash (2):

{
"_index": "amlooser_60_2018.07.02",
"_type": "doc",
"_id": "QMOAWmQBF5mYg5ijtf2D",
"_version": 1,
"_score": 2.0808823,
"_source": {
"caa-env": "%{[fields][caa-env]}",
"caa-type2": "%{[fields][caa-type2]}",
"@version": "1",
"topic": "%{[fields][log_topic]}",
"caa-srvip": "%{[fields][caa-srvip]}",
"@timestamp": "2018-07-02T10:19:01.740Z",
"timestamp": "%{year}-%{month}-%{day} %{time}",
"caa-type": "%{[fields][caa-type]}",
"caa-allocid": "%{[fields][caa-allocid]}",
"caa-module": "%{[fields][caa-module]}",
"caa-image": "%{[fields][caa-image]}",
"caa-host": "%{[fields][caa-host]}",
"log_topic": "%{[fields][log_topic]}",
"tags": [
"beats_input_codec_plain_applied",
"_jsonparsefailure",
"_grokparsefailure"
],
"caa-bloc": "%{[fields][caa-bloc]}",
"message": "2018-07-02T10:07:56.620Z {name=vl-a-rxx-56} [Other: 0.5 ms]"
},
"fields": {
"@timestamp": [
"2018-07-02T10:19:01.740Z"
]
},
"highlight": {
"message": [
"2018-07-02T10:07:56.620Z {name=@kibana-highlighted-field@vl@/kibana-highlighted-field@-@kibana-highlighted-field@a@/kibana-highlighted-field@-@kibana-highlighted-field@rxx@/kibana-highlighted-field@-@kibana-highlighted-field@56@/kibana-highlighted-field@} [Other: 0.5 ms]"
]
}
}

Thanks for your participation!

2

How are these two Logstash instances configured?

3

Hello magnusbaeck,

Thanks you for your fast reply, here's the reply to your question:

  • version used: "6.3.0" and "6.4.0"

standalone logstash; ( working well):

output {
stdout {
codec => "json"
}
if ([log_topic] == "logs.caa.devrct.applications") {
elasticsearch {
hosts => ["vl-a-rxx-60:9200"]
index => "logs.caa.devrct.applications_59_%{+YYYY.MM.dd}"
}
}
else if ([log_topic] == "logs.caa.devrct.os") {
elasticsearch {
hosts => ["vl-a-rxx-60:9200"]
index => "logs.caa.devrct.os_59_%{+YYYY.MM.dd}"
}
}
else if ([log_topic] == "logs.caa.devrct.net") {
elasticsearch {
hosts => ["vl-a-rxx-60:9200"]
index => "logs.caa.devrct.net_59_%{+YYYY.MM.dd}"
}
}
else if ([log_topic] == "logs.caa.devrct.middlewares") {
elasticsearch {
hosts => ["vl-a-rxx-60:9200"]
index => "logs.caa.pprod.middlewares_59_%{+YYYY.MM.dd}"
}
}
else {
elasticsearch {
hosts => ["vl-a-rxx-60:9200"]
index => "amlooser_59_%{+YYYY.MM.dd}"
}

    }

}

Two logstash connected:
logstash (1):

stdout {
codec => "json"
}
lumberjack
{
hosts => "vl-a-rxx-60"
port => 5002
ssl_certificate => "/apps/logstash/logstash-6.2.3/config/conf.d/lumberjack.crt"

            }

logstash (2):

input {
beats
{
port => 5002
client_inactivity_timeout => 1200
ssl => true
ssl_certificate => "/etc/logstash/conf.d/lumberjack.crt"
ssl_key => "/etc/logstash/conf.d/lumberjack.key"
}

    }

let me know if you want full configuration.
thanks again,

4

Use a lumberjack input to receive data from a lumberjack output, not a beats input.

I'm a bit surprised your current configuration even works.

5

Okay, but i already used lumberjack before with "6.3.0" and it was with same result.
Message still malformed, but with actual version i can't find "input lumberjack" with offline package.

If you can point me to right link for "lumberjack input plugin" for "6.4.0" it will be a pleasure.

6

If you can point me to right link for "lumberjack input plugin" for "6.4.0" it will be a pleasure.

You can't install it with the logstash-plugin command? Then I don't know.

7

Yes, logstash-plugin command use online mode or to package existing one or to create a zip or tgz one.
Visibly there's no lumberjack for newer version of logstash, all files i found in the internet was for older version "2.X" and i can't find the.

At all, same version that was installed in "rpm" i reinstall it and there's no lumberjack detected....

8

I tried, "tcp" plugin in input and output:

output

tcp
{
hosts => "vl-a-rxx-60"
port => 5000
codec => json
}

input

tcp
{
port => 5000
codec => json
}

and result is same:

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.