# Malformed logstash message

**URL:** <https://discuss.elastic.co/t/malformed-logstash-message/138205>\
**Category:** Logstash\
**Created:** [July 2, 2018, 12:21pm UTC](https://discuss.elastic.co/t/malformed-logstash-message/138205 "2018-07-02T12:21:50Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![Labibme](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/labibme/32/31733_2.png) [@Labibme](https://discuss.elastic.co/u/Labibme)\
**Post date:** [July 2, 2018, 12:21pm UTC](https://discuss.elastic.co/t/malformed-logstash-message/138205/1 "2018-07-02T12:21:50Z")

</div>

Hello Everyone,

I'm working in a new logstash plateform built from kafka output, archtiecture is as shown below:

" filebeat -\> kafka -\> logstash ( first site SSL encryption) -\> logstash ( second site ssl decryption) -\> elasticsearch -\> kibana"

Issue, is shown between ( logstash and the second logstash) instance.

you can find below two différent messages for same "input" from kafka:

logstash (1)

> "message": " [Other: 0.5 ms]",

logstash (2)

> "caa-bloc": "%{[fields][caa-bloc]}",  
> "message": "2018-07-02T10:07:56.620Z {name=vl-a-rxx-56} [Other: 0.5 ms]"

and complete JSON message:

logstash (1)

> {  
> "\_index": "logs.caa.devrct.applications\_59\_2018.07.02",  
> "\_type": "doc",  
> "\_id": "FMSAWmQBF5mYg5ij5l8q",  
> "\_version": 1,  
> "\_score": 17.41422,  
> "\_source": {  
> "caa-bloc": "%{[fields][caa-bloc]}",  
> "caa-type": "application",  
> "input": {  
> "type": "log"  
> },  
> "host": {  
> "name": "vl-a-rxx-56"  
> },  
> "offset": 8543392,  
> "message": " [Other: 0.5 ms]",  
> "tags": [  
> "\_grokparsefailure"  
> ],  
> "prospector": {  
> "type": "log"  
> },  
> "caa-allocid": "%{[fields][caa-allocid]}",  
> "caa-srvip": "10.108.99.222",  
> "caa-env": "horsprod",  
> "source": "/apps/kafka/confluent-4.0.0/logs/kafkaServer-gc.log.0.current",  
> "@version": "1",  
> "beat": {  
> "name": "vl-a-rxx-56",  
> "version": "6.3.0",  
> "hostname": "vl-a-rxx-56"  
> },  
> "log\_topic": "logs.caa.devrct.applications",  
> "caa-module": "kafka",  
> "caa-image": "%{[fields][caa-image]}",  
> "topic": "logs.caa.devrct.applications",  
> "fields": {  
> "caa-type": "application",  
> "caa-env": "horsprod",  
> "log\_topic": "logs.caa.devrct.applications",  
> "caa-module": "kafka",  
> "caa-srvip": "10.108.99.222",  
> "caa-host": "vl-a-rxx-56"  
> },  
> "timestamp": "%{year}-%{month}-%{day} %{time}",  
> "caa-type2": "%{[fields][caa-type2]}",  
> "@timestamp": "2018-07-02T10:19:12.060Z",  
> "caa-host": "vl-a-rxx-56"  
> },  
> "fields": {  
> "@timestamp": [  
> "2018-07-02T10:19:12.060Z"  
> ]  
> },  
> "highlight": {  
> "beat.hostname": [  
> "@kibana-highlighted-field@vl@/kibana-highlighted-field@-@kibana-highlighted-field@a@/kibana-highlighted-field@-@kibana-highlighted-field@rxx@/kibana-highlighted-field@-@kibana-highlighted-field@56@/kibana-highlighted-field@"  
> ],  
> "message": [  
> "[@kibana-highlighted-field@Other@/kibana-highlighted-field@: @kibana-highlighted-field@0.5@/kibana-highlighted-field@ @kibana-highlighted-field@ms@/kibana-highlighted-field@]"  
> ]  
> }  
> }

logstash (2):

> {  
> "\_index": "amlooser\_60\_2018.07.02",  
> "\_type": "doc",  
> "\_id": "QMOAWmQBF5mYg5ijtf2D",  
> "\_version": 1,  
> "\_score": 2.0808823,  
> "\_source": {  
> "caa-env": "%{[fields][caa-env]}",  
> "caa-type2": "%{[fields][caa-type2]}",  
> "@version": "1",  
> "topic": "%{[fields][log\_topic]}",  
> "caa-srvip": "%{[fields][caa-srvip]}",  
> "@timestamp": "2018-07-02T10:19:01.740Z",  
> "timestamp": "%{year}-%{month}-%{day} %{time}",  
> "caa-type": "%{[fields][caa-type]}",  
> "caa-allocid": "%{[fields][caa-allocid]}",  
> "caa-module": "%{[fields][caa-module]}",  
> "caa-image": "%{[fields][caa-image]}",  
> "caa-host": "%{[fields][caa-host]}",  
> "log\_topic": "%{[fields][log\_topic]}",  
> "tags": [  
> "beats\_input\_codec\_plain\_applied",  
> "\_jsonparsefailure",  
> "\_grokparsefailure"  
> ],  
> "caa-bloc": "%{[fields][caa-bloc]}",  
> "message": "2018-07-02T10:07:56.620Z {name=vl-a-rxx-56} [Other: 0.5 ms]"  
> },  
> "fields": {  
> "@timestamp": [  
> "2018-07-02T10:19:01.740Z"  
> ]  
> },  
> "highlight": {  
> "message": [  
> "2018-07-02T10:07:56.620Z {name=@kibana-highlighted-field@vl@/kibana-highlighted-field@-@kibana-highlighted-field@a@/kibana-highlighted-field@-@kibana-highlighted-field@rxx@/kibana-highlighted-field@-@kibana-highlighted-field@56@/kibana-highlighted-field@} [Other: 0.5 ms]"  
> ]  
> }  
> }

Thanks for your participation!

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [July 2, 2018, 12:35pm UTC](https://discuss.elastic.co/t/malformed-logstash-message/138205/2 "2018-07-02T12:35:25Z")

</div>

How are these two Logstash instances configured?

---

<div class="post-metadata">

**Author:** ![Labibme](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/labibme/32/31733_2.png) [@Labibme](https://discuss.elastic.co/u/Labibme)\
**Post date:** [July 2, 2018, 12:40pm UTC](https://discuss.elastic.co/t/malformed-logstash-message/138205/3 "2018-07-02T12:40:40Z")

</div>

Hello magnusbaeck,

Thanks you for your fast reply, here's the reply to your question:

- version used: "6.3.0" and "6.4.0"

standalone logstash; ( working well):

> output {  
> stdout {  
> codec =\> "json"  
> }  
> if ([log\_topic] == "logs.caa.devrct.applications") {  
> elasticsearch {  
> hosts =\> ["vl-a-rxx-60:9200"]  
> index =\> "logs.caa.devrct.applications\_59\_%{+YYYY.MM.dd}"  
> }  
> }  
> else if ([log\_topic] == "logs.caa.devrct.os") {  
> elasticsearch {  
> hosts =\> ["vl-a-rxx-60:9200"]  
> index =\> "logs.caa.devrct.os\_59\_%{+YYYY.MM.dd}"  
> }  
> }  
> else if ([log\_topic] == "[logs.caa.devrct.net](http://logs.caa.devrct.net)") {  
> elasticsearch {  
> hosts =\> ["vl-a-rxx-60:9200"]  
> index =\> "logs.caa.devrct.net\_59\_%{+YYYY.MM.dd}"  
> }  
> }  
> else if ([log\_topic] == "logs.caa.devrct.middlewares") {  
> elasticsearch {  
> hosts =\> ["vl-a-rxx-60:9200"]  
> index =\> "logs.caa.pprod.middlewares\_59\_%{+YYYY.MM.dd}"  
> }  
> }  
> else {  
> elasticsearch {  
> hosts =\> ["vl-a-rxx-60:9200"]  
> index =\> "amlooser\_59\_%{+YYYY.MM.dd}"  
> }
> 
> ```
> }
> 
> ```
> 
> }

Two logstash connected:  
logstash (1):

> stdout {  
> codec =\> "json"  
> }  
> lumberjack  
> {  
> hosts =\> "vl-a-rxx-60"  
> port =\> 5002  
> ssl\_certificate =\> "/apps/logstash/logstash-6.2.3/config/conf.d/lumberjack.crt"
> 
> ```
> }
> 
> ```

logstash (2):

> input {  
> beats  
> {  
> port =\> 5002  
> client\_inactivity\_timeout =\> 1200  
> ssl =\> true  
> ssl\_certificate =\> "/etc/logstash/conf.d/lumberjack.crt"  
> ssl\_key =\> "/etc/logstash/conf.d/lumberjack.key"  
> }
> 
> ```
> }
> 
> ```

let me know if you want full configuration.  
thanks again,

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [July 2, 2018, 1:36pm UTC](https://discuss.elastic.co/t/malformed-logstash-message/138205/4 "2018-07-02T13:36:46Z")

</div>

Use a lumberjack input to receive data from a lumberjack output, not a beats input.

I'm a bit surprised your current configuration even works.

---

<div class="post-metadata">

**Author:** ![Labibme](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/labibme/32/31733_2.png) [@Labibme](https://discuss.elastic.co/u/Labibme)\
**Post date:** [July 2, 2018, 2:14pm UTC](https://discuss.elastic.co/t/malformed-logstash-message/138205/5 "2018-07-02T14:14:34Z")

</div>

Okay, but i already used lumberjack before with "6.3.0" and it was with same result.  
Message still malformed, but with actual version i can't find "input lumberjack" with offline package.

If you can point me to right link for "lumberjack input plugin" for "6.4.0" it will be a pleasure.

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [July 2, 2018, 2:32pm UTC](https://discuss.elastic.co/t/malformed-logstash-message/138205/6 "2018-07-02T14:32:23Z")

</div>

> If you can point me to right link for "lumberjack input plugin" for "6.4.0" it will be a pleasure.

You can't install it with the `logstash-plugin` command? Then I don't know.

---

<div class="post-metadata">

**Author:** ![Labibme](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/labibme/32/31733_2.png) [@Labibme](https://discuss.elastic.co/u/Labibme)\
**Post date:** [July 2, 2018, 2:35pm UTC](https://discuss.elastic.co/t/malformed-logstash-message/138205/7 "2018-07-02T14:35:31Z")

</div>

Yes, logstash-plugin command use online mode or to package existing one or to create a zip or tgz one.  
Visibly there's no lumberjack for newer version of logstash, all files i found in the internet was for older version "2.X" and i can't find the.

At all, same version that was installed in "rpm" i reinstall it and there's no lumberjack detected....

---

<div class="post-metadata">

**Author:** ![Labibme](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/labibme/32/31733_2.png) [@Labibme](https://discuss.elastic.co/u/Labibme)\
**Post date:** [July 2, 2018, 4:52pm UTC](https://discuss.elastic.co/t/malformed-logstash-message/138205/8 "2018-07-02T16:52:42Z")

</div>

I tried, "tcp" plugin in input and output:

output

> tcp  
> {  
> hosts =\> "vl-a-rxx-60"  
> port =\> 5000  
> codec =\> json  
> }

input

> tcp  
> {  
> port =\> 5000  
> codec =\> json  
> }

and result is same:

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 30, 2018, 4:52pm UTC](https://discuss.elastic.co/t/malformed-logstash-message/138205/9 "2018-07-30T16:52:46Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.