# Logstash json in text message

**URL:** <https://discuss.elastic.co/t/logstash-json-in-text-message/97285>\
**Category:** Logstash\
**Created:** [August 16, 2017, 3:00pm UTC](https://discuss.elastic.co/t/logstash-json-in-text-message/97285 "2017-08-16T15:00:35Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![annaK](https://avatars.discourse-cdn.com/v4/letter/a/e8c25b/32.png) [@annaK](https://discuss.elastic.co/u/annaK)\
**Post date:** [August 16, 2017, 3:00pm UTC](https://discuss.elastic.co/t/logstash-json-in-text-message/97285/1 "2017-08-16T15:00:35Z")

</div>

Hi all 🙂

I have a log line that looks like this:  
`time='2017-08-16 12:39:43.22253194 +0000 UTC' container_name='/r-stack' source='stdout' data='{"name":"stack","hostname":"bc202dc10bc9","pid":28,"level":50,"msg":"this is a message","time":"2017-08-16T12:39:43.106Z","v":0}'`

I first use a grok to match time, container name, source and data.  
In data there is the json string. But how can I tell logstash this is a json string which should be interpreted as json?

I tried it with json\_encode, but I am not able to install it:  
./logstash/bin/logstash-plugin install logstash-filter-json\_encode results in:

> Picked up \_JAVA\_OPTIONS: -Djava.net.preferIPv4Stack=true  
> Validating logstash-filter-json\_encode  
> Installing logstash-filter-json\_encode  
> Picked up \_JAVA\_OPTIONS: -Djava.net.preferIPv4Stack=true  
> Error Bundler::InstallError, retrying 1/10  
> An error occurred while installing logstash-core-event-java (5.2.2), and Bundler cannot continue.  
> Make sure that `gem install logstash-core-event-java -v '5.2.2'` succeeds before bundling.  
> WARNING: SSLSocket#session= is not supported

and trying to gem install logstash-core-event-java -v '5.2.2' results in

> ERROR: Could not find a valid gem 'logstash-core-event-java' (= 5.2.2), here is why:  
> Found logstash-core-event-java (5.2.2), but was for platform java

Can you help me on how to proceed from here? Do I really need the json\_encode or is there another possibility?  
I am using logstash 5.2.2

Thanks in advance 🙂

Regards,  
Anna

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [August 17, 2017, 5:15am UTC](https://discuss.elastic.co/t/logstash-json-in-text-message/97285/2 "2017-08-17T05:15:43Z")

</div>

Use a kv filter to parse the line (no need to use grok), then apply a json filter (not json\_encode) to the resulting `data` field.

---

<div class="post-metadata">

**Author:** ![annaK](https://avatars.discourse-cdn.com/v4/letter/a/e8c25b/32.png) [@annaK](https://discuss.elastic.co/u/annaK)\
**Post date:** [August 17, 2017, 6:48am UTC](https://discuss.elastic.co/t/logstash-json-in-text-message/97285/3 "2017-08-17T06:48:52Z")

</div>

Hi Magnus,

thanks for your reply.  
I changed my config file use the kv filter and it works like a charm for my example I provided 🙂

But in production I have a data string that counts about 600 characters. Can I tell logstash to fully parse this data line? Because it does not get fully parsed and therefore I get a jsonparsefailure and the exception:  
`:exception=>#<LogStash::Json::ParserError: Unexpected end-of-input in VALUE_STRING`

Regards,  
Anna

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [August 17, 2017, 8:37am UTC](https://discuss.elastic.co/t/logstash-json-in-text-message/97285/4 "2017-08-17T08:37:40Z")

</div>

Can't really help without seeing the data.

---

<div class="post-metadata">

**Author:** ![annaK](https://avatars.discourse-cdn.com/v4/letter/a/e8c25b/32.png) [@annaK](https://discuss.elastic.co/u/annaK)\
**Post date:** [August 17, 2017, 8:53am UTC](https://discuss.elastic.co/t/logstash-json-in-text-message/97285/5 "2017-08-17T08:53:50Z")

</div>

This is a example data:  
`time='2017-08-16 12:39:43.22253194 +0000 UTC' container_name='/r-stack-messenger' source='stdout' data='{"name":"service","hostname":"hostname1","pid":28,"level":50,"msg":"no-kafka-client Metadata request failed: NoKafkaConnectionError [kafka.test.com:9093]: Error: connect ECONNREFUSED 172.26.190.134:9093\n{ [NoKafkaConnectionError: Error: connect ECONNREFUSED 172.26.190.134:9093]\n name: 'NoKafkaConnectionError',\n server: 'kafka.test.com:9093',\n message: 'Error: connect ECONNREFUSED 123.45.67.890:9093' } \n[at apply (/app/node_modules/no-kafka/node_modules/lodash/lodash.js:482:27)]","time":"2017-08-16T12:39:43.106Z","v":0}'`

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [August 17, 2017, 9:19am UTC](https://discuss.elastic.co/t/logstash-json-in-text-message/97285/6 "2017-08-17T09:19:16Z")

</div>

The kv filter fails because of the single quotes inside the `data` value. If `data` always is the last key=value pair you can use a grok filter to extract everything that comes after "data=" to a field so that the whole string can be sent to the json filter.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [September 14, 2017, 9:19am UTC](https://discuss.elastic.co/t/logstash-json-in-text-message/97285/7 "2017-09-14T09:19:33Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
