# Json is storing as string in es **URL:** \ **Category:** Logstash\ **Created:** [July 13, 2017, 5:25pm UTC](https://discuss.elastic.co/t/json-is-storing-as-string-in-es/93062 "2017-07-13T17:25:09Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![Narayanan\_Sukumar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/narayanan_sukumar/32/19546_2.png) [@Narayanan\_Sukumar](https://discuss.elastic.co/u/Narayanan_Sukumar)\ **Post date:** [July 13, 2017, 5:25pm UTC](https://discuss.elastic.co/t/json-is-storing-as-string-in-es/93062/1 "2017-07-13T17:25:10Z")
ELK I created is filebeat - \> KAFKA -\> logstash -\> ES &Kibana (AWS elastic search) the logs are generated in json format and it is pushed to kafka with below config: ========================= filebeat.prospectors: - input\_type: log paths: - /srv/my\_app/src/logs/\*.log document\_type: my\_app path.home: /opt/filebeat-5.4.2-linux-x86\_64/ path.config: {path.home} path.data: {path.home}/data filebeat.registry\_file: ${path.data}/registry logging.level: debug logging.to\_files: true logging.to\_syslog: false logging.files: path: /var/log/mybeat name: mybeat.log keepfiles: 7 output.kafka: enabled: true hosts: ["myhost.kafka1","myhostkafka2","myhostkafka3"] topic: my\_app version: 0.9.0.0 worker: 2 compression: gzip logging.to\_files: true logging.files: /tmp/filebeat\_kafka.log ============================================ # Logstash config is below input { # kafka { bootstrap\_servers =\> "myhost.kafka1","myhostkafka2","myhostkafka3" topics =\> ["my\_app"] codec =\> json } output { # if [type\_name] == "my\_app" { elasticsearch { hosts =\> ["eshost1"] index =\> "%{[type\_name]}-%{+YYYY.MM.dd}" document\_type =\> "%{[type\_name]}" } } } filter { ``` mutate { add_field => {"index_name" => '%{type}'} add_field => {"type_name" => '%{type}'} } if [@metadata][type] == "my_app" { json { source => "message" } date { match => [time ,"ISO8601"] } } ``` # } # The issue here is the message field is in json format and it is stored in es as string (I can see that through kibana) image attached below ![](https://us1.discourse-cdn.com/elastic/original/3X/c/b/cbbeea4079979dcab3ae84192fb8132ddac74f31.png) # Sample format of message: {"name":"xxxx","vertical":"xxxx","clusterid":50,"hostname":"xxxx","pid":7833,"route":"xxxxx","uuid":"xxxx","level":30,"opened\_from":"xxx","osVersion":"5.1.1","imei":"xxxxxx","networkType":"4G","language":"en","long":"81.4510731","lat":"25.748469","playStore":"true","version":"5.9.1","client":"androidapp","deviceName":"vivo\_V3","deviceManufacturer":"vivo","deviceIdentifier":"vivo-vivoV3-862738034446452","sso\_token":"fa7e30a5-28b3-44fd-b528-7ee465817cd7","client\_ip":"xxxxxxxx","user\_agent":"Dalvik/2.1.0 (Linux; U; Android 5.1.1; vivo V3 Build/LMY47V)","path":"xxxxxxxx","url":"xxxxx","verb":"GET","operation":"xxxxx","point":"exit","api\_method":"xxxx","response\_code":"xxxxx","response\_time":9,"status\_code":xxxx,"msg":"xxxx","time":"2017-07-13T16:52:16.156Z","src":{"file":"xxxxx","line":xxxxx},"v":0} Please help me in figuring this out. ---
**Author:** ![ELK\_GB](https://avatars.discourse-cdn.com/v4/letter/e/a9adbd/32.png) [@ELK\_GB](https://discuss.elastic.co/u/ELK_GB)\ **Post date:** [July 13, 2017, 7:17pm UTC](https://discuss.elastic.co/t/json-is-storing-as-string-in-es/93062/2 "2017-07-13T19:17:27Z")
I have been struggling with the same issue for a while. Same setup but I have metricbeat as the source, and now I cant visualize anything if I use codec json with the kafka input, i actually get zero data even though my index rate is very high but zero data in Kibana! I'm using 5.4 for everything across my cluster. Unless I do this: ``` filter { json { source => "message" } } ``` input codec json wont work and everything in ES is just raw test. If I use the above the data in ES will start getting congested and get super delayed, to a point where everything is just having the same time stamp! any luck with your issue ? ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [July 13, 2017, 7:18pm UTC](https://discuss.elastic.co/t/json-is-storing-as-string-in-es/93062/3 "2017-07-13T19:18:40Z")
> if [@metadata][type] == "my\_app" { I don't see `[@metadata][type]` being populated anywhere. Is this condition ever true? I'd expect the kafka input's `codec => json` to do the job, though. ---
**Author:** ![ELK\_GB](https://avatars.discourse-cdn.com/v4/letter/e/a9adbd/32.png) [@ELK\_GB](https://discuss.elastic.co/u/ELK_GB)\ **Post date:** [July 13, 2017, 8:33pm UTC](https://discuss.elastic.co/t/json-is-storing-as-string-in-es/93062/4 "2017-07-13T20:33:53Z")
here is what resolved my issue, hopefully will help you, im running LS 5.4 and this resolved the json codec not working. `bin/logstash-plugin update logstash-input-kafka` update your LS Kafka input plugin. stop LS then run the above command. BTW this may not work: ``` if [@metadata][type] == "my_app" { json { source => "message" } ``` check this topic: > [@Logstash 5.2 input kafka working but no output to elasticsearch](https://discuss.elastic.co/t/logstash-5-2-input-kafka-working-but-no-output-to-elasticsearch/74720): > > Issue Summary: filebeat outputs to kafka 0.10 topic logstash indexer inputs from kafka 0.10 outputs to elasticsearch Problem: document count on elasticsearch doesn't change Note: I'm able to get filebeat=\>logstash=\>elasticsearch working ok. When I try to add the kafka broker and adjust input/output accordingly, the content consumed by logstash doesn't output to elasticsearch. Versions logstash version # /usr/share/logstash/bin/logstash -V logstash 5.2.0 installed from [elastic.co](http://elastic.co) repository… ---
**Author:** ![Narayanan\_Sukumar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/narayanan_sukumar/32/19546_2.png) [@Narayanan\_Sukumar](https://discuss.elastic.co/u/Narayanan_Sukumar)\ **Post date:** [July 14, 2017, 7:07am UTC](https://discuss.elastic.co/t/json-is-storing-as-string-in-es/93062/5 "2017-07-14T07:07:36Z")
You guys are right. After replacing the filters without checking the metadata(it is not there in the logs), it is started working. Thank you Guys. 🙂 ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [August 11, 2017, 7:07am UTC](https://discuss.elastic.co/t/json-is-storing-as-string-in-es/93062/6 "2017-08-11T07:07:36Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.