# The grok part from logstash didn't take effect in elasticsearch **URL:** \ **Category:** Logstash\ **Created:** [July 20, 2018, 10:07am UTC](https://discuss.elastic.co/t/the-grok-part-from-logstash-didnt-take-effect-in-elasticsearch/140867 "2018-07-20T10:07:55Z")\ **Posts on this page:** 17\ **Page:** 1
**Author:** ![Acacia1986](https://avatars.discourse-cdn.com/v4/letter/a/958977/32.png) [@Acacia1986](https://discuss.elastic.co/u/Acacia1986)\ **Post date:** [July 20, 2018, 10:07am UTC](https://discuss.elastic.co/t/the-grok-part-from-logstash-didnt-take-effect-in-elasticsearch/140867/1 "2018-07-20T10:07:56Z")
Below is my logstash config: ![logstash](https://us1.discourse-cdn.com/elastic/original/3X/e/0/e04bb8040e7c37cd96cc70e5e16c5ad0d2db30b9.png) And I find only the document\_type is effect after I start the logstash. I go to elasticsearch: hostname:9200/indexName and find as below: ![elasticsearch](https://us1.discourse-cdn.com/elastic/original/3X/7/4/74ba0a32b368a72b0bc2309e60dd998cb57ea22d.png) So I find the grok part didn't take effect. Is there anything can help to debug or any other idea? Notes: I have check all the logs but no erro or warning. ---
**Author:** ![Acacia1986](https://avatars.discourse-cdn.com/v4/letter/a/958977/32.png) [@Acacia1986](https://discuss.elastic.co/u/Acacia1986)\ **Post date:** [July 20, 2018, 10:09am UTC](https://discuss.elastic.co/t/the-grok-part-from-logstash-didnt-take-effect-in-elasticsearch/140867/2 "2018-07-20T10:09:07Z")
I have restart the ELK part but also the same behavior. ---
**Author:** ![Acacia1986](https://avatars.discourse-cdn.com/v4/letter/a/958977/32.png) [@Acacia1986](https://discuss.elastic.co/u/Acacia1986)\ **Post date:** [July 20, 2018, 10:46am UTC](https://discuss.elastic.co/t/the-grok-part-from-logstash-didnt-take-effect-in-elasticsearch/140867/3 "2018-07-20T10:46:00Z")
I debug run logstash and find this as below； ![debug](https://us1.discourse-cdn.com/elastic/original/3X/b/f/bff7ae63c4974aee1a3c06c54b0739dc510ce336.png) Any idea about this issue? ---
**Author:** ![Acacia1986](https://avatars.discourse-cdn.com/v4/letter/a/958977/32.png) [@Acacia1986](https://discuss.elastic.co/u/Acacia1986)\ **Post date:** [July 20, 2018, 10:57am UTC](https://discuss.elastic.co/t/the-grok-part-from-logstash-didnt-take-effect-in-elasticsearch/140867/4 "2018-07-20T10:57:23Z")
Yes, I think I got why can't parsing as Filter config. My origin log will not generate as grok format. So it can't match with grok. My question is: How to intellect matching the logs with diff format in logs? use condition? But how to use it? ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [July 22, 2018, 6:16pm UTC](https://discuss.elastic.co/t/the-grok-part-from-logstash-didnt-take-effect-in-elasticsearch/140867/6 "2018-07-22T18:16:47Z")
Don't use grok to parse JSON! Use a json filter or json codec. ---
**Author:** ![Acacia1986](https://avatars.discourse-cdn.com/v4/letter/a/958977/32.png) [@Acacia1986](https://discuss.elastic.co/u/Acacia1986)\ **Post date:** [July 23, 2018, 10:14am UTC](https://discuss.elastic.co/t/the-grok-part-from-logstash-didnt-take-effect-in-elasticsearch/140867/7 "2018-07-23T10:14:54Z")
Thanks， But my log have diff format of json. How to handle that. For example: 2 types 1.{"elapsed time":40.666,"executed by":{"user":"Administrator","password":"Welcome1"},"request"：{"method":"POST","uri":"jobs","payload":{"jobtype":"Import","parameters":{"zipFileName":"Coola.zip"}}}} 2.{"elapsed time":2.666, "request":{"method":"POST","uri":"jobs","payload":{"application":"ASmp","db":"Sample","jobtype":"maxl","parameters":{"script":"createfilters\_ASmp.msh"}}}} ---
**Author:** ![Acacia1986](https://avatars.discourse-cdn.com/v4/letter/a/958977/32.png) [@Acacia1986](https://discuss.elastic.co/u/Acacia1986)\ **Post date:** [July 23, 2018, 10:22am UTC](https://discuss.elastic.co/t/the-grok-part-from-logstash-didnt-take-effect-in-elasticsearch/140867/8 "2018-07-23T10:22:52Z")
I will hit [1] "\_jsonparsefailure" during the log format as: {"elapsed time":40.666,"executed by":{"user":"Administrator","password":"Welcome1"},"request"：{"method":"POST","uri":"jobs","payload":{"jobtype":"Import","parameters":{"zipFileName":"Coola.zip"}}}} But does this is also a json format? ---
**Author:** ![Acacia1986](https://avatars.discourse-cdn.com/v4/letter/a/958977/32.png) [@Acacia1986](https://discuss.elastic.co/u/Acacia1986)\ **Post date:** [July 23, 2018, 10:37am UTC](https://discuss.elastic.co/t/the-grok-part-from-logstash-didnt-take-effect-in-elasticsearch/140867/9 "2018-07-23T10:37:06Z")
I mean if there is non-standard json format in my log. Then how to deal with that? ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [July 24, 2018, 6:51am UTC](https://discuss.elastic.co/t/the-grok-part-from-logstash-didnt-take-effect-in-elasticsearch/140867/10 "2018-07-24T06:51:23Z")
In your first example, the colon after "request" isn't actually a colon but some other character that looks like a colon. Your second example doesn't have that problem. Apart from that those JSON strings are fine. ---
**Author:** ![Acacia1986](https://avatars.discourse-cdn.com/v4/letter/a/958977/32.png) [@Acacia1986](https://discuss.elastic.co/u/Acacia1986)\ **Post date:** [July 24, 2018, 8:05am UTC](https://discuss.elastic.co/t/the-grok-part-from-logstash-didnt-take-effect-in-elasticsearch/140867/11 "2018-07-24T08:05:53Z")
Thanks,magnusbaeck. I have to verify my json is valid here: [https://jsonlint.com/](https://jsonlint.com/) {"elapsed time":40.666,"executed by":{"user":"BDServiceAdministrator","password":"Welcome1"},"request":{"method":"POST","uri":"jobs","payload":{"jobtype":"lcmImport","parameters":{"zipFileName":"CocaCola.zip"}}}} but my logstash still can't parse it, logstash point to line line 1 column 16 which always point to "elapsed time":40.666, I try to remove it and re-parsing but will point to next one "executed by": ---
**Author:** ![Acacia1986](https://avatars.discourse-cdn.com/v4/letter/a/958977/32.png) [@Acacia1986](https://discuss.elastic.co/u/Acacia1986)\ **Post date:** [July 24, 2018, 8:22am UTC](https://discuss.elastic.co/t/the-grok-part-from-logstash-didnt-take-effect-in-elasticsearch/140867/12 "2018-07-24T08:22:03Z")
Hey, I find this [https://github.com/logstash-plugins/logstash-codec-json/issues/35](https://github.com/logstash-plugins/logstash-codec-json/issues/35) which is said empty key is valid in json but logstash can't parsing it. So I change my json format to {"elapsedtime":40.666,"executedby":{"user":"Administrator","password":"Welcome1"},"request":{"method":"POST","uri":"jobs","payload":{"jobtype":"lcmImport","parameters":{"zipFileName":"Coc.zip"}}}} And logstash can parsing successful. But the my second json format didn't have this issue. Second format as: {"elapsed time":0.675, "request":{"method":"PUT","uri":"files/applications/ASOsamp/Sample/createfilters\_ASOsamp.msh?overwrite=true"}} it always works well. Do you know why? ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [July 24, 2018, 9:34am UTC](https://discuss.elastic.co/t/the-grok-part-from-logstash-didnt-take-effect-in-elasticsearch/140867/13 "2018-07-24T09:34:10Z")
If I copy/paste the JSON string above into [jsonlint.com](http://jsonlint.com) it works just fine. ---
**Author:** ![Acacia1986](https://avatars.discourse-cdn.com/v4/letter/a/958977/32.png) [@Acacia1986](https://discuss.elastic.co/u/Acacia1986)\ **Post date:** [July 24, 2018, 9:36am UTC](https://discuss.elastic.co/t/the-grok-part-from-logstash-didnt-take-effect-in-elasticsearch/140867/14 "2018-07-24T09:36:09Z")
Yes, It work find and valid. But for logstash parse part it will failed. ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [July 24, 2018, 9:40am UTC](https://discuss.elastic.co/t/the-grok-part-from-logstash-didnt-take-effect-in-elasticsearch/140867/15 "2018-07-24T09:40:44Z")
Works fine for me with Logstash 6.1: ```nohighlight $ cat test.config input { stdin { codec => json_lines } } output { stdout { codec => rubydebug } } $ cat data {"elapsed time":40.666,"executed by":{"user":"BDServiceAdministrator","password":"Welcome1"},"request":{"method":"POST","uri":"jobs","payload":{"jobtype":"lcmImport","parameters":{"zipFileName":"CocaCola.zip"}}}} $ logstash -f test.config < data Sending Logstash's logs to /home/magnus/logstash/logstash-6.1.0/logs which is now configured via log4j2.properties [2018-07-24T11:39:58,360][INFO][logstash.modules.scaffold] Initializing module {:module_name=>"fb_apache", :directory=>"/home/magnus/logstash/logstash-6.1.0/modules/fb_apache/configuration"} [2018-07-24T11:39:58,372][INFO][logstash.modules.scaffold] Initializing module {:module_name=>"netflow", :directory=>"/home/magnus/logstash/logstash-6.1.0/modules/netflow/configuration"} [2018-07-24T11:39:58,778][WARN][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified [2018-07-24T11:39:59,313][INFO][logstash.runner] Starting Logstash {"logstash.version"=>"6.1.0"} [2018-07-24T11:39:59,765][INFO][logstash.agent] Successfully started Logstash API endpoint {:port=>9600} [2018-07-24T11:40:02,160][INFO][logstash.pipeline] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>8, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>5, "pipeline.max_inflight"=>1000, :thread=>"#"} [2018-07-24T11:40:02,224][INFO][logstash.pipeline] Pipeline started {"pipeline.id"=>"main"} [2018-07-24T11:40:02,363][INFO][logstash.agent] Pipelines running {:count=>1, :pipelines=>["main"]} { "executed by" => { "password" => "Welcome1", "user" => "BDServiceAdministrator" }, "@version" => "1", "@timestamp" => 2018-07-24T09:40:02.376Z, "host" => "bertie", "elapsed time" => 40.666, "request" => { "uri" => "jobs", "payload" => { "jobtype" => "lcmImport", "parameters" => { "zipFileName" => "CocaCola.zip" } }, "method" => "POST" } } [2018-07-24T11:40:03,004][INFO][logstash.pipeline] Pipeline terminated {"pipeline.id"=>"main"} ``` ---
**Author:** ![Acacia1986](https://avatars.discourse-cdn.com/v4/letter/a/958977/32.png) [@Acacia1986](https://discuss.elastic.co/u/Acacia1986)\ **Post date:** [July 26, 2018, 10:00am UTC](https://discuss.elastic.co/t/the-grok-part-from-logstash-didnt-take-effect-in-elasticsearch/140867/16 "2018-07-26T10:00:14Z")
Thanks, I am failed with below json but not the one which you use. Could you help to use below json to take a try in 6.1? I am working with 5.6.3, I think it doesn't matter with diff version. {"elapsed time":40.666,"executed by":{"user":"Administrator","password":"Welcome1"},"request"：{"method":"POST","uri":"jobs","payload":{"jobtype":"Import","parameters":{"zipFileName":"Coola.zip"}}}} ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [July 27, 2018, 4:04pm UTC](https://discuss.elastic.co/t/the-grok-part-from-logstash-didnt-take-effect-in-elasticsearch/140867/17 "2018-07-27T16:04:09Z")
As I said earlier, the colon after "request" isn't actually a colon but some other character that _looks_ like a colon. To be extremely explicit: ``` ..."request"：{"method"... ^ | ----- this is not a colon ``` ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [August 24, 2018, 4:04pm UTC](https://discuss.elastic.co/t/the-grok-part-from-logstash-didnt-take-effect-in-elasticsearch/140867/18 "2018-08-24T16:04:09Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.