# Parsing Json to Cloudwatch Log Group in Logstash config **URL:** \ **Category:** Logstash\ **Created:** [April 3, 2019, 9:14pm UTC](https://discuss.elastic.co/t/parsing-json-to-cloudwatch-log-group-in-logstash-config/175275 "2019-04-03T21:14:33Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![drj](https://avatars.discourse-cdn.com/v4/letter/d/d07c76/32.png) [@drj](https://discuss.elastic.co/u/drj)\ **Post date:** [April 3, 2019, 9:14pm UTC](https://discuss.elastic.co/t/parsing-json-to-cloudwatch-log-group-in-logstash-config/175275/1 "2019-04-03T21:14:33Z")
I am in the process trying to change my logstash config from `grok` to `json` ... Also Is there a way to have all fields in`json` to become fields over in kibana, rather than the whole json blob being stuck in the message field? So it can be easy to filter things like `level` for e.g. Here is my logstash config: ``` input { cloudwatch_logs { start_position => "end" log_group => "aws_logs" log_group_prefix => ["true"] } ``` } filter { grok { match =\> { "message" =\> "%{TIMESTAMP\_ISO8601}\t%{UUID:[lambda][request\_id]}\t%{GREEDYDATA:message}" } overwrite =\> ["message"] tag\_on\_failure =\> } ``` grok { match => { "message" => "(?:START|END) RequestId: %{UUID:[lambda][request_id]}" } tag_on_failure => [] } grok { match => { "message" => "REPORT RequestId: %{UUID:[lambda][request_id]}\tDuration: %{BASE16FLOAT:[lambda][duration]} ms\tBilled Duration: %{BASE16FLOAT:[lambda][billed_duration]} ms \tMemory Size: %{BASE10NUM:[lambda][memory_size]} MB\tMax Memory Used: %{BASE10NUM:[lambda][memory_used]} MB" } tag_on_failure => [] } mutate { convert => { "[lambda][duration]" => "integer" "[lambda][billed_duration]" => "integer" "[lambda][memory_size]" => "integer" "[lambda][memory_used]" => "integer" } } ``` } output { amazon\_es { hosts =\> ["[es-test.amazonaws.com](http://es-test.amazonaws.com)"] region =\> "us-east-1" index =\> "logs-%{+YYYY.MM.dd}" } stdout { codec =\> rubydebug } } ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [April 3, 2019, 9:38pm UTC](https://discuss.elastic.co/t/parsing-json-to-cloudwatch-log-group-in-logstash-config/175275/2 "2019-04-03T21:38:15Z")
> [@drj](#): > > Is there a way to have all fields in `json` to become fields over in kibana, rather than the whole json blob being stuck in the message field? That's what a [json](https://www.elastic.co/guide/en/logstash/current/plugins-filters-json.html) filter does - parse JSON into fields on the event. ``` json { source => "message" } ``` ---
**Author:** ![drj](https://avatars.discourse-cdn.com/v4/letter/d/d07c76/32.png) [@drj](https://discuss.elastic.co/u/drj)\ **Post date:** [April 4, 2019, 1:04pm UTC](https://discuss.elastic.co/t/parsing-json-to-cloudwatch-log-group-in-logstash-config/175275/3 "2019-04-04T13:04:40Z")
i changed my config to : `input { ``` cloudwatch_logs { start_position => "end" log_group => ["/aws"] log_group_prefix => ["true"] } ``` } filter{ json{ source =\> "message" } } output { amazon\_es { hosts =\> ["logs.endpoint:443"] region =\> "us-east-1" index =\> "logs-%{+YYYY.MM.dd}" } stdout { codec =\> rubydebug } }` Now i get this failure message - \_jsonparsefailure ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [April 4, 2019, 1:17pm UTC](https://discuss.elastic.co/t/parsing-json-to-cloudwatch-log-group-in-logstash-config/175275/4 "2019-04-04T13:17:36Z")
What does the rubydebug output look like for one event? ---
**Author:** ![drj](https://avatars.discourse-cdn.com/v4/letter/d/d07c76/32.png) [@drj](https://discuss.elastic.co/u/drj)\ **Post date:** [April 4, 2019, 1:32pm UTC](https://discuss.elastic.co/t/parsing-json-to-cloudwatch-log-group-in-logstash-config/175275/5 "2019-04-04T13:32:40Z")
Here is an example. "lambda" =\> { "request\_id" =\> "xxxxxxxxxxxxxxxxxxxx" }, "@version" =\> "1", "message" =\> "START RequestId: xxxxxxxxxxxxxxxxxx Version: $LATEST\n", "@timestamp" =\> 2019-04-02T03:55:09.xxxx, "cloudwatch\_logs" =\> { "log\_group" =\> "/aws", "log\_stream" =\> "2019/04/02/xxxxxxxxxxxxxxxxx", "event\_id" =\> "xxxxxxxxxxxxxxxxxxxx", "ingestion\_time" =\> 2019-04-02T03:55:21.503Z } ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [April 4, 2019, 2:16pm UTC](https://discuss.elastic.co/t/parsing-json-to-cloudwatch-log-group-in-logstash-config/175275/6 "2019-04-04T14:16:16Z")
There is no JSON in that message. ---
**Author:** ![drj](https://avatars.discourse-cdn.com/v4/letter/d/d07c76/32.png) [@drj](https://discuss.elastic.co/u/drj)\ **Post date:** [April 4, 2019, 2:32pm UTC](https://discuss.elastic.co/t/parsing-json-to-cloudwatch-log-group-in-logstash-config/175275/7 "2019-04-04T14:32:57Z")
``` "@timestamp" => 2019-04-04T12:59xxxxxxxx, ``` "cloudwatch\_logs" =\> { "event\_id" =\> "xxxxxxxxxxxxxxxxxxxxxx", "ingestion\_time" =\> 2019-04-04T13:00:01.xxxx, "log\_stream" =\> "2019/04/04/[$LATEST]xxxxxxxxxx", "log\_group" =\> "/aws/xxxxx/" }, "message" =\> "Adding env: DB\_NAME\n", "tags" =\> [ [0] "\_jsonparsefailure" ], "@version" =\> "1" ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [May 2, 2019, 2:33pm UTC](https://discuss.elastic.co/t/parsing-json-to-cloudwatch-log-group-in-logstash-config/175275/8 "2019-05-02T14:33:08Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.