# Need to filter below AWS API gateway logs **URL:** \ **Category:** Logstash\ **Created:** [March 25, 2019, 4:22pm UTC](https://discuss.elastic.co/t/need-to-filter-below-aws-api-gateway-logs/173796 "2019-03-25T16:22:33Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![Piyush\_Sonigra](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/piyush_sonigra/32/42735_2.png) [@Piyush\_Sonigra](https://discuss.elastic.co/u/Piyush_Sonigra)\ **Post date:** [March 25, 2019, 4:22pm UTC](https://discuss.elastic.co/t/need-to-filter-below-aws-api-gateway-logs/173796/1 "2019-03-25T16:22:33Z")
```auto {"messageType":"DATA_MESSAGE", "owner":"144360258", "logGroup":"API-Gateway-Execution-Logs_x63d3nk/live", "logStream":"d645920e395fedad7bbbed0eca3fe2e0","subscriptionFilters":["API-Gateway-Execution-Logs_x63d3nr84klive"], "logEvents":[{"id":"3463781636667557636544562987631175646966498","timestamp":1553213404230,"message":"(d7b307ed-4c36-11e9-bb5e-b7d673a) Extended Request Id: W6sqaGhwDoEFavA="}, {"id":"3463781636781291437057069165653007810157004","timestamp":1553213404281,"message":"(d7b307ed-4c36-11e9-bb5e-b7d673a) Verifying Usage Plan for request: d7b307ed-4c36-11e9-bb5e-b7d673a. API Key: API Stage: x63d3nk/live"}, {"id":"3463781636781291437057069165653007810157004","timestamp":1553213404282,"message":"(d7b307ed-4c36-11e9-bb5e-b7d673a) API Key authorized because method 'OPTIONS /v2' does not require API Key. Request will not contribute to throttle or quota limits"}, {"id":"3463781636781291437057069165653007810157004","timestamp":1553213404282,"message":"(d7b307ed-4c36-11e9-bb5e-b7d673a) Usage Plan check succeeded for API Key and API Stage x63d3nk/live"}, {"id":"3463781636781291437057069165653007810157004","timestamp":1553213404282,"message":"(d7b307ed-4c36-11e9-bb5e-b7d673a) Starting execution for request: d7b307ed-4c36-11e9-bb5e-b7d673a"}, {"id":"3463781636781291437057069165653007810157004","timestamp":1553213404282,"message":"(d7b307ed-4c36-11e9-bb5e-b7d673a) HTTP Method: OPTIONS, Resource Path: /api/v2"},{"id":"346378163678352151157698359732240390","timestamp":1553213404282,"message":"(d7b307ed-4c36-11e9-bb5e-b7d673a) Successfully completed execution"}, {"id":"3463781636781291437057069165653007810157004","timestamp":1553213404282,"message":"(d7b307ed-4c36-11e9-bb5e-b7d673a) Method completed with status: 200"}] } Filter i tried, filter { grok { match => { "message" => "%{GREEDYDATA:wd}" } } json{ source => "wd" target => "js" } mutate { add_field => { "t1" => "%{[js][logEvents][message]}"} } } ``` ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [March 25, 2019, 9:30pm UTC](https://discuss.elastic.co/t/need-to-filter-below-aws-api-gateway-logs/173796/2 "2019-03-25T21:30:52Z")
You aren't likely to get any response if you simply post a json blob into your body like that. What have you tried so far? What problems are you facing? ---
**Author:** ![Piyush\_Sonigra](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/piyush_sonigra/32/42735_2.png) [@Piyush\_Sonigra](https://discuss.elastic.co/u/Piyush_Sonigra)\ **Post date:** [March 26, 2019, 6:29am UTC](https://discuss.elastic.co/t/need-to-filter-below-aws-api-gateway-logs/173796/3 "2019-03-26T06:29:33Z")
i tried filter { grok { match =\> { "message" =\> "%{GREEDYDATA:wd}" } } json{ source =\> "wd" target =\> "js" } mutate { add\_field =\> { "t1" =\> "%{[js][logEvents][message]}"} } } ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [March 26, 2019, 1:52pm UTC](https://discuss.elastic.co/t/need-to-filter-below-aws-api-gateway-logs/173796/4 "2019-03-26T13:52:11Z")
> [@Piyush\_Sonigra](#): > > grok { > match =\> { "message" =\> "%{GREEDYDATA:wd}" } > } > json{ > source =\> "wd" > target =\> "js" > } Using grok like that does not make any sense. It is clearer and cheaper to do ``` json { source => "message" target => "js" } ``` If you want to extract all the messages from the logEvents array I think you will need to use ruby. ``` ruby { code => ' a = [] event.get("[js][logEvents]").each { |x| a << x["message"] } event.set("t1", a) ' } ``` will get you ``` "t1" => [ [0] "(d7b307ed-4c36-11e9-bb5e-b7d673a) Extended Request Id: W6sqaGhwDoEFavA=", [1] "(d7b307ed-4c36-11e9-bb5e-b7d673a) Verifying Usage Plan for request: d7b307ed-4c36-11e9-bb5e-b7d673a. API Key: API Stage: x63d3nk/live", [2] "(d7b307ed-4c36-11e9-bb5e-b7d673a) API Key authorized because method 'OPTIONS /v2' does not require API Key. Request will not contribute to throttle or quota limits", [3] "(d7b307ed-4c36-11e9-bb5e-b7d673a) Usage Plan check succeeded for API Key and API Stage x63d3nk/live", [4] "(d7b307ed-4c36-11e9-bb5e-b7d673a) Starting execution for request: d7b307ed-4c36-11e9-bb5e-b7d673a", [5] "(d7b307ed-4c36-11e9-bb5e-b7d673a) HTTP Method: OPTIONS, Resource Path: /api/v2", [6] "(d7b307ed-4c36-11e9-bb5e-b7d673a) Successfully completed execution", [7] "(d7b307ed-4c36-11e9-bb5e-b7d673a) Method completed with status: 200" ], ``` ---
**Author:** ![Piyush\_Sonigra](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/piyush_sonigra/32/42735_2.png) [@Piyush\_Sonigra](https://discuss.elastic.co/u/Piyush_Sonigra)\ **Post date:** [March 26, 2019, 5:16pm UTC](https://discuss.elastic.co/t/need-to-filter-below-aws-api-gateway-logs/173796/5 "2019-03-26T17:16:15Z")
Thanks @Badger i have parsed using json,split and used value using mutate [https://www.elastic.co/guide/en/logstash/current/plugins-filters-split.html](https://www.elastic.co/guide/en/logstash/current/plugins-filters-split.html) filter { json { source =\> "message" } split { field =\> "logEvents" } mutate { add\_field =\> ["time", "%{[logEvents][timestamp]}"] } date { locale =\> "en" timezone =\> "UCT" match =\> ["time", "MMM dd yyyy HH:mm:ss","MMM d yyyy HH:mm:ss", "ISO8601"] } } ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [April 23, 2019, 5:16pm UTC](https://discuss.elastic.co/t/need-to-filter-below-aws-api-gateway-logs/173796/6 "2019-04-23T17:16:30Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.