Need to filter below AWS API gateway logs

Elastic Stack Logstash
1 
{"messageType":"DATA_MESSAGE",
"owner":"144360258",
"logGroup":"API-Gateway-Execution-Logs_x63d3nk/live",
"logStream":"d645920e395fedad7bbbed0eca3fe2e0","subscriptionFilters":["API-Gateway-Execution-Logs_x63d3nr84klive"],
"logEvents":[{"id":"3463781636667557636544562987631175646966498","timestamp":1553213404230,"message":"(d7b307ed-4c36-11e9-bb5e-b7d673a) Extended Request Id: W6sqaGhwDoEFavA="},
             {"id":"3463781636781291437057069165653007810157004","timestamp":1553213404281,"message":"(d7b307ed-4c36-11e9-bb5e-b7d673a) Verifying Usage Plan for request: d7b307ed-4c36-11e9-bb5e-b7d673a. API Key:  API Stage: x63d3nk/live"},
             {"id":"3463781636781291437057069165653007810157004","timestamp":1553213404282,"message":"(d7b307ed-4c36-11e9-bb5e-b7d673a) API Key  authorized because method 'OPTIONS /v2' does not require API Key. Request will not contribute to throttle or quota limits"},
             {"id":"3463781636781291437057069165653007810157004","timestamp":1553213404282,"message":"(d7b307ed-4c36-11e9-bb5e-b7d673a) Usage Plan check succeeded for API Key  and API Stage x63d3nk/live"},
             {"id":"3463781636781291437057069165653007810157004","timestamp":1553213404282,"message":"(d7b307ed-4c36-11e9-bb5e-b7d673a) Starting execution for request: d7b307ed-4c36-11e9-bb5e-b7d673a"},
             {"id":"3463781636781291437057069165653007810157004","timestamp":1553213404282,"message":"(d7b307ed-4c36-11e9-bb5e-b7d673a) HTTP Method: OPTIONS, Resource Path: /api/v2"},{"id":"346378163678352151157698359732240390","timestamp":1553213404282,"message":"(d7b307ed-4c36-11e9-bb5e-b7d673a) Successfully completed execution"},
             {"id":"3463781636781291437057069165653007810157004","timestamp":1553213404282,"message":"(d7b307ed-4c36-11e9-bb5e-b7d673a) Method completed with status: 200"}]
}

Filter i tried,
filter {
    grok {
        match => { "message" => "%{GREEDYDATA:wd}" }
    }
    json{
        source => "wd"
        target => "js"
    }
    mutate {
        add_field => { "t1" => "%{[js][logEvents][message]}"}
    }
}
2

You aren't likely to get any response if you simply post a json blob into your body like that.

What have you tried so far?
What problems are you facing?

3

i tried

filter {
grok {
match => { "message" => "%{GREEDYDATA:wd}" }
}
json{
source => "wd"
target => "js"
}
mutate {
add_field => { "t1" => "%{[js][logEvents][message]}"}
}
}

4

Using grok like that does not make any sense. It is clearer and cheaper to do

json { source => "message" target => "js" }

If you want to extract all the messages from the logEvents array I think you will need to use ruby.

    ruby {
        code => '
            a = []
            event.get("[js][logEvents]").each { |x|
                a << x["message"]
            }
            event.set("t1", a)
        '
    }

will get you

        "t1" => [
    [0] "(d7b307ed-4c36-11e9-bb5e-b7d673a) Extended Request Id: W6sqaGhwDoEFavA=",
    [1] "(d7b307ed-4c36-11e9-bb5e-b7d673a) Verifying Usage Plan for request: d7b307ed-4c36-11e9-bb5e-b7d673a. API Key:  API Stage: x63d3nk/live",
    [2] "(d7b307ed-4c36-11e9-bb5e-b7d673a) API Key  authorized because method 'OPTIONS /v2' does not require API Key. Request will not contribute to throttle or quota limits",
    [3] "(d7b307ed-4c36-11e9-bb5e-b7d673a) Usage Plan check succeeded for API Key  and API Stage x63d3nk/live",
    [4] "(d7b307ed-4c36-11e9-bb5e-b7d673a) Starting execution for request: d7b307ed-4c36-11e9-bb5e-b7d673a",
    [5] "(d7b307ed-4c36-11e9-bb5e-b7d673a) HTTP Method: OPTIONS, Resource Path: /api/v2",
    [6] "(d7b307ed-4c36-11e9-bb5e-b7d673a) Successfully completed execution",
    [7] "(d7b307ed-4c36-11e9-bb5e-b7d673a) Method completed with status: 200"
],
1 Like
5

Thanks @Badger

i have parsed using json,split and used value using mutate https://www.elastic.co/guide/en/logstash/current/plugins-filters-split.html

filter {
json {
source => "message"
}
split {
field => "logEvents"
}
mutate {
add_field => ["time", "%{[logEvents][timestamp]}"]
}
date {
locale => "en"
timezone => "UCT"
match => [ "time", "MMM dd yyyy HH:mm:ss","MMM d yyyy HH:mm:ss", "ISO8601" ]
}
}

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.