ElasticSearch Input and data manipulation

Elastic Stack Logstash
1

I am trying to use the ES input plugin for logstash to further manipulate some data and reinsert the data into ES.
I have a logstash filter like this:

input {
    elasticsearch {
      hosts => [
            "ms-master01.mydomain.priv",
            "ms-master02.mydomain.priv",
            "ms-master03.mydomain.priv"
        ]
      index => "radio-2019.12.16"
      query => '{
            "query": {
                "match": {
                    "_id": "iYzVEG8BVKyVzDtpHTCW"
                }
            }
        }'
      docinfo => true
      user => "admin"
      password => "xxxxx"
    }
}
filter {
    json { 
        source => "message" 
    }
    mutate {
        remove_field => ["method"]
    }
    if [queryString] =~ "^\?aw_0_\w*" or [queryString] =~ "^\?amsparams" or [queryString] =~ "^\?listenerid" {
        kv {
            source => "queryString"
            field_split => "&?"
            target => "queryParams"
            remove_field => ["queryString"]
        }
    }
}
output {
  stdout { codec => rubydebug
    }
}

but the output is not modified from the original.
The "mutate" field is still there and the "queryString" field is not divided.

{
          "src_ip" => {
                "as_org" => "Wind Tre S.p.A.",
              "latitude" => 43.1479,
              "location" => {
            "lat" => 43.1479,
            "lon" => 12.1097
        },
                    "ip" => "151.35.152.50",
             "longitude" => 12.1097,
                   "asn" => 1267,
        "continent_code" => "EU",
         "country_code3" => "IT",
          "country_name" => "Italy",
         "country_code2" => "IT",
              "timezone" => "Europe/Rome"
    },
       "userAgent" => {
        "os_name" => "Other",
             "os" => "Other",
         "device" => "Other",
           "name" => "Other",
          "build" => ""
    },
       "bytesSent" => 2919631,
      "sysloghost" => "ms-radio-edge03.farmcal.med.priv",
        "facility" => "radio",
      "@timestamp" => 2019-12-16T22:29:59.000Z,
        "response" => 200,
         "referer" => "-",
         "appname" => "nginx",
     "queryString" => "?aw_0_1st.version=6.9.7%3Aandroid28&aw_0_1st.ts=1576535180&aw_0_1st.playerid=mediamondvirgin_android_player&aw_0_awz.listenerid=dc7a6614-56ec-4fdc-8ea6-34e2c76ab625&aw_0_req.lmt=0&aw_0_1st.gpslat=43.1918&aw_0_1st.gpslong=13.3733&sdkiad=1&call=1&aw_0_req.permissions=C10010CCCC&aw_0_1st.suppstatus=LOC_OK-SON_NOI&aw0_req.uiMode=1&aw_0_awz.appVers=5.1.1:76",
    "userAgentLog" => "UserAgent",
        "@version" => "1",
        "severity" => "INFO",
     "relativeUrl" => "/Virgin.mp3",
    "clientIpType" => "external",
     "requestTime" => 218,
          "method" => "GET"
}

Could you give me some help ?

Regards,
Stefano Bossi

2

When I run that message though that filter I get

    "queryParams" => {
        "aw_0_1st.gpslong" => "13.3733",
     "aw_0_awz.listenerid" => "dc7a6614-56ec-4fdc-8ea6-34e2c76ab625",
                  "sdkiad" => "1",
            ...

I do not think you are running the configuration that you think you are. I cannot think of any way that the remove_field could fail to remove that field.

3

I think the problem is that the result from the ES query is not formatted as I think...

Anyway, the result from ES is a "message" ?

Regards,
S.

4

Yes!
I am ashamed! I was launching the wrong command line! My script works.
Sorry for wasting your time.

Thanks,
S.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.