# ElasticSearch Input and data manipulation

**URL:** <https://discuss.elastic.co/t/elasticsearch-input-and-data-manipulation/217252>\
**Category:** Logstash\
**Created:** [January 30, 2020, 5:42pm UTC](https://discuss.elastic.co/t/elasticsearch-input-and-data-manipulation/217252 "2020-01-30T17:42:44Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![Stefano\_Bossi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stefano_bossi/32/41743_2.png) [@Stefano\_Bossi](https://discuss.elastic.co/u/Stefano_Bossi)\
**Post date:** [January 30, 2020, 5:42pm UTC](https://discuss.elastic.co/t/elasticsearch-input-and-data-manipulation/217252/1 "2020-01-30T17:42:44Z")

</div>

I am trying to use the ES input plugin for logstash to further manipulate some data and reinsert the data into ES.  
I have a logstash filter like this:

```
input {
    elasticsearch {
      hosts => [
            "ms-master01.mydomain.priv",
            "ms-master02.mydomain.priv",
            "ms-master03.mydomain.priv"
        ]
      index => "radio-2019.12.16"
      query => '{
            "query": {
                "match": {
                    "_id": "iYzVEG8BVKyVzDtpHTCW"
                }
            }
        }'
      docinfo => true
      user => "admin"
      password => "xxxxx"
    }
}
filter {
    json { 
        source => "message" 
    }
    mutate {
        remove_field => ["method"]
    }
    if [queryString] =~ "^\?aw_0_\w*" or [queryString] =~ "^\?amsparams" or [queryString] =~ "^\?listenerid" {
        kv {
            source => "queryString"
            field_split => "&?"
            target => "queryParams"
            remove_field => ["queryString"]
        }
    }
}
output {
  stdout { codec => rubydebug
    }
}

```

but the output is not modified from the original.  
The "mutate" field is still there and the "queryString" field is not divided.

```
{
          "src_ip" => {
                "as_org" => "Wind Tre S.p.A.",
              "latitude" => 43.1479,
              "location" => {
            "lat" => 43.1479,
            "lon" => 12.1097
        },
                    "ip" => "151.35.152.50",
             "longitude" => 12.1097,
                   "asn" => 1267,
        "continent_code" => "EU",
         "country_code3" => "IT",
          "country_name" => "Italy",
         "country_code2" => "IT",
              "timezone" => "Europe/Rome"
    },
       "userAgent" => {
        "os_name" => "Other",
             "os" => "Other",
         "device" => "Other",
           "name" => "Other",
          "build" => ""
    },
       "bytesSent" => 2919631,
      "sysloghost" => "ms-radio-edge03.farmcal.med.priv",
        "facility" => "radio",
      "@timestamp" => 2019-12-16T22:29:59.000Z,
        "response" => 200,
         "referer" => "-",
         "appname" => "nginx",
     "queryString" => "?aw_0_1st.version=6.9.7%3Aandroid28&aw_0_1st.ts=1576535180&aw_0_1st.playerid=mediamondvirgin_android_player&aw_0_awz.listenerid=dc7a6614-56ec-4fdc-8ea6-34e2c76ab625&aw_0_req.lmt=0&aw_0_1st.gpslat=43.1918&aw_0_1st.gpslong=13.3733&sdkiad=1&call=1&aw_0_req.permissions=C10010CCCC&aw_0_1st.suppstatus=LOC_OK-SON_NOI&aw0_req.uiMode=1&aw_0_awz.appVers=5.1.1:76",
    "userAgentLog" => "UserAgent",
        "@version" => "1",
        "severity" => "INFO",
     "relativeUrl" => "/Virgin.mp3",
    "clientIpType" => "external",
     "requestTime" => 218,
          "method" => "GET"
}

```

Could you give me some help ?

Regards,  
Stefano Bossi

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [January 30, 2020, 6:09pm UTC](https://discuss.elastic.co/t/elasticsearch-input-and-data-manipulation/217252/2 "2020-01-30T18:09:37Z")

</div>

When I run that message though that filter I get

```
    "queryParams" => {
        "aw_0_1st.gpslong" => "13.3733",
     "aw_0_awz.listenerid" => "dc7a6614-56ec-4fdc-8ea6-34e2c76ab625",
                  "sdkiad" => "1",
            ...

```

I do not think you are running the configuration that you think you are. I cannot think of any way that the remove\_field could fail to remove that field.

---

<div class="post-metadata">

**Author:** ![Stefano\_Bossi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stefano_bossi/32/41743_2.png) [@Stefano\_Bossi](https://discuss.elastic.co/u/Stefano_Bossi)\
**Post date:** [January 30, 2020, 7:22pm UTC](https://discuss.elastic.co/t/elasticsearch-input-and-data-manipulation/217252/3 "2020-01-30T19:22:19Z")

</div>

I think the problem is that the result from the ES query is not formatted as I think...

Anyway, the result from ES is a "message" ?

Regards,  
S.

---

<div class="post-metadata">

**Author:** ![Stefano\_Bossi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stefano_bossi/32/41743_2.png) [@Stefano\_Bossi](https://discuss.elastic.co/u/Stefano_Bossi)\
**Post date:** [January 31, 2020, 8:04am UTC](https://discuss.elastic.co/t/elasticsearch-input-and-data-manipulation/217252/4 "2020-01-31T08:04:01Z")

</div>

Yes!  
I am ashamed! I was launching the wrong command line! My script works.  
Sorry for wasting your time.

Thanks,  
S.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [February 28, 2020, 8:04am UTC](https://discuss.elastic.co/t/elasticsearch-input-and-data-manipulation/217252/5 "2020-02-28T08:04:24Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.