# Logstash - based on filed type create a new field

**URL:** https://discuss.elastic.co/t/logstash-based-on-filed-type-create-a-new-field/283684
**Category:** Logstash
**Tags:** docker
**Created:** [September 8, 2021, 2:53pm UTC](https://discuss.elastic.co/t/logstash-based-on-filed-type-create-a-new-field/283684 "2021-09-08T14:53:04Z")
**Posts on this page:** 7
**Page:** 1

<div class="post-metadata">

### Author: ![antonisnyc94](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/antonisnyc94/32/94316_2.png) [@antonisnyc94](https://discuss.elastic.co/u/antonisnyc94)
#### Post date: [September 8, 2021, 2:53pm UTC](https://discuss.elastic.co/t/logstash-based-on-filed-type-create-a-new-field/283684/1 "2021-09-08T14:53:04Z")

</div>

Hello,

I've been trying for 1-2 days to figure out why to create new fields when the type of the event is object but I cant seem to make it work..

This is the error im getting:  
`Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"uat-gitlab-2021.32", :_type=>"_doc", :_routing=>nil}, #<LogStash::Event:0x323c63bd>], :response=>{"index"=>{"_index"=>"uat-gitlab-2021.32", "_type"=>"_doc", "_id"=>"KsvhxXsBdDJn5u6LtA8n", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse field [body.json.params.value] of type [text] in document with id 'KsvhxXsBdDJn5u6LtA8n'. Preview of field's value: '{features={vault_secrets=[FILTERED], return_exit_code=null, image=null, shared=null, variables=[FILTERED], cache=null, masking=null, cancelable=null, session=null, trace_reset=[FILTERED], terminal=null, services=null, multi_build_steps=null, upload_multiple_artifacts=null, raw_variables=[FILTERED], trace_checksum=[FILTERED], proxy=null, refspecs=null, upload_raw_artifacts=null, artifacts_exclude=null, trace_size=[FILTERED], artifacts=null}, shell=bash, executor=docker+machine, name=gitlab-runner, version=14.0.0, config={gpus=}, platform=linux, architecture=amd64, revision=3b6f852e}'", "caused_by"=>{"type"=>"illegal_state_exception", "reason"=>"Can't get text on a START_OBJECT at 1:5253"}}}}}`

For logstash conf i tried:  
1.

```auto
                ruby {
                  code => '
                  case event.get("[body][json][params][value]")
                  when !String
                      event.set("[body.json.params.valueObj]", event.get("[body][json][params][value]"))
                      event.remove("[body][json][params][value]")
                  end
                  '

```

1. 

```auto
 if [body][json][params][value] =~ /^{.*}/ {
                 mutate { rename => { "[body][json][params][value]" => "[body][json][params][valueObj] "}     
  }

```

1. 

```auto
 if [body][json][params][value] =~ /^{.*}/ {
               json {
                   source => "[body][json][params][value]"
                   target => "[body][json][params][valueObj]"
                   skip_on_invalid_json => false
                 }
  }

```

Nothings seems to work and troubleshooting it is not easy. I'd truly appreciate any help..

Best regards,  
Tony

---

<div class="post-metadata">

### Author: ![antonisnyc94](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/antonisnyc94/32/94316_2.png) [@antonisnyc94](https://discuss.elastic.co/u/antonisnyc94)
#### Post date: [September 8, 2021, 6:59pm UTC](https://discuss.elastic.co/t/logstash-based-on-filed-type-create-a-new-field/283684/2 "2021-09-08T18:59:24Z")

</div>

Raw Data: (NOT ACCEPTED)

`[2021-09-08T18:55:25,478][WARN][logstash.filters.json][main][a1efeb8776d794510296c76804c68bd0a857d8f01a03cb367e7ec4e4c17dbf98] Error parsing json {:source=>"[body][json][params]", :raw=>[{"value"=>{"version"=>"14.0.0", "platform"=>"linux", "executor"=>"kubernetes", "config"=>{"gpus"=>""}, "revision"=>"3b6f852e", "architecture"=>"amd64", "shell"=>"bash", "features"=>{"artifacts_exclude"=>nil, "session"=>nil, "vault_secrets"=>"[FILTERED]", "upload_raw_artifacts"=>nil, "artifacts"=>nil, "proxy"=>nil, "shared"=>nil, "services"=>nil, "cancelable"=>nil, "trace_reset"=>"[FILTERED]", "terminal"=>nil, "multi_build_steps"=>nil, "image"=>nil, "masking"=>nil, "trace_checksum"=>"[FILTERED]", "variables"=>"[FILTERED]", "upload_multiple_artifacts"=>nil, "raw_variables"=>"[FILTERED]", "trace_size"=>"[FILTERED]", "cache"=>nil, "return_exit_code"=>nil, "refspecs"=>nil}, "name"=>"gitlab-runner"}, "key"=>"info"}, {"value"=>"[FILTERED]", "key"=>"token"}, {"value"=>"5352b6a612acbdc4e964b498a7c287cb", "key"=>"last_update"}]`

RAW DATA: (ACCEPTED)  
`raw=>[{"value"=>"10.130.138.12", "key"=>"host"}]`

---

<div class="post-metadata">

### Author: ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [September 8, 2021, 8:44pm UTC](https://discuss.elastic.co/t/logstash-based-on-filed-type-create-a-new-field/283684/3 "2021-09-08T20:44:49Z")

</div>

The difference between those two is that in one the [value] field is text ("10.130.138.12") and in the other it is an object. See [here](https://discuss.elastic.co/t/problem-logstash-outputs-elasticsearch-could-not-index-event-to-elasticsearch-wazuh-alerts-3-x-2020-05-30/235038/6) for one of the many threads about this.

Decide which you want it to be (string or object) and modify your events where it has the wrong type accordingly.

---

<div class="post-metadata">

### Author: ![antonisnyc94](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/antonisnyc94/32/94316_2.png) [@antonisnyc94](https://discuss.elastic.co/u/antonisnyc94)
#### Post date: [September 8, 2021, 9:04pm UTC](https://discuss.elastic.co/t/logstash-based-on-filed-type-create-a-new-field/283684/4 "2021-09-08T21:04:41Z")

</div>

Hey Badger,

Thank you for your prompt response! In the 1st RAW data sample i posted above it appears that it's an array of objects with key and value keys. How do I modify my 1st attempt as shown in the original post to go through the loop and change the fieldname when object is present. Im new to logstash and even newer to ruby. Im sorry. I truly appreciate your response!

Regards,  
Tony

---

<div class="post-metadata">

### Author: ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [September 8, 2021, 11:14pm UTC](https://discuss.elastic.co/t/logstash-based-on-filed-type-create-a-new-field/283684/5 "2021-09-08T23:14:01Z")

</div>

You could try

```
    ruby {
        code => '
            data = event.get("[body][json][params]")
            data.each_index { |x|
                if data[x]["value"].is_a? Hash
                    data[x]["valueObj"] = data[x].delete("value")
                end
            }
            event.set("[body][json][params]", data)
        '
    }

```

You will probably need to add some tests to avoid exceptions.

---

<div class="post-metadata">

### Author: ![antonisnyc94](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/antonisnyc94/32/94316_2.png) [@antonisnyc94](https://discuss.elastic.co/u/antonisnyc94)
#### Post date: [September 9, 2021, 10:43am UTC](https://discuss.elastic.co/t/logstash-based-on-filed-type-create-a-new-field/283684/6 "2021-09-09T10:43:42Z")

</div>

@Badger, you have no idea how grateful I am for your help! I wish you all the best!

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [October 7, 2021, 10:43am UTC](https://discuss.elastic.co/t/logstash-based-on-filed-type-create-a-new-field/283684/7 "2021-10-07T10:43:57Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
