Logstash - based on filed type create a new field

[antonisnyc94]

Hello,

I've been trying for 1-2 days to figure out why to create new fields when the type of the event is object but I cant seem to make it work..

This is the error im getting:
Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"uat-gitlab-2021.32", :_type=>"_doc", :_routing=>nil}, #<LogStash::Event:0x323c63bd>], :response=>{"index"=>{"_index"=>"uat-gitlab-2021.32", "_type"=>"_doc", "_id"=>"KsvhxXsBdDJn5u6LtA8n", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse field [body.json.params.value] of type [text] in document with id 'KsvhxXsBdDJn5u6LtA8n'. Preview of field's value: '{features={vault_secrets=[FILTERED], return_exit_code=null, image=null, shared=null, variables=[FILTERED], cache=null, masking=null, cancelable=null, session=null, trace_reset=[FILTERED], terminal=null, services=null, multi_build_steps=null, upload_multiple_artifacts=null, raw_variables=[FILTERED], trace_checksum=[FILTERED], proxy=null, refspecs=null, upload_raw_artifacts=null, artifacts_exclude=null, trace_size=[FILTERED], artifacts=null}, shell=bash, executor=docker+machine, name=gitlab-runner, version=14.0.0, config={gpus=}, platform=linux, architecture=amd64, revision=3b6f852e}'", "caused_by"=>{"type"=>"illegal_state_exception", "reason"=>"Can't get text on a START_OBJECT at 1:5253"}}}}}

For logstash conf i tried:
1.

                ruby {
                  code => '
                  case event.get("[body][json][params][value]")
                  when !String
                      event.set("[body.json.params.valueObj]", event.get("[body][json][params][value]"))
                      event.remove("[body][json][params][value]")
                  end
                  '

 if [body][json][params][value] =~ /^{.*}/    {
                 mutate { rename => { "[body][json][params][value]" => "[body][json][params][valueObj] "}     
  }

 if [body][json][params][value] =~ /^{.*}/    {
               json {
                   source => "[body][json][params][value]"
                   target => "[body][json][params][valueObj]"
                   skip_on_invalid_json => false
                 }
  }

Nothings seems to work and troubleshooting it is not easy. I'd truly appreciate any help..

Best regards,
Tony

[antonisnyc94]

Raw Data: (NOT ACCEPTED)

[2021-09-08T18:55:25,478][WARN ][logstash.filters.json ][main][a1efeb8776d794510296c76804c68bd0a857d8f01a03cb367e7ec4e4c17dbf98] Error parsing json {:source=>"[body][json][params]", :raw=>[{"value"=>{"version"=>"14.0.0", "platform"=>"linux", "executor"=>"kubernetes", "config"=>{"gpus"=>""}, "revision"=>"3b6f852e", "architecture"=>"amd64", "shell"=>"bash", "features"=>{"artifacts_exclude"=>nil, "session"=>nil, "vault_secrets"=>"[FILTERED]", "upload_raw_artifacts"=>nil, "artifacts"=>nil, "proxy"=>nil, "shared"=>nil, "services"=>nil, "cancelable"=>nil, "trace_reset"=>"[FILTERED]", "terminal"=>nil, "multi_build_steps"=>nil, "image"=>nil, "masking"=>nil, "trace_checksum"=>"[FILTERED]", "variables"=>"[FILTERED]", "upload_multiple_artifacts"=>nil, "raw_variables"=>"[FILTERED]", "trace_size"=>"[FILTERED]", "cache"=>nil, "return_exit_code"=>nil, "refspecs"=>nil}, "name"=>"gitlab-runner"}, "key"=>"info"}, {"value"=>"[FILTERED]", "key"=>"token"}, {"value"=>"5352b6a612acbdc4e964b498a7c287cb", "key"=>"last_update"}]

RAW DATA: (ACCEPTED)
raw=>[{"value"=>"10.130.138.12", "key"=>"host"}]

[Badger]

The difference between those two is that in one the [value] field is text ("10.130.138.12") and in the other it is an object. See here for one of the many threads about this.

Decide which you want it to be (string or object) and modify your events where it has the wrong type accordingly.

[antonisnyc94]

Hey Badger,

Thank you for your prompt response! In the 1st RAW data sample i posted above it appears that it's an array of objects with key and value keys. How do I modify my 1st attempt as shown in the original post to go through the loop and change the fieldname when object is present. Im new to logstash and even newer to ruby. Im sorry. I truly appreciate your response!

Regards,
Tony

[Badger]

You could try

    ruby {
        code => '
            data = event.get("[body][json][params]")
            data.each_index { |x|
                if data[x]["value"].is_a? Hash
                    data[x]["valueObj"] = data[x].delete("value")
                end
            }
            event.set("[body][json][params]", data)
        '
    }

You will probably need to add some tests to avoid exceptions.

[antonisnyc94]

@Badger, you have no idea how grateful I am for your help! I wish you all the best!