# Logstash - Convert JSON array and delete whitespaces from key fields

**URL:** <https://discuss.elastic.co/t/logstash-convert-json-array-and-delete-whitespaces-from-key-fields/332419>\
**Category:** Logstash\
**Created:** [May 3, 2023, 12:28pm UTC](https://discuss.elastic.co/t/logstash-convert-json-array-and-delete-whitespaces-from-key-fields/332419 "2023-05-03T12:28:18Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![h49nakxs](https://avatars.discourse-cdn.com/v4/letter/h/e68b1a/32.png) [@h49nakxs](https://discuss.elastic.co/u/h49nakxs)\
**Post date:** [May 3, 2023, 12:28pm UTC](https://discuss.elastic.co/t/logstash-convert-json-array-and-delete-whitespaces-from-key-fields/332419/1 "2023-05-03T12:28:18Z")

</div>

Hi there,

I'm using Logstash to receive events from winlogbeat and send them to Kafka which will ultimately send them further. To be able to correctly process those events at the end of the pipe, I need to :

- Convert the JSON array to a supported type (for example : string).
- Delete the whitespaces included in certain key fields.

And if possible, without knowing the input data in advance.  
Here is an example of the data I receive :

```auto
{
  "key1": {
    "key2": "value1",
    "key3": "value2",
    "key4": [
      "value3",
      "value4"
    ]
  }
  "key5": {
    "key6": "value5",
    "key7": {
      "key8 with space": "value6",
      "key9 with space": "value7"
	}
  }
}

```

I would like to flatten the key4 array and remove the spaces in key8 and key9 fields.  
Here is my actual Logstash pipeline configuration :

```auto
input {
    beats {
	port => "5044"
    }
}

filter {
    mutate {
	convert => {
	    "key4" => "string"
	}
    }

    kv {
	remove_char_key => " "
    }
}

output {
    stdout { codec => json}
}

```

But none of the filter does anything.

I've also tried several variation using "json" filter, "split" filter or "mutate rename" filter but none of them helped.

I think I may not completely understand how Logstash process the input data 🙂

Thanks a lot in advance for your help.

---

<div class="post-metadata">

**Author:** ![h49nakxs](https://avatars.discourse-cdn.com/v4/letter/h/e68b1a/32.png) [@h49nakxs](https://discuss.elastic.co/u/h49nakxs)\
**Post date:** [May 4, 2023, 1:27pm UTC](https://discuss.elastic.co/t/logstash-convert-json-array-and-delete-whitespaces-from-key-fields/332419/2 "2023-05-04T13:27:29Z")

</div>

For those interested, I've figured out how to proceed.

To convert the JSON array, I've used a grok filter to extract the values to a new root field and then a mutate filter to delete the original array :

```auto
    grok {
	break_on_match => false
	match => {
	    "[key1][key4][0]" => "%{GREEDYDATA:key1_key4_0}"
	    "[key1][key4[1]" => "%{GREEDYDATA:key1_key4_1}"
	 }
    }

<...>

    mutate {
	remove_field => ["[key1][key4]" ]
    }

```

I don't know how to do it without knowing the keys in advance however.

And to delete spaces included in key fields, I've used a mutate filter :

```auto
filter {
    mutate {
	rename => { "[key5][key7][key8 with space]" => "[key5][key7][key8_without_space]"}
	rename => { "[key5][key7][key9 with space]" => "[key5][key7][key9_without_space]"}
    }

```

I've also tried to do it automatically without writing all the keys down with a ruby code filter :

```auto
	ruby {
	code => '
	    hash = event.to_hash
	    hash.each do |k,v|
	    k.strip!
	    event.set(k,v)
	    end
	    '
	}

```

But that didn't work. I guess it's related to the fact that the key fields to rename are nested fields.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [May 4, 2023, 7:55pm UTC](https://discuss.elastic.co/t/logstash-convert-json-array-and-delete-whitespaces-from-key-fields/332419/3 "2023-05-04T19:55:44Z")

</div>

> [@h49nakxs](#):
>
> I don't know how to do it without knowing the keys in advance however.

You could do that using ruby....

```
    ruby {
        init => '
            def doSomething(object, name, event)
#puts "Working on #{name}"
#Removed "if object" test since we need to process null valued fields
                    if object.kind_of?(Hash) and object != {}
                        object.each { |k, v| doSomething(v, "#{name}[#{k}]", event) }
                    elsif object.kind_of?(Array) and object != []
                        event.set(name, object.to_s)
                    else
                        lastElement = name.gsub(/^.*\[/, "").gsub(/\]$/, "")
                        if lastElement =~ / /
                            event.set(name.gsub(" ", ""), event.remove(name))
                        end
                end
            end
        '
        code => '
            event.to_hash.each { |k, v|
                doSomething(v, "[#{k}]", event)
            }
        '
    }

```

which will produce

```
      "key5" => {
    "key6" => "value5",
    "key7" => {
        "key8withspace" => "value6",
        "key9withspace" => "value7"
    }
},
      "key1" => {
    "key4" => "[\"value3\", \"value4\"]",
    "key2" => "value1",
    "key3" => "value2"
}

```

---

<div class="post-metadata">

**Author:** ![h49nakxs](https://avatars.discourse-cdn.com/v4/letter/h/e68b1a/32.png) [@h49nakxs](https://discuss.elastic.co/u/h49nakxs)\
**Post date:** [May 9, 2023, 11:55am UTC](https://discuss.elastic.co/t/logstash-convert-json-array-and-delete-whitespaces-from-key-fields/332419/4 "2023-05-09T11:55:33Z")

</div>

OK, I will try this. Thank you !

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [June 6, 2023, 11:56am UTC](https://discuss.elastic.co/t/logstash-convert-json-array-and-delete-whitespaces-from-key-fields/332419/5 "2023-06-06T11:56:35Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.