Gsub not working

Elastic Stack Logstash
1

Hello,

I'm trying to remove [] from my data as it is not supported in Kibana.
Generally, I'm doing it very well with gsub (with http input).

Now I'm using rabbitmq input and gsub is not working. I can't figure out why, and I've tested different solutions (see below). Note that I've tried other mutate options like rename and it is working well. I also tried gsub to replace for example , with ; and it is not working.

input {
  rabbitmq {
    host => "x.x.x.x"
    subscription_retry_interval_seconds => 5
    exchange => "amq.topic"
    key => "#"
    auto_delete => "true"
    user => "X"
    password => "x"
    metadata_enabled => "true"
    codec => "json"
}
 }

filter {
   mutate  {
      #gsub => [ "metrics","\]",""]
      #gsub => [ "metrics","\[",""]
      gsub => [ "metrics","\\[|\\]",""]
     
}
}

output {

This is an example of the json i'm receiving

{
  "_index": "proteus-rabbitmq-2017.09.12",
  "_type": "logs",
  "_id": "AV51Q5agOmTEVnBW6Jvj",
  "_score": null,
  "_source": {
    "@timestamp": "2017-09-12T08:45:16.697Z",
    "@version": "1",
    "topic": "@metadata",
    "timestamp": "2017-09-12T08:45:16.680Z",
    "tags": [],
    "metrics": [
      {
        "dataType": "float",
        "name": "temperature",
        "value": 14,
        "timestamp": "2017-09-12T08:45:16.680Z"
      },
      {
        "dataType": "float",
        "name": "humidity",
        "value": 56,
        "timestamp": "2017-09-12T08:45:16.680Z"
      }
    ]
  },
  "fields": {
    "@timestamp": [
      1505205916697
    ],
    "metrics_test.timestamp": [
      1505205916680,
      1505205916680
    ],
    "timestamp": [
      1505205916680
    ]
  },
  "sort": [
    1505205916697
  ]
}
2

As you have a json codec in the input, the metrics field is an object and not a string, which probably is why gsub fails.

3

Hi, thanks for your answer.

No, the example is one of my multiple attempts. I also tried without the json codec in the input...with the same result...

4

What does the input look like and what is the desired output?

5

this is the input :

{
"timestamp": “2017-06-20T11:44:08.383357”,
"metrics": [{
"name": “temperature”,
"timestamp": “2017-06-20T11:44:08.383357”,
"dataType": “float”,
"value": “25.5”
},
{
"name": “humidity”,
"timestamp": “2017-06-20T11:44:08.383357”,
"dataType": “float”,
"value": “50”
}]
}

I want the square brackets removed, so it would be like this :

{
"timestamp": “2017-06-20T11:44:08.383357”,
"metrics": {
"name": “temperature”,
"timestamp": “2017-06-20T11:44:08.383357”,
"dataType": “float”,
"value": “25.5”
},
{
"name": “humidity”,
"timestamp": “2017-06-20T11:44:08.383357”,
"dataType": “float”,
"value": “50”
}
}
6

If you remove the square brackets it is no longer valid JSON. Is that really what you want?

Would it not make more sense to convert it to something like this:

{
    "timestamp": "2017-06-20T11:44:08.383357",
    "temperature": 25.5,
    "humidity": 50
}
7

the problem is that kibana is not recognising anything inside the brackets.
With another application, i simply removed the bracket, and it recognized 2 separate events.
One for metrics.name = temperature and one for metrics.name = humidity. This is what i want

With the brackets, kibana recognized nothing but a string event inside metrics field.

Do you think of another way to do it ?

8

If you want them as separate events you might be able to use the split filter.

9

Yes, i understand i can use split for separating in 2 events. But the brackets will still be there and they are not supported by kibana

10

Once you have split them, you know each event will have only one object in the array, and you can then use the mutate filter to copy the fields down to the main event and then delete the metrics field.

11

It worked !!!!

Thanks soooo much, you saved my day :smiley:

12

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.