# Initialize counter in logstash and replace the '%{\[message\]\[0\]}'?

**URL:** <https://discuss.elastic.co/t/initialize-counter-in-logstash-and-replace-the-message-0/304912>\
**Category:** Logstash\
**Created:** [May 17, 2022, 9:36am UTC](https://discuss.elastic.co/t/initialize-counter-in-logstash-and-replace-the-message-0/304912 "2022-05-17T09:36:14Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![alex\_vermex](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alex_vermex/32/101267_2.png) [@alex\_vermex](https://discuss.elastic.co/u/alex_vermex)\
**Post date:** [May 17, 2022, 9:36am UTC](https://discuss.elastic.co/t/initialize-counter-in-logstash-and-replace-the-message-0/304912/1 "2022-05-17T09:36:14Z")

</div>

Hi,  
after doing the split filter, so my message was an array ok, so **%{[message][0]} %{[message][1]}**... that gives me their value fine. now I have an array of length 30 for example and I start the condition `if {}`  
example logstash.conf:

```auto
filter {
mutate { split => {"message" => "|"} }
mutate {
      add_field => { "DateTime" => "%{[message][0]}" }
      add_field => { "version" => "%{[message][1]}" } ............
}
if [a] == "Offline" {
              mutate { add_field => {"example1" => "%{[message][29]}"} }
            }
}

```

I mean, can I create a counter for example to replace `%{[message][29]}` with `x+1` and I declare `int x=0` or something like that because it's a bad idea to keep doing it manually `%{[message][0]} %{[message][1]} %{[message][29]} %{[message][40]} %{[message][50]} ......`😕

Thanks!

---

<div class="post-metadata">

**Author:** ![alex\_vermex](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alex_vermex/32/101267_2.png) [@alex\_vermex](https://discuss.elastic.co/u/alex_vermex)\
**Post date:** [May 17, 2022, 10:05am UTC](https://discuss.elastic.co/t/initialize-counter-in-logstash-and-replace-the-message-0/304912/2 "2022-05-17T10:05:50Z")

</div>

i want to do a loop from %{[message][0]} to number\_of\_elements i did this

```auto
filter {
 mutate { split => {"message" => "|"} }
ruby {
      code => "event.set('number_of_elements', event.get('message').length)"
      }
ruby {
        code => '
            for i in number_of_elements
                k = "message#{i}"
                s = event.get(" ... ??????????")
                if s
                    event.set(k, s)
                end
            end
        '
    }
}

```

I know that my configuration was so bad but i just want to add field just make it somthing like this

```auto
mutate {
      add_field => { "DateTime" => "%{[message][x+1]}" }
      add_field => { "version" => "%{[message][x+1]}" }
} 

```

and make `x= x+1` in the loop i mean when a field added `x++` hope that is possible 😕

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [May 17, 2022, 12:25pm UTC](https://discuss.elastic.co/t/initialize-counter-in-logstash-and-replace-the-message-0/304912/3 "2022-05-17T12:25:50Z")

</div>

Can you give an example of your message? It is not clear how your message look and what you want to do.

From where are you going to get the field names?

---

<div class="post-metadata">

**Author:** ![alex\_vermex](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alex_vermex/32/101267_2.png) [@alex\_vermex](https://discuss.elastic.co/u/alex_vermex)\
**Post date:** [May 17, 2022, 1:08pm UTC](https://discuss.elastic.co/t/initialize-counter-in-logstash-and-replace-the-message-0/304912/4 "2022-05-17T13:08:05Z")

</div>

Thank you for the reply.  
yes here an example of my message

```auto
2312-15:45:43:904|V2.5.4|PAI |Gojf40555saa|Comp|Ofline|M|076|561-30687|34aa|580796|Total: 123|
2312-15:45:44:904|V2.5.4|DOW |Gsanqajkd0555saa|Commmp|Offffline|S|044|561-30687|34aa|580796|Comp|Ofline|M|076|561-30687||iapp|kkslllssTotal: 123|
...................

```

I just want to add fied for each | so i did the `split filter` so

```auto
mutate {
      add_field => { "DateTime" => "%{[message][0]}" }
      add_field => { "version" => "%{[message][1]}" }
      add_field => { "Operation" => "%{[message][2]}" }
......
    }

```

I have files that contain these logs so admin knows every field name so he will give me all possible cases so in this situation I have so many `if{}` statement for example in one log sometimes contains 40 `pipeline "|" ` sometimes 50 ... so it's a bad idea to keep doing it manually `%{[message][0]} %{[message][1]} %{[message][29]} %{[message][40]} %{[message][50]} ......`  
So i'm asking if there is any solution for this for example create a counter `x` can replace `%{[message][0]}` to `%{[message][x]} %{[message][x+1]} %{[message][x+1]}` and it should increment `x` i mean x=x+1.  
I mean if it can be like this

```auto
mutate {
      add_field => { "DateTime" => "%{[message][x]}" }
      add_field => { "version" => "%{[message][x+1]}" }
      add_field => { "Operation" => "%{[message][x+1]}" }
}
      if ([version] == V1) {
           mutate {
                add_field => { "example" => "%{[message][x+1]}" }
                           }
      }
mutate {
      add_field => { "IdMerchant" => "%{[message][x+1]}" }
    }

```

Note: I'm using filebeat to read logs and then i send the data to kafka and logstash consume this data --\> Elasticsearch --\> kibana.  
Thanks

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [May 17, 2022, 2:17pm UTC](https://discuss.elastic.co/t/initialize-counter-in-logstash-and-replace-the-message-0/304912/5 "2022-05-17T14:17:22Z")

</div>

You probably can do that with a ruby code and the ruby filter, but it is still not clear from where are you going to get the field names.

For example:

```auto
add_field => { "DateTime" => "%{[message][x]}" }

```

The field named `DateTime` is hardcoded, the same for the others fields.

The example you gave you are dynamically setting the value, but not the field name, how will you get the correct field name?

Also, does your log messages have any order? For example, if you have a message with 5 fields, and another one with 10 fields, are the fields always in the same order, like this:

```auto
field1|field2|field3|field4|field5
field1|field2|field3|field4|field5|field6|field7|field8|field9|field10

```

If you have something like that, or can change your source file to be something like that, you can easily parse your messages using the [csv filter](https://www.elastic.co/guide/en/logstash/current/plugins-filters-csv.html), you would just need to set all the possible columns in the order they appear and the filter will populate them when they exist.

---

<div class="post-metadata">

**Author:** ![alex\_vermex](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alex_vermex/32/101267_2.png) [@alex\_vermex](https://discuss.elastic.co/u/alex_vermex)\
**Post date:** [May 17, 2022, 2:43pm UTC](https://discuss.elastic.co/t/initialize-counter-in-logstash-and-replace-the-message-0/304912/6 "2022-05-17T14:43:59Z")

</div>

These are transaction logs so the developer who made the application "Gateway" writes the logs in file with the variables that made so here are the fields names comes from. So the developer who made the application who gave me the fields names.  
my log messages are not in the same order 😕  
I already tried the ruby filter but didn't know how to do it, I get the message size with

```auto
ruby {
      code => "event.set('number_of_elements', event.get('message').length)"
      }

```

but i didn't know how to initialize a variable and make it ++ i mean for example `x=x+1` in a loop or ... something like that maybe can you please correct me or if there to something else

```auto
ruby {
        code => '
            for i in number_of_elements
                k = "message#{i}"
                s = event.get(" ... ??????????")
                if s
                    event.set(k, s)
                end
            end
        '
    }

```

Thanks.

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [May 17, 2022, 3:36pm UTC](https://discuss.elastic.co/t/initialize-counter-in-logstash-and-replace-the-message-0/304912/7 "2022-05-17T15:36:07Z")

</div>

I do not know much of ruby, but to get a field from the event to the ruby code you use something like this:

```auto
fieldName_in_ruby_code = event.get("[fieldName_in_logstash]")

```

Then I think you could iterate in that array using `each_with_index`.

```auto
fieldName_in_ruby_code.each_with_index do |val, index|
    ruby code
end

```

But again, where will the field names come from? How will you know that the value of `message[2]` in one message is different from the value of `message[2]` and should be in another field?

Your examples only shows iterations over the value, not the field name.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [May 17, 2022, 3:51pm UTC](https://discuss.elastic.co/t/initialize-counter-in-logstash-and-replace-the-message-0/304912/8 "2022-05-17T15:51:51Z")

</div>

If there are a limited number of formats you may be able to [count the number of columns](https://discuss.elastic.co/t/using-ruby-to-count-the-number-of-fields-in-a-message-then-apply-csv-depending-on-the-count/177880/2) and use that to determine which csv filter to use.

---

<div class="post-metadata">

**Author:** ![alex\_vermex](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alex_vermex/32/101267_2.png) [@alex\_vermex](https://discuss.elastic.co/u/alex_vermex)\
**Post date:** [May 17, 2022, 4:16pm UTC](https://discuss.elastic.co/t/initialize-counter-in-logstash-and-replace-the-message-0/304912/9 "2022-05-17T16:16:03Z")

</div>

Thank you @leandrojmp @Badger  
yes I think I'll do that @Badger last thing take a look at that maybe it works?

```auto
filter {
mutate { split => {"message" => "|"} }
      ruby {
      code => "event.set('number_of_elements', event.get('message').length)"
      }
      ruby {
      code => "event.set('x', 0)"
      }

      ruby {
        code => '
            for i in number_of_elements
                x = x + 1
                i = i + 1
            end
        '
    }
 mutate {
      add_field => { "test_x" => "%{x}" }
      add_field => { "DateTime" => "%{[message][%{x}]}" }
      add_field => { "version" => "%{[message][x+1]}" }
      add_field => { "GateWay Operation" => "%{[message][2]}" }
    }
}

```

It gives me

```auto
"number_of_elements" => 51,
"version" => "%{[message][x+1]}",
"GateWay Operation" => "PAI ",
"test_x" => "0",
"Idtransaction" => "Gojfaaaajkl,lksq=",
"x" => 0,

```

but didn't get the field **DateTime** `Exception caught while applying mutate filter {:exception=>"Invalid FieldReference: `[message][%{x`"}`  
how can i enter the value of x under `%{[message][%{x}]}` if you have any idea please ?  
Thanks!!

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [May 17, 2022, 4:28pm UTC](https://discuss.elastic.co/t/initialize-counter-in-logstash-and-replace-the-message-0/304912/10 "2022-05-17T16:28:12Z")

</div>

It won't work this way, you need to do everything inside one ruby filter and you can't use the variables from your ruby code outside it, in the mutate filter.

The loop will only exist in inside the ruby code, so you will need to use `event.set` if you want to add fields.

But as I said, your field names are all static, it makes no sense to have a loop on the values if you are you going to statically set the field names, you need to provide more context about the name of the fields.

If you splitted your `message` into an array, you can access it using the index in any mutate filter.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [June 14, 2022, 4:28pm UTC](https://discuss.elastic.co/t/initialize-counter-in-logstash-and-replace-the-message-0/304912/11 "2022-06-14T16:28:53Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.