# Logstash parse arrays in array by grok

**URL:** <https://discuss.elastic.co/t/logstash-parse-arrays-in-array-by-grok/204777>\
**Category:** Logstash\
**Created:** [October 23, 2019, 6:04am UTC](https://discuss.elastic.co/t/logstash-parse-arrays-in-array-by-grok/204777 "2019-10-23T06:04:01Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![newkate12](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/newkate12/32/45037_2.png) [@newkate12](https://discuss.elastic.co/u/newkate12)\
**Post date:** [October 23, 2019, 6:04am UTC](https://discuss.elastic.co/t/logstash-parse-arrays-in-array-by-grok/204777/1 "2019-10-23T06:04:02Z")

</div>

Hi I'm trying to import csv and parse array data in a specific column.

My csv data is like this.

TXN\_TIME|TENANT\_ID|TAG\_ID|TAG\_VALUE  
2019-06-14 10:00:00.693|T00|GTW000000007-001-1|[[1.9744,3.0427,1.9170],[1.9743,3.0061,1.9171],[1.9109,3.0061,1.8107],[1.9109,2.9267,1.9280],[1.9109,2.9267,1.8376],[1.9243,2.9353,1.8376],[1.9243,2.9670,1.8376],[1.9243,2.9670,1.8486],[1.9316,2.9670,1.8486],[1.9402,2.9670,1.8486],[1.9524,2.9670,1.8486],[1.9597,2.9133,1.8486],[1.9597,2.9853,1.8486],  
....(so many arrays here!)......  
[1.9756,2.9560,1.7949],[1.9719,2.9560,1.7949],[1.9719,2.9585,1.9731],[1.9609,2.9585,1.9731],[1.9243,2.9560,1.9621],[1.9365,2.9585,1.8864],[1.9365,2.9573,1.8584],[1.9670,2.9573,1.8584],[1.9670,2.9536,1.8767]]

Each array element means [x\_axis, y\_axis, z\_axis], so I want to split those arrays by each axis.

This is my logstash conf file.

```
input {
  file {
    path => ["C:/Users/Desktop/3axis_MultiTagDataList-test.csv"]
    start_position => "beginning"
    sincedb_path => "nul"
  }
}

filter {
  csv {
    separator => "|"
    skip_header => true
    columns => ["TXN_TIME","TENANT_ID","TAG_ID","TAG_VALUE"]
  }

  mutate{convert=>["TENANT_ID","string"]}
  mutate{convert=>["TAG_ID","string"]}
 
  mutate{strip => ["TAG_VALUE"]}

  grok{
       break_on_match => false
       match=>{"TAG_VALUE"=>"\[(?<x_axis>\d*.\d*),(?<y_axis>\d*.\d*),(?<z_axis>\d*.\d*)\]"}
       }
  mutate {
	remove_field => ["message"]  
}
}

output {
  elasticsearch {
    hosts => "http://localhost:9200"
    index => "my_multiaxis_test_06"
  }
  stdout {
   codec => rubydebug 
  }
}

```

What I expect first was that, I could use grok match regex to directly extract x/y/z axis data from each array element, and I thought regex pattern continues to check every element array that matches [x\_axis, y\_axis, z\_axis] in "TAG\_VALUE" field by using 'break\_on\_match' set to false.

But, I saw only the first element ([1.9744,3.0427,1.9170]) is parsed into x\_axis, y\_axis, z\_axis fields.  
What's wrong with my script and how can I make grok match regex check all the arrays in array and finally collect the values in each axis field?

I want the result like this,  
"@timestamp" =\> 2019-10-23T05:52:43.495Z  
"x\_axis" =\>  
[0] =\> 1.9744  
[1] =\> 1.9109  
[2] =\> 1.9109  
...  
[255] =\> 1.9670  
"y\_axis" =\>  
[0] =\> 3.0061  
[1] =\> 3.0061  
[2] =\> 2.9267  
...  
[255] =\>2.9536  
"z\_axis" =\>  
[0] =\> 1.9171  
[1] =\> 1.8107  
[2] =\> 2.9267  
...  
[255] =\>1.8767

x\_axis, y\_axis, z\_axis fields should be double array type.  
Help me, I kinda desperate to figure out.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [October 23, 2019, 1:19pm UTC](https://discuss.elastic.co/t/logstash-parse-arrays-in-array-by-grok/204777/2 "2019-10-23T13:19:06Z")

</div>

grok does not put multiple matches into an array. However the scan method of a string does in ruby, so you can do this in a ruby filter...

```
    ruby {
        code => '
            m = event.get("TAG_VALUE").scan(/\[(\d+.\d+),(\d+.\d+),(\d+.\d+)\]/)
            x = []; y = []; z = []
            m.each { |a|
                x << a[0].to_f
                y << a[1].to_f
                z << a[2].to_f
            }
            event.set("x", x)
            event.set("y", y)
            event.set("z", z)
        '
    }

```

Error handling is left as an exercise for the reader.

---

<div class="post-metadata">

**Author:** ![newkate12](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/newkate12/32/45037_2.png) [@newkate12](https://discuss.elastic.co/u/newkate12)\
**Post date:** [October 24, 2019, 2:24am UTC](https://discuss.elastic.co/t/logstash-parse-arrays-in-array-by-grok/204777/3 "2019-10-24T02:24:39Z")

</div>

Ah it perfectly works. Thanks so much!

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [November 21, 2019, 2:24am UTC](https://discuss.elastic.co/t/logstash-parse-arrays-in-array-by-grok/204777/4 "2019-11-21T02:24:40Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.