# Parsing csv file through Logstash

**URL:** <https://discuss.elastic.co/t/parsing-csv-file-through-logstash/274792>\
**Category:** Logstash\
**Created:** [June 2, 2021, 6:26pm UTC](https://discuss.elastic.co/t/parsing-csv-file-through-logstash/274792 "2021-06-02T18:26:55Z")\
**Posts on this page:** 19\
**Page:** 1

<div class="post-metadata">

**Author:** ![zaeemmasood](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/zaeemmasood/32/102383_2.png) [@zaeemmasood](https://discuss.elastic.co/u/zaeemmasood)\
**Post date:** [June 2, 2021, 6:26pm UTC](https://discuss.elastic.co/t/parsing-csv-file-through-logstash/274792/1 "2021-06-02T18:26:55Z")

</div>

Hi All,

I am running ELK stack 7.6.2. I need to parse a csv through Logstash. My logstash conf file looks like this:

```auto
input {
  file {
    path => "/opt/gtal/ictal/elasticsearch/app/logstash/stage/STATS-01062021.txt"
    start_position => "beginning"
    sincedb_path => "/dev/null"
  }
}
filter {
  csv {
      separator => ","
      skip_header => "true"
      columns => ["Total call statistics for each process","User call statistics for each process"]
  }
}
output {
      file {
       path => "/opt/gtal/ictal/elasticsearch/logs/maa/rubydebug.txt"
       codec => rubydebug
     }

    elasticsearch {
     hosts => ["xx.xx.xx.xxx:3045"]
     user => "xxxxxxxxxx"
     password => "xxxxxxxxxxxxxx"
         index => "demo-csv-%{+YYYY.MM.dd}"

  }
}

```

While I am able to display the result in Kibana, the data is spread all over. The CSV file is also complicated. Please see below:

```auto
Total call statistics for each process:
{'BServer': 3105489,
 'CServer': 10146,
 'CMngr': 5679760,
 'CiCServer-19000101:19951231': 0,
 'CiCServer-20191001:20191231': 34,
 'CiCServer-20200101:20200331': 114134,
 'CiCServer-20200401:20200630': 4967513,
 'CiCServer-20200701:20200930': 4968153,
 'CiCServer-20201001:20201231': 5057632,
 'CiCServer-20210101:20210331': 5158988,
 'CiCServer-20210401:20210630': 5351872,
 'CiCServer-20210701:20210930': 0,
 'CiCServer-20211001:20211231': 0,
 'CfgServer': 52,
 'CRskhist': 3548,
 'gccdServer': 6,
 'gfnServer': 1328904,
 'kServer': 14363,
 'madServer': 11045554,
 'mmcServer': 6,
 'qsServer': 6,
 'RrServer': 681,
 'StnServer': 194801}
User call statistics for each process:
{'BServer': {'': 7.0,
               'ccvma': 153766.0,
               'ccvrts': 2951576.0,
               'fqenv': 3.0,
               'frtbU': 137.0},
 'CServer': {'': 3.0, 'ccvma': 10140.0, 'fqenv': 3.0},
 'CMngr': {'': 182.0,
                        'ccvma': 190594.0,
                        'ccvrts': 5488978.0,
                        'fqenv': 6.0},
 'CiCServer-19000101:19951231': {},
 'CiCServer-19960101:20001231': {},
 'CiCServer-20010101:20011231': {},
 'CiCServer-20190101:20190331': {'': 9.0, 'ccvma': 9.0},
 'CiCServer-20190401:20190630': {'': 16.0, 'ccvma': 18.0},
 'CiCServer-20190701:20190930': {'': 16.0, 'ccvma': 18.0},
 'CiCServer-20191001:20191231': {'': 16.0, 'ccvma': 18.0},
 'CiCServer-20200101:20200331': {'': 16.0,
                                         'ccvma': 1768.0,
                                         'ccvrts': 112350.0},
 'CiCServer-20200401:20200630': {'': 11.0,
                                         'ccvma': 45092.0,
                                         'ccvrts': 4922410.0},
 'CiCServer-20200701:20200930': {'': 11.0,
                                         'ccvma': 45645.0,
                                         'ccvrts': 4922497.0},
 'CiCServer-20201001:20201231': {'': 16.0,
                                         'ccvma': 4074128.0,
                                         'ccvrts': 983488.0},
 'CiCServer-20210101:20210331': {'': 16.0,
                                         'ccvma': 4824540.0,
                                         'ccvrts': 334432.0},
 'CiCServer-20210401:20210630': {'': 7341.0,
                                         'ccvma': 4683996.0,
                                         'ccvrts': 660535.0},
 'CiCServer-20210701:20210930': {},
 'CiCServer-20211001:20211231': {},
 'CfgServer': {'': 6.0, 'ccvrts': 40.0, 'fqenv': 6.0},
 'CRskhist': {'': 3.0, 'ccvma': 3542.0, 'fqenv': 3.0},
 'gccdServer': {'': 3.0, 'fqenv': 3.0},
 'gfnServer': {'': 433.0,
                  'ccvrts': 1328038.0,
                  'fqenv': 3.0,
                  'nameservice': 430.0},
 'kServer': {'': 3.0, 'ccvrts': 14357.0, 'fqenv': 3.0},
 'madServer': {'': 162.0,
               'ccvma': 358873.0,
               'ccvrts': 10686372.0,
               'fqenv': 9.0,
               'frtbU': 138.0},
 'mmcServer': {'': 3.0, 'fqenv': 3.0},
 'qsServer': {'': 3.0, 'fqenv': 3.0},
 'RrServer': {'': 3.0, 'ccvma': 675.0, 'fqenv': 3.0},
 'StnServer': {'': 3.0, 'ccvrts': 194795.0, 'fqenv': 3.0}}

```

Please guide me as I am new to this. thanks

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [June 2, 2021, 7:24pm UTC](https://discuss.elastic.co/t/parsing-csv-file-through-logstash/274792/2 "2021-06-02T19:24:54Z")

</div>

> [@zaeemmasood](#):
>
> ```auto
> Total call statistics for each process:
> {'BServer': 3105489,
> 'CServer': 10146,
> 'CMngr': 5679760,
> 'CiCServer-19000101:19951231': 0,
> 'CiCServer-20191001:20191231': 34,
> 'CiCServer-20200101:20200331': 114134,
> 'CiCServer-20200401:20200630': 4967513,
> 'CiCServer-20200701:20200930': 4968153,
> 'CiCServer-20201001:20201231': 5057632,
> 'CiCServer-20210101:20210331': 5158988,
> 'CiCServer-20210401:20210630': 5351872,
> 'CiCServer-20210701:20210930': 0,
> 'CiCServer-20211001:20211231': 0,
> 'CfgServer': 52,
> 'CRskhist': 3548,
> 'gccdServer': 6,
> 'gfnServer': 1328904,
> 'kServer': 14363,
> 'madServer': 11045554,
> 'mmcServer': 6,
> 'qsServer': 6,
> 'RrServer': 681,
> 'StnServer': 194801}
> User call statistics for each process:
> {'BServer': {'': 7.0,
> 'ccvma': 153766.0,
> 'ccvrts': 2951576.0,
> 'fqenv': 3.0,
> 'frtbU': 137.0},
> 'CServer': {'': 3.0, 'ccvma': 10140.0, 'fqenv': 3.0},
> 'CMngr': {'': 182.0,
> 'ccvma': 190594.0,
> 'ccvrts': 5488978.0,
> 'fqenv': 6.0},
> 'CiCServer-19000101:19951231': {},
> 'CiCServer-19960101:20001231': {},
> 'CiCServer-20010101:20011231': {},
> 'CiCServer-20190101:20190331': {'': 9.0, 'ccvma': 9.0},
> 'CiCServer-20190401:20190630': {'': 16.0, 'ccvma': 18.0},
> 'CiCServer-20190701:20190930': {'': 16.0, 'ccvma': 18.0},
> 'CiCServer-20191001:20191231': {'': 16.0, 'ccvma': 18.0},
> 'CiCServer-20200101:20200331': {'': 16.0,
> 'ccvma': 1768.0,
> 'ccvrts': 112350.0},
> 'CiCServer-20200401:20200630': {'': 11.0,
> 'ccvma': 45092.0,
> 'ccvrts': 4922410.0},
> 'CiCServer-20200701:20200930': {'': 11.0,
> 'ccvma': 45645.0,
> 'ccvrts': 4922497.0},
> 'CiCServer-20201001:20201231': {'': 16.0,
> 'ccvma': 4074128.0,
> 'ccvrts': 983488.0},
> 'CiCServer-20210101:20210331': {'': 16.0,
> 'ccvma': 4824540.0,
> 'ccvrts': 334432.0},
> 'CiCServer-20210401:20210630': {'': 7341.0,
> 'ccvma': 4683996.0,
> 'ccvrts': 660535.0},
> 'CiCServer-20210701:20210930': {},
> 'CiCServer-20211001:20211231': {},
> 'CfgServer': {'': 6.0, 'ccvrts': 40.0, 'fqenv': 6.0},
> 'CRskhist': {'': 3.0, 'ccvma': 3542.0, 'fqenv': 3.0},
> 'gccdServer': {'': 3.0, 'fqenv': 3.0},
> 'gfnServer': {'': 433.0,
> 'ccvrts': 1328038.0,
> 'fqenv': 3.0,
> 'nameservice': 430.0},
> 'kServer': {'': 3.0, 'ccvrts': 14357.0, 'fqenv': 3.0},
> 'madServer': {'': 162.0,
> 'ccvma': 358873.0,
> 'ccvrts': 10686372.0,
> 'fqenv': 9.0,
> 'frtbU': 138.0},
> 'mmcServer': {'': 3.0, 'fqenv': 3.0},
> 'qsServer': {'': 3.0, 'fqenv': 3.0},
> 'RrServer': {'': 3.0, 'ccvma': 675.0, 'fqenv': 3.0},
> 'StnServer': {'': 3.0, 'ccvrts': 194795.0, 'fqenv': 3.0}}
> 
> ```

Is that the contents of your file? It is not a CSV.

---

<div class="post-metadata">

**Author:** ![zaeemmasood](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/zaeemmasood/32/102383_2.png) [@zaeemmasood](https://discuss.elastic.co/u/zaeemmasood)\
**Post date:** [June 2, 2021, 8:36pm UTC](https://discuss.elastic.co/t/parsing-csv-file-through-logstash/274792/3 "2021-06-02T20:36:09Z")

</div>

Yes you are right. It is a text file. Sorry for misleading. Is there a way to parse this using logstash?

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [June 2, 2021, 9:15pm UTC](https://discuss.elastic.co/t/parsing-csv-file-through-logstash/274792/4 "2021-06-02T21:15:27Z")

</div>

> [@zaeemmasood](#):
>
> Is there a way to parse this using logstash?

No problem. Ingest the entire file as a single event. To do that use a file input with a multiline codec that never matches the data in the file

```
codec => multiline { pattern => "^Spalanzani" negate => true what => previous auto_flush_interval => 1 multiline_tag => "" }

```

Then chop the data into two parts using grok. Save these under [@metadata] so that they are available in logstash but not indexed in elasticsearch. Then convert the data to JSON (which requires double quotes) and parse

```
    grok { match => { "message" => "Total call statistics for each process:%{GREEDYDATA:[@metadata][totalCalls]}User call statistics for each process:%{GREEDYDATA:[@metadata][userCalls]}" } remove_field => ["message"] }
    mutate {
        gsub => [
            "[@metadata][totalCalls]", "'", '"',
            "[@metadata][userCalls]", "'", '"'
        ]
    }
    json { source => "[@metadata][userCalls]" target => "userCalls" }
    json { source => "[@metadata][totalCalls]" target => "totalCalls" }

```

That will result in an event with

```
"totalCalls" => {
                      "mmcServer" => 6,
    "CiCServer-20200701:20200930" => 4968153,
    "CiCServer-20211001:20211231" => 0,
                        "BServer" => 3105489,
...
},
 "userCalls" => {
                      "mmcServer" => {
             "" => 3.0,
        "fqenv" => 3.0
    },
    "CiCServer-20200701:20200930" => {
              "" => 11.0,
        "ccvrts" => 4922497.0,
         "ccvma" => 45645.0
    },

```

"" is a valid field name in JSON, but a lot of things object to empty names. I think elasticsearch may be one, so you might get a mapping exception. If so, you could insert

```
"[@metadata][userCalls]", "''", "'empty'",

```

at the top of the list of gsubs in the mutate filter.

---

<div class="post-metadata">

**Author:** ![tmp13](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tmp13/32/78005_2.png) [@tmp13](https://discuss.elastic.co/u/tmp13)\
**Post date:** [June 3, 2021, 6:34am UTC](https://discuss.elastic.co/t/parsing-csv-file-through-logstash/274792/5 "2021-06-03T06:34:55Z")

</div>

Hi. Its so hard....  
Topic creator have json, but yes he must del  
_"Total call statistics for each process:"_ and _"User call statistics for each process:"_  
from this file and replace **'** to **"**  
Simple way do it in bash (if its on linux)  
`cat YouFile.txt |awk '{if(!index($0,"call statistics for each process")){gsub(/\x027/,"\"",$0);print $0}}' >> jsonfile.txt`

End next if logstash use some input with  
`codec="json"`

---

<div class="post-metadata">

**Author:** ![zaeemmasood](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/zaeemmasood/32/102383_2.png) [@zaeemmasood](https://discuss.elastic.co/u/zaeemmasood)\
**Post date:** [June 3, 2021, 8:24pm UTC](https://discuss.elastic.co/t/parsing-csv-file-through-logstash/274792/6 "2021-06-03T20:24:26Z")

</div>

Hello Badger,

Thanks! It worked as expected after I changed as per your recommendations

---

<div class="post-metadata">

**Author:** ![zaeemmasood](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/zaeemmasood/32/102383_2.png) [@zaeemmasood](https://discuss.elastic.co/u/zaeemmasood)\
**Post date:** [June 4, 2021, 6:20pm UTC](https://discuss.elastic.co/t/parsing-csv-file-through-logstash/274792/7 "2021-06-04T18:20:14Z")

</div>

Hi,

I was wondering if it is possible to assign a generic variable name say "serverName" and assign the values to it? e.g:

serverName: BServer  
serverName: CServer  
serverName: CMngr  
serverName: CiCServer-19000101:19951231  
...................  
and so on.

This would perhaps help me in creating a visualization chart based on individual server.

```auto
Total call statistics for each process:
{'BServer': 3105489,
 'CServer': 10146,
 'CMngr': 5679760,
 'CiCServer-19000101:19951231': 0,
 'CiCServer-20191001:20191231': 34,
 'CiCServer-20200101:20200331': 114134,
 'CiCServer-20200401:20200630': 4967513,
 'CiCServer-20200701:20200930': 4968153,
 'CiCServer-20201001:20201231': 5057632,
 'CiCServer-20210101:20210331': 5158988,
 'CiCServer-20210401:20210630': 5351872,
 'CiCServer-20210701:20210930': 0,
 'CiCServer-20211001:20211231': 0,
 'CfgServer': 52,
 'CRskhist': 3548,
 'gccdServer': 6,
 'gfnServer': 1328904,
 'kServer': 14363,
 'madServer': 11045554,
 'mmcServer': 6,
 'qsServer': 6,
 'RrServer': 681,
 'StnServer': 194801}
User call statistics for each process:
{'BServer': {'': 7.0,
               'ccvma': 153766.0,
               'ccvrts': 2951576.0,
               'fqenv': 3.0,
               'frtbU': 137.0},
 'CServer': {'': 3.0, 'ccvma': 10140.0, 'fqenv': 3.0},
 'CMngr': {'': 182.0,
                        'ccvma': 190594.0,
                        'ccvrts': 5488978.0,
                        'fqenv': 6.0},
 'CiCServer-19000101:19951231': {},
 'CiCServer-19960101:20001231': {},
 'CiCServer-20010101:20011231': {},
 'CiCServer-20190101:20190331': {'': 9.0, 'ccvma': 9.0},
 'CiCServer-20190401:20190630': {'': 16.0, 'ccvma': 18.0},
 'CiCServer-20190701:20190930': {'': 16.0, 'ccvma': 18.0},
 'CiCServer-20191001:20191231': {'': 16.0, 'ccvma': 18.0},
 'CiCServer-20200101:20200331': {'': 16.0,
                                         'ccvma': 1768.0,
                                         'ccvrts': 112350.0},
 'CiCServer-20200401:20200630': {'': 11.0,
                                         'ccvma': 45092.0,
                                         'ccvrts': 4922410.0},
 'CiCServer-20200701:20200930': {'': 11.0,
                                         'ccvma': 45645.0,
                                         'ccvrts': 4922497.0},
 'CiCServer-20201001:20201231': {'': 16.0,
                                         'ccvma': 4074128.0,
                                         'ccvrts': 983488.0},
 'CiCServer-20210101:20210331': {'': 16.0,
                                         'ccvma': 4824540.0,
                                         'ccvrts': 334432.0},
 'CiCServer-20210401:20210630': {'': 7341.0,
                                         'ccvma': 4683996.0,
                                         'ccvrts': 660535.0},
 'CiCServer-20210701:20210930': {},
 'CiCServer-20211001:20211231': {},
 'CfgServer': {'': 6.0, 'ccvrts': 40.0, 'fqenv': 6.0},
 'CRskhist': {'': 3.0, 'ccvma': 3542.0, 'fqenv': 3.0},
 'gccdServer': {'': 3.0, 'fqenv': 3.0},
 'gfnServer': {'': 433.0,
                  'ccvrts': 1328038.0,
                  'fqenv': 3.0,
                  'nameservice': 430.0},
 'kServer': {'': 3.0, 'ccvrts': 14357.0, 'fqenv': 3.0},
 'madServer': {'': 162.0,
               'ccvma': 358873.0,
               'ccvrts': 10686372.0,
               'fqenv': 9.0,
               'frtbU': 138.0},
 'mmcServer': {'': 3.0, 'fqenv': 3.0},
 'qsServer': {'': 3.0, 'fqenv': 3.0},
 'RrServer': {'': 3.0, 'ccvma': 675.0, 'fqenv': 3.0},
 'StnServer': {'': 3.0, 'ccvrts': 194795.0, 'fqenv': 3.0}}

```

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [June 4, 2021, 6:30pm UTC](https://discuss.elastic.co/t/parsing-csv-file-through-logstash/274792/8 "2021-06-04T18:30:15Z")

</div>

> [@zaeemmasood](#):
>
> I was wondering if it is possible to assign a generic variable name say "serverName" and assign the values to it?

Almost certainly. But instead of

```
"userCalls" => {
                      "mmcServer" => {
             "" => 3.0,
        "fqenv" => 3.0
    },
    "CiCServer-20200701:20200930" => {
              "" => 11.0,
        "ccvrts" => 4922497.0,
         "ccvma" => 45645.0
    },

```

etc. what do you want the data to look like?

---

<div class="post-metadata">

**Author:** ![zaeemmasood](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/zaeemmasood/32/102383_2.png) [@zaeemmasood](https://discuss.elastic.co/u/zaeemmasood)\
**Post date:** [June 4, 2021, 6:53pm UTC](https://discuss.elastic.co/t/parsing-csv-file-through-logstash/274792/9 "2021-06-04T18:53:53Z")

</div>

They could be:

serverName: mmcServer.empty  
serverName: mmcServer.fqenv

serverName: CiCServer-20200701:20200930.ccvrts  
serverName: CiCServer-20200701:20200930.ccvma

Hope I understood your question correctly.

Thanks

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [June 4, 2021, 7:58pm UTC](https://discuss.elastic.co/t/parsing-csv-file-through-logstash/274792/10 "2021-06-04T19:58:27Z")

</div>

But if serverName is the key then where does the value go? Here is one option for what you could do -- one event per server. Change the json filter to store the parsed data in [@metadata] and then use ruby

```
    json { source => "[@metadata][userCalls]" target => "[@metadata][parsedUserCalls]" }
    json { source => "[@metadata][totalCalls]" target => "[@metadata][parsedTotalCalls]" }
    ruby {
        code => '
            a = []
            userCalls = event.get("[@metadata][parsedUserCalls]")
            totalCalls = event.get("[@metadata][parsedTotalCalls]")
            totalCalls.each { |k, v|
                server = {}
                server["serverName"] = k
                server["totalCalls"] = v
                server["userCalls"] = userCalls[k]
                a << server
            }
            event.set("servers", a)
        '
    }

```

That will get you

```
   "servers" => [
    [0] {
         "userCalls" => {},
        "serverName" => "CiCServer-20210701:20210930",
        "totalCalls" => 0
    },
    [1] {
         "userCalls" => {
            "empty" => 3.0,
            "fqenv" => 3.0
        },
        "serverName" => "gccdServer",
        "totalCalls" => 6
    },

```

etc. You might want to use a split filter to separate each filter into its own event, and then use [this](https://discuss.elastic.co/t/how-to-dynamically-move-nested-key-value-to-root-level/180006/2) to move the contents of [server] to the root.

But do not let my guesses drive your data design. You need to describe exactly how you want they keys and values to appear.

---

<div class="post-metadata">

**Author:** ![zaeemmasood](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/zaeemmasood/32/102383_2.png) [@zaeemmasood](https://discuss.elastic.co/u/zaeemmasood)\
**Post date:** [June 7, 2021, 2:37pm UTC](https://discuss.elastic.co/t/parsing-csv-file-through-logstash/274792/11 "2021-06-07T14:37:48Z")

</div>

Thanks. My aim is to create a visualization based on the output, regardless of how the data is manipulated/ displayed.

I will try your suggestion.

---

<div class="post-metadata">

**Author:** ![zaeemmasood](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/zaeemmasood/32/102383_2.png) [@zaeemmasood](https://discuss.elastic.co/u/zaeemmasood)\
**Post date:** [June 7, 2021, 8:18pm UTC](https://discuss.elastic.co/t/parsing-csv-file-through-logstash/274792/12 "2021-06-07T20:18:06Z")

</div>

> [@Badger](#):
>
> You might want to use a split filter to separate each filter into its own event

Hi Badger, I am struggling with getting this part done... I tried various options but it did not work out.

Can you please guide?

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [June 7, 2021, 8:34pm UTC](https://discuss.elastic.co/t/parsing-csv-file-through-logstash/274792/13 "2021-06-07T20:34:57Z")

</div>

If you have a top level field called servers which is an array then

```
split { field => "servers" }

```

will result in as many events as there are entries in the servers array, each containing a field called servers that contains one of those entries.

---

<div class="post-metadata">

**Author:** ![zaeemmasood](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/zaeemmasood/32/102383_2.png) [@zaeemmasood](https://discuss.elastic.co/u/zaeemmasood)\
**Post date:** [June 7, 2021, 9:30pm UTC](https://discuss.elastic.co/t/parsing-csv-file-through-logstash/274792/14 "2021-06-07T21:30:45Z")

</div>

So this is how the logtsash config looks now:

```auto
input {
  file {
    path => "/opt/gportal/icgportal/elasticsearch/app/logstash/stage/CVMQA_UAT_STATS-*.txt"
    codec => multiline { pattern => "^Spalanzani" negate => true what => previous auto_flush_interval => 1 multiline_tag => "" }
    start_position => "beginning"
    sincedb_path => "/dev/null"
  }
}
filter {
  csv {
      separator => ","
      skip_header => "true"
      columns => ["Total call statistics for each process","User call statistics for each process"]
  }
  grok { match => { "message" => "Total call statistics for each process:%{GREEDYDATA:[@metadata][totalCalls]}User call statistics for each process:%{GREEDYDATA:[@metadata][userCalls]}" } r
emove_field => ["message"] }
    mutate {
        gsub => [
            "[@metadata][userCalls]", "''", "'empty'",
            "[@metadata][totalCalls]", "'", '"',
            "[@metadata][userCalls]", "'", '"'
        ]
    }
        json { source => "[@metadata][userCalls]" target => "[@metadata][parsedUserCalls]" }
        json { source => "[@metadata][totalCalls]" target => "[@metadata][parsedTotalCalls]" }
		ruby {
            code => '
                a = []
                userCalls = event.get("[@metadata][parsedUserCalls]")
                totalCalls = event.get("[@metadata][parsedTotalCalls]")
                totalCalls.each { |k, v|
                    server = {}
                    server["serverName"] = k
                    server["totalCalls"] = v
                    server["userCalls"] = userCalls[k]
                    a << server
            }
            event.set("servers", a)
        '
    }
	
}
output {
      file {
       path => "/opt/gportal/icgportal/elasticsearch/logs/mqa/rubydebug.txt"
       codec => rubydebug
     }

    elasticsearch {
     hosts => ["xx.xx.xxx.xxx:3045"]
     user => "elastic"
     password => "xxxxxxxxxxxxxxxxxxx"
         index => "demo-csv-%{+YYYY.MM.dd}"

  }
}

```

Where can I add `split { field => "servers" }` ?

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [June 7, 2021, 9:59pm UTC](https://discuss.elastic.co/t/parsing-csv-file-through-logstash/274792/15 "2021-06-07T21:59:14Z")

</div>

After the ruby filter.

---

<div class="post-metadata">

**Author:** ![zaeemmasood](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/zaeemmasood/32/102383_2.png) [@zaeemmasood](https://discuss.elastic.co/u/zaeemmasood)\
**Post date:** [June 11, 2021, 5:17pm UTC](https://discuss.elastic.co/t/parsing-csv-file-through-logstash/274792/16 "2021-06-11T17:17:44Z")

</div>

Thanks Badger!

---

<div class="post-metadata">

**Author:** ![zaeemmasood](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/zaeemmasood/32/102383_2.png) [@zaeemmasood](https://discuss.elastic.co/u/zaeemmasood)\
**Post date:** [June 11, 2021, 8:23pm UTC](https://discuss.elastic.co/t/parsing-csv-file-through-logstash/274792/17 "2021-06-11T20:23:40Z")

</div>

Hi Badger. Thanks for the help. Fields now start to look fine. When I try to replace the txt file with a new one I get the following error in the log:

```auto
[2021-06-11T16:16:39,863][ERROR][logstash.filters.ruby][main] Ruby exception occurred: undefined method `each' for nil:NilClass
[2021-06-11T16:16:39,878][WARN][logstash.filters.split][main] Only String and Array types are splittable. field:servers is of type = NilClass
[2021-06-11T16:16:39,882][ERROR][logstash.filters.ruby][main] Ruby exception occurred: undefined method `each' for nil:NilClass

```

My config file looks like following:

```auto
input {
  file {
    path => "/opt/gtal/iptal/elasticsearch/app/logstash/stage/CVMQA_UAT_STATS-*.txt"
    codec => multiline { pattern => "^Spalanzani" negate => true what => previous auto_flush_interval => 1 multiline_tag => "" }
    start_position => "beginning"
    sincedb_path => "/dev/null"
  }
}
filter {
  csv {
      separator => ","
      skip_header => "true"
      columns => ["Total call statistics for each process","User call statistics for each process"]
  }
grok { match => { "message" => "Total call statistics for each process:%{GREEDYDATA:[@metadata][totalCalls]}User call statistics for each process:%{GREEDYDATA:[@metadata][userCalls]}" } rem
ove_field => ["message"] }
    mutate {
        remove_field => ["host"]
        gsub => [
            "[@metadata][userCalls]", "''", "'empty'",
            "[@metadata][totalCalls]", "'", '"',
            "[@metadata][userCalls]", "'", '"'
        ]
    }
        json { source => "[@metadata][userCalls]" target => "[@metadata][parsedUserCalls]" }
        json { source => "[@metadata][totalCalls]" target => "[@metadata][parsedTotalCalls]" }
        ruby {
            code => '
                a = []
                userCalls = event.get("[@metadata][parsedUserCalls]")
                totalCalls = event.get("[@metadata][parsedTotalCalls]")
                totalCalls.each { |k, v|
                    server = {}
                    server["serverName"] = k
                    server["totalCalls"] = v
                    server["userCalls"] = userCalls[k]
                    a << server
            }
            event.set("servers", a)
        '
    }
        split { field => "servers" }

        ruby {
        code => '
            event.get("servers").each { |k, v|
                event.set(k,v)
            }
            event.remove("servers")
        '
    }

            }

output {
      file {
       path => "/opt/gtal/iptal/elasticsearch/logs/mqa/rubydebug.txt"
       codec => rubydebug
     }

    elasticsearch {
     hosts => ["xx-xxx-xxx:3045"]
     user => "xxxxxx"
     password => "xxxxxxxxxxxxxxx"
         index => "demo-csv-%{+YYYY.MM.dd}"

  }
}

```

I also tried to add the following in the config as per `https://discuss.elastic.co/t/only-string-and-array-types-are-splittable-field-splitedjson-is-of-type-nilclass/191725/2`

```auto
if [message] drop {}

```

but still the error remains.

Please guide.

Thanks

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [June 11, 2021, 9:01pm UTC](https://discuss.elastic.co/t/parsing-csv-file-through-logstash/274792/18 "2021-06-11T21:01:28Z")

</div>

The first "undefined method" error is happening for one of

```
            userCalls = event.get("[@metadata][parsedUserCalls]")
            totalCalls = event.get("[@metadata][parsedTotalCalls]")

```

Once a ruby filter gets an exception the rest of the code is not executed (unless you catch the exception), so it never does

```
        event.set("servers", a)

```

That cause the split to fail, because the field it is trying to split is nil. Similarly for the ruby filter that moves things from the non-existent [servers] field.

The example code I wrote contains no error handling, including not verifying that data exists before trying to use it. In the real world you need to add the checks to handle missing data.

I suspect the grok is failing for your new file.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 9, 2021, 9:02pm UTC](https://discuss.elastic.co/t/parsing-csv-file-through-logstash/274792/19 "2021-07-09T21:02:14Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.