# CSV Output not what expected

**URL:** <https://discuss.elastic.co/t/csv-output-not-what-expected/191629>\
**Category:** Logstash\
**Created:** [July 22, 2019, 11:48am UTC](https://discuss.elastic.co/t/csv-output-not-what-expected/191629 "2019-07-22T11:48:33Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![Ruben\_Aguilar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruben_aguilar/32/46500_2.png) [@Ruben\_Aguilar](https://discuss.elastic.co/u/Ruben_Aguilar)\
**Post date:** [July 22, 2019, 11:48am UTC](https://discuss.elastic.co/t/csv-output-not-what-expected/191629/1 "2019-07-22T11:48:33Z")

</div>

Hello,

I have to parse an XML file and extract certain variables from it. The parsing part is solved. And from the XML filter I get three variables in this way:

"DX\_METERNAME" =\> [  
[0] "DX\_KILOM",  
[1] "DX\_KILOM",  
[2] "DX\_KILOM"  
],  
"DX\_NEWREADING" =\> [  
[0] "26680",  
[1] "27818",  
[2] "22026"  
],  
"DX\_EXTERNAL\_REFERENCE" =\> [  
[0] "CRA20-2",  
[1] "CRA62",  
[2] "CRA66"  
]  
Now I want to save them in a CSV File with, obsiously three rows (this number may vary depending on the information in the XML). What I want is something like:

DX\_KILOM, 26680, CRA20-2  
DX\_KILOM, 27818, CRA62  
DX\_KILOM, 22026, CRA66

But the output in the file is like (in one single line):

"[""DX\_KILOM"", ""DX\_KILOM"", ""DX\_KILOM""]","[""CRA20-2"", ""CRA62"", ""CRA66""]","[""26680"", ""27818"", ""22026""]"

I have also tried using the join filter, but the result is more or less the same (everything in the same line):  
"DX\_KILOM,DX\_KILOM,DX\_KILOM","CRA20-2,CRA62,CRA66","26680,27818,22026"

Here is my code for the output (LOGSTASH Version 7.1):

```
    output{
    stdout{}
    csv{  
       fields => ["DX_METERNAME", "DX_EXTERNAL_REFERENCE" , "DX_NEWREADING"]
      
      path => "/output_path/output.csv"
       csv_options => {
            "write_headers" => true
            "headers" =>["DX_METERNAME","DX_NEWREADING","DX_EXTERNAL_REFERENCE"]
            "col_sep" => ","        	
            }
    }
    }

```

Can anyone help me?  
Thanks in advance.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [July 22, 2019, 12:59pm UTC](https://discuss.elastic.co/t/csv-output-not-what-expected/191629/2 "2019-07-22T12:59:43Z")

</div>

I would do that using ruby.

```
    ruby {
        code => '
            meterName = event.get("DX_METERNAME")
            newReading = event.get("DX_NEWREADING")
            extRef = event.get("DX_EXTERNAL_REFERENCE")
            if meterName and newReading and extRef
                a = []
                meterName.each_index { |x|
                    h = {}
                    h["DX_METERNAME"] = meterName[x]
                    h["DX_NEWREADING"] = newReading[x].to_i
                    h["DX_EXTERNAL_REFERENCE"] = extRef[x]
                    a << h
                }
                event.set("arrayOfHashes", a)
            end
        '
        remove_field => ["DX_METERNAME", "DX_NEWREADING", "DX_EXTERNAL_REFERENCE"]
    }
    split { field => "arrayOfHashes" }
    ruby {
        code => '
            event.get("arrayOfHashes").each { |k, v|
                event.set(k, v)
            }
            event.remove("arrayOfHashes")
        '
    }
```

---

<div class="post-metadata">

**Author:** ![Ruben\_Aguilar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruben_aguilar/32/46500_2.png) [@Ruben\_Aguilar](https://discuss.elastic.co/u/Ruben_Aguilar)\
**Post date:** [July 23, 2019, 11:36am UTC](https://discuss.elastic.co/t/csv-output-not-what-expected/191629/3 "2019-07-23T11:36:50Z")

</div>

First of all, tanks for your quick response. The code works perfectly when there is no headers. But when I add the header option this is the output in the csv file:

```
DX_METERNAME,DX_EXTERNAL_REFERENCE,DX_NEWREADING
DX_KILOM,CRA20-2,26680
DX_METERNAME,DX_EXTERNAL_REFERENCE,DX_NEWREADING
DX_KILOM,CRA62,27818
DX_METERNAME,DX_EXTERNAL_REFERENCE,DX_NEWREADING
DX_KILOM,CRA66,22026

```

I really don't undersand why it inserts the header between every line.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [July 23, 2019, 12:41pm UTC](https://discuss.elastic.co/t/csv-output-not-what-expected/191629/4 "2019-07-23T12:41:10Z")

</div>

> [@Ruben\_Aguilar](#):
>
> I really don't undersand why it inserts the header between every line.

It is just the way it works. Each event is output by a separate call to [to\_csv](https://apidock.com/ruby/Array/to_csv). If csv\_options says to add a header then every call will add it and you will get one per event.

---

<div class="post-metadata">

**Author:** ![Ruben\_Aguilar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruben_aguilar/32/46500_2.png) [@Ruben\_Aguilar](https://discuss.elastic.co/u/Ruben_Aguilar)\
**Post date:** [July 24, 2019, 6:52am UTC](https://discuss.elastic.co/t/csv-output-not-what-expected/191629/5 "2019-07-24T06:52:26Z")

</div>

Thank you, very much. Very helpfull!

I am still thinking this CSV output feature should create a new row for every element in the arrays in "fields" section.

This could be a new feature.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [August 21, 2019, 6:52am UTC](https://discuss.elastic.co/t/csv-output-not-what-expected/191629/6 "2019-08-21T06:52:28Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.