Unable to output in csv

Elastic Stack Logstash
1

I am trying to output CSV of select fields but I am not getting the correct data even with a simplified config
My config:

input {
generator {
add_field => ["foo","bar"]
count => 1
}
}
output {
stdout { codec => rubydebug }
csv {
path => "test.csv"
fields => ["foo"]
}
}

However my test.only ever contains:
2016-11-03T11:07:25.878Z OSTML0204389 Hello world!

I am expecting it to contain "bar"

Thanks
Steve

2

How about having the path outside the csv tag like this within the file tag:

file {
	path => "yourpath/output.csv"
      }

Remove the csv tag and give it a go. Try inserting your fields parameter within the file tag if this works.

3

Using just file:

input {
        generator {
          add_field => ["foo","bar"]
          count => 1
        }
      }
output {
stdout { codec => rubydebug }
  file {
    path => "test.csv"
  }
}

produces output:

{"sequence":0,"@timestamp":"2016-11-03T12:00:07.986Z","foo":"bar","@version":"1","host":"OSTML0204389","message":"Hello world!"}

adding fields to a file tag causes a config exception. I really need the csv output as I need to transform select fields into a new csv with a different separator character.

4

So in that case your earlier config seems ok according to this if you're using ELK 5.0.

So you're saying it doesn't feed anything into the csv file? Or you're getting any kind of errors?

5

Yes using logstash-5.0.0. No Errors are given.

it creates test.csv but contains a single line:

2016-11-03T12:09:25.100Z OSTML0204389 Hello world!

The CSV plugin is picking up the option:

[2016-11-03T12:09:25,040][DEBUG][logstash.outputs.csv ] config LogStash::Outputs::CSV/@fields = ["foo"]

6

If I generate 5 messages (by changing count to 5) I just get a single long line, so it is not even adding newlines:

2016-11-03T12:14:12.816Z OSTML0204389 Hello world!2016-11-03T12:14:12.837Z OSTML0204389 Hello world!2016-11-03T12:14:12.840Z OSTML0204389 Hello world!2016-11-03T12:14:12.842Z OSTML0204389 Hello world!2016-11-03T12:14:12.844Z OSTML0204389 Hello world!

As far as I can tell the logstash-output-csv plugin is not compatible with logstash 5. It defines a receive event that is never called. If I look at the how to define a logstash output plugin it uses multi_receive. The examples config I have been testing is from the output-csv unit tests. My confidence in using logstash with even its shipped plugins in production has been damaged.

From reading the code for the file ouput plugin it looks like a better way to do CSV output might be to write a new csv codec instead. I will go dig out a ruby tutorial...