Read CSV without introducing, and having to remove, a message field?

Elastic Stack Logstash
1

I can use Logstash to read a file of JSON Lines from stdin without introducing—and, hence, without needing to remove—a message field:

input {
  stdin {
   codec => json_lines
  }
}

(No filter section required; the config outputs to Elasticsearch.)

Is there some combination of codec and filter that I can use to read a CSV file from stdin, without having to explicitly remove the message field?

Here’s what I’m currently using to read a CSV file from stdin:

input {
  stdin {
  }
}
filter {
  csv {
    columns => [ ... ]
    remove_field => ["message"]
  }
}

I think the answer is “No”, because the csv filter requires a source field name (and the default value is "message"), but perhaps there’s a way to do it that I can’t see.

I’m currently testing in Logstash 2.4, where the default codec for stdin is line. I notice that, from Logstash 5.1 (I’m about to start using 5.2), the default codec for stdin is plain. I can’t tell from the user documentation for any of these codecs whether they output a message field: in practice, json_lines doesn’t, line does. I’d appreciate tips on that. “Read the codec source” would be a fair comment :slight_smile:.

2

Yep, you guessed it. Its not possible.
You would need a csv_lines codec.
No one has written it yet.

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.