IMAP input CSV filter

Elastic Stack Logstash
1

Hi there,

I have setup IMAP as an input - this works fine as I get the email.

The e-mail content is plain text and a simple CSV file: eg

Thu Feb 09 21:01:04 2017,28.6,87.2,Fridge 10283739,2.7895,-54
Thu Feb 09 21:15:04 2017,28.6,87.2,Fridge 10283739,2.7895,-54
Thu Feb 09 21:30:04 2017,2.6,3.2,Fridge 10283739,2.7895,-54

But when I process this only the first row gets processed:

         "date" => "Fri, 30 Jun 2017 16:19:37 +0000",
 "temp_reading" => "28.6",
         "batt" => "2.7895",
     "@version" => "1",
  "hum_reading" => "87.2",
         "rssi" => "-54",
      "message" => "Thu Feb 09 21:01:04 2017,28.6,87.2,Fridge 10283739,2.7895,-54\nThu Feb 09 21:15:04 2017,28.6,87.2,Fridge 10283739,2.7895,-54\nThu Feb 09 21:30:04 2017,2.6,3.2,Fridge 10283739,2.7895,-54\n\n",

"@timestampInEmail" => 2017-02-09T21:01:04.000Z,
"@timestamp" => 2017-06-30T16:19:37.000Z,
"timestampInEmail" => "Thu Feb 09 21:01:04 2017",
"location" => "Fridge 10283739"

Am I missing a config somewhere to allow this to process the entire message content?

filter {

csv {
    columns => ["timestampInEmail","temp_reading","hum_reading","location","batt","rssi"]
    separator => ","
}

##
## Remove fields
##
mutate {
    remove_field => ["message-id", "received", "content-type", 
    "list-unsubscribe", "dkim-signature", "x-msfbl", "received-spf", 
    "authentication-results", "mime-version", "return-path", "x-zohomail-sender",
    "content-transfer-encoding", "delivered-to", "x-zohomail", "x-zohomail-owner",
    "list-id", "arc-message-signature","subject", "x-received","to", "from","arc-authentication-results",
    "arc-seal"]
}

date {
    locale => "en"
    match => ["timestampInEmail", "EEE MMM dd HH:mm:ss yyyy"]
    timezone => "Europe/London"
    target => "@timestampInEmail"
}

}

Any assistance will be much appreciated.

Thanks,

2

You need to split your single event into multiple events (one per line of CSV). You can use the split filter for that.

3

Thanks!

That was easy if you know how :slight_smile:

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.