IMAP input CSV filter

Hi there,

I have setup IMAP as an input - this works fine as I get the email.

The e-mail content is plain text and a simple CSV file: eg

Thu Feb 09 21:01:04 2017,28.6,87.2,Fridge 10283739,2.7895,-54
Thu Feb 09 21:15:04 2017,28.6,87.2,Fridge 10283739,2.7895,-54
Thu Feb 09 21:30:04 2017,2.6,3.2,Fridge 10283739,2.7895,-54

But when I process this only the first row gets processed:

         "date" => "Fri, 30 Jun 2017 16:19:37 +0000",
 "temp_reading" => "28.6",
         "batt" => "2.7895",
     "@version" => "1",
  "hum_reading" => "87.2",
         "rssi" => "-54",
      "message" => "Thu Feb 09 21:01:04 2017,28.6,87.2,Fridge 10283739,2.7895,-54\nThu Feb 09 21:15:04 2017,28.6,87.2,Fridge 10283739,2.7895,-54\nThu Feb 09 21:30:04 2017,2.6,3.2,Fridge 10283739,2.7895,-54\n\n",

"@timestampInEmail" => 2017-02-09T21:01:04.000Z,
"@timestamp" => 2017-06-30T16:19:37.000Z,
"timestampInEmail" => "Thu Feb 09 21:01:04 2017",
"location" => "Fridge 10283739"

Am I missing a config somewhere to allow this to process the entire message content?

filter {

csv {
    columns => ["timestampInEmail","temp_reading","hum_reading","location","batt","rssi"]
    separator => ","
}

##
## Remove fields
##
mutate {
    remove_field => ["message-id", "received", "content-type", 
    "list-unsubscribe", "dkim-signature", "x-msfbl", "received-spf", 
    "authentication-results", "mime-version", "return-path", "x-zohomail-sender",
    "content-transfer-encoding", "delivered-to", "x-zohomail", "x-zohomail-owner",
    "list-id", "arc-message-signature","subject", "x-received","to", "from","arc-authentication-results",
    "arc-seal"]
}

date {
    locale => "en"
    match => ["timestampInEmail", "EEE MMM dd HH:mm:ss yyyy"]
    timezone => "Europe/London"
    target => "@timestampInEmail"
}

}

Any assistance will be much appreciated.

Thanks,

You need to split your single event into multiple events (one per line of CSV). You can use the split filter for that.

Thanks!

That was easy if you know how :slight_smile:

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.