Csv filter parsing error

Elastic Stack Logstash
1

Hello,

I have multiple csv files I need to ingest. Among the csv files I have 13 column header names variations. I would like to use the autodetect_column_names to avoid manually configuring these column names. Also please note each csv file has two empty column headers at the end.

Looking at previous post I saw a suggested fix using the below configuration, which helped with my error exception: #<RuntimeError: Invalid FieldReference: []`>`.

mutate { gsub => ["message", "[[]]", " "] }

I am no longer receiving the _csvparsefailure and logs are getting parsed a bit. However, they are not aligned. For example, I have:
"GeoIP City": "12/24/2024 08:14:23" when it should be a city
"first_name": null when there should be a name.

My full filter is as below

<filter {
mutate {
gsub => ["message", "[\[\]]", " "]
}

csv {
    separator => ","
    autodetect_column_names => true
    #skip_empty_columns => true

}}>

Here is a sample output of a row data with the additional error I get

`Error parsing csv {field: "message", source: "test@email.com,"",FY26 Q3 ALL,"","",No,"",No,"",No,"","","",,unknown,unknown,unknown,2.0.0,"Apr 23 18:46:21 postfix-smtp postfix/smtp[5969]: 4: to=test@email.com, relay=aspmx.l.google.com[x.x.x.x]:25, delay=1.2, delays=0.12/0/0.29/0.83, dsn=2.0.0, status=sent (250 2.0.0 OK 177-12.53 - gsmtp)",04/23/2026 14:46:21,,,,,,Test,Name", exception: #<RuntimeError: Invalid FieldReference: [Apr 23 18:41:38 postfix-smtp postfix/smtp[5601]: 4: to=<other@email.com>, ``relay=aspmx.l.google.com``[y.y.y.y]:25, delay=1.1, delays=0.12/0/0.3/0.67, dsn=x.0.0, status=sent (250 2.0.0 OK 177-2e.16 - gsmtp)]>}`

Any help you could provide will be much appreciated.

2

Can you put a sample, the header and data lines?

3

You have commas inside the field in the column 19.

Try something like this:

input {
  generator {
       message => 'test@email.com,"",FY26 Q3 ALL,"","",No,"",No,"",No,"","","",,unknown,unknown,unknown,2.0.0,"Apr 23 18:46:21 postfix-smtp postfix/smtp[5969]: 4: to=test@email.com, relay=aspmx.l.google.com[x.x.x.x]:25, delay=1.2, delays=0.12/0/0.29/0.83, dsn=2.0.0, status=sent (250 2.0.0 OK 177-12.53 - gsmtp)",04/23/2026 14:46:21,,,,,,Test,Name'
       count => 1
  }
}
filter{
    mutate{
	 copy => { "message" => "[@metadata][temp]" }
	}
	
	# replace ,\s with something will not show up i.e "=>" in copied temp field
    mutate{
	 gsub => ["[@metadata][temp]", ", ", "=>"]
	}
	
	# parse
	csv{
	source => "[@metadata][temp]"
	separator => ","
	#skip_empty_columns  =>  false
	}
	
	#return back the field separator
     mutate{
	 gsub => ["column19", "=>", ", "]
	}
    mutate{
	remove_field => [ "event",  "host", "@timestamp", "@version", "message"] 
	}

}
output {
    stdout { codec => rubydebug{ metadata => true}} 
}

Output:

{
      "column1" => "test@email.com",
     "column23" => nil,
...
    "@metadata" => {
        "temp" => "test@email.com,\"\",FY26 Q3 ALL,\"\",\"\",No,\"\",No,\"\",No,\"\",\"\",\"\",,unknown,unknown,unknown,2.0.0,\"Apr 23 18:46:21 postfix-smtp postfix/smtp[5969]: 4: to=test@email.com=>relay=aspmx.l.google.com[x.x.x.x]:25=>delay=1.2=>delays=0.12/0/0.29/0.83=>dsn=2.0.0=>status=sent (250 2.0.0 OK 177-12.53 - gsmtp)\",04/23/2026 14:46:21,,,,,,Test,Name"
    },
     "column19" => "Apr 23 18:46:21 postfix-smtp postfix/smtp[5969]: 4: to=test@email.com, relay=aspmx.l.google.com[x.x.x.x]:25, delay=1.2, delays=0.12/0/0.29/0.83, dsn=2.0.0, status=sent (250 2.0.0 OK 177-12.53 - gsmtp)",
      "message" => "test@email.com,\"\",FY26 Q3 ALL,\"\",\"\",No,\"\",No,\"\",No,\"\",\"\",\"\",,unknown,unknown,unknown,2.0.0,\"Apr 23 18:46:21 postfix-smtp postfix/smtp[5969]: 4: to=test@email.com, relay=aspmx.l.google.com[x.x.x.x]:25, delay=1.2, delays=0.12/0/0.29/0.83, dsn=2.0.0, status=sent (250 2.0.0 OK 177-12.53 - gsmtp)\",04/23/2026 14:46:21,,,,,,Test,Name",
...
}