Error with Logstash CSV Filter plugin

I am having strange errors with a specific word used in Column headers. All automatic column header recognition works fine, except for when column headers are called FName. Is this a limitation of the CSV Filter plugin?

The Error Log:

[2020-01-21T13:13:55,817][WARN ][logstash.outputs.elasticsearch][pipelineLLM] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"llmlogs-2020.01-000001", :routing=>nil, :_type=>"_doc"}, #<LogStash::Event:0x11a7198>], :response=>{"index"=>{"_index"=>"llmlogs-2020.01-000001", "_type"=>"_doc", "_id"=>"k9cGyG8BK6Mt9ZOQBvTl", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse field [FName] of type [float] in document with id 'k9cGyG8BK6Mt9ZOQBvTl'. Preview of field's value: 'FName'", "caused_by"=>{"type"=>"number_format_exception", "reason"=>"For input string: \"FName\""}}}}}

You have to fix the mappings.

In the error message you clearly see:

[FName] of type [float] 

But Elasticsearch already received a document that was mapped as "string".

As a quick fix you can do data conversion in Logstash:

mutate {
  convert => {
    "Fname" => "string"
  }
}

The columns of my CSV are autodetected, so I haven't mapped anything manually so far. When I look into my Kibana Index Patterns, FName is shown as number.

The same goes for my Elasticsearch data:

FName is the column header. All values in the fields under this column header are decimal point numbers, so float is correct. The error seems to indicate that the parser is trying to cast the column header name "FName" as well, not just the values underneath FName in the column. Or am I misreading that?

I cannot see any documents where FName was recognised as a string.

My Mapping template below:

{
  "numeric_detection": true,
  "dynamic": true,
  "dynamic_templates": [
    {
      "machinename": {
        "mapping": {
          "copy_to": "machinename",
          "type": "string"
        },
        "match": "*.csv"
      }
    }
  ],
  "date_detection": false,
  "properties": {
    "Timestamp": {
      "format": "yyyy-MM-dd'T'HH:mm:ss.SSS'Z'",
      "type": "date"
    }
  }
}

Here's part of the mapping in the index after the detection runs:

      "EngineMisc": {
        "type": "float"
      },
      "EnginePreInit": {
        "type": "float"
      },
      "FMallocUnused": {
        "type": "float"
      },
      "FName": {
        "type": "float"
      },
      "FileSystem": {
        "type": "float"
      },
      "GC": {
        "type": "float"
      },
      "GenericPlatformMallocCrash": {
        "type": "float"
      },
      "InitUObject": {
        "type": "float"
      },
      "LoadMapMisc": {
        "type": "float"
      },

My Logstash config:

input {
  beats {
    port => 5044
  }
}
filter {
  csv {
   id => "LLM"
   autogenerate_column_names => false
   autodetect_column_names => true
   skip_empty_columns => true
   }
}
output {
  stdout { codec => rubydebug }
  elasticsearch { hosts => ["localhost:9200"]
                  index => "llmlogs-2020.01-000001"
  }
}

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.