Struggling with mutate - not uploading after conversion

Elastic Stack Logstash
1

Hello all,

I am brand new to Elastic and working with a CSV. After my initial upload, everything converted to strings so I used mutate to convert. However, my file no longer uploads. I've gone through all the manuals I can and still can't find what's wrong.

I'm open to any suggestions and directions to other places to research how to fix this.

Thank you so much.

input {
file {
  path => "/Users/mike/fia/test/*.csv"
  start_position => "beginning"
  sincedb_path => "dev/null"}
  }

filter
  {
    csv
      {
    separator => ","
        columns => [
        "General Offense Number",
        "Event Clearance Group",
        "Event Clearance Date",
        "District/Sector"
        "Longitude",
        "Latitude",
        "Initial Type Group",
        "At Scene Time"]
    remove_field => ["message"]
    }

  mutate {
     convert => {
    "General Offense Number" => "integer",
    "Event Clearance Group" => "string",
    "District/Sector" => "string",
    "Longitude" => "geopoint.lon",
    "Latitude" => "geopoint.lat",
    "Initial Type Group" => "string"
    "At Scene Time"]
    }
  }
   
date {
match => ["Event Clearance Data", "MM/dd/yyyy hh:mm"
    match => "At Scene Time" = "MM/dd/yyyy HH:mm"]
}
}

 output
  {
     elasticsearch
      {
        hosts => "http://localhost:9200"
    index => "index4"
      }
    stdout {}
   }
2

If there is nothing in stdout then it's probably;

Add another / at the start of the path.

3

Ok, I added the "/" and still am getting the error.

I've been doing some research on date handling and I think that's where it is getting thrown off.

input {
file {
path => "/Users/mike/fia/test/*.csv"
start_position => "beginning"
sincedb_path => "/dev/null"}
  }

filter
{
csv
  {
separator => ","
    columns => [
    "General Offense Number",
    "Event Clearance Group",
    "Event Clearance Date",
    "District/Sector",
    "Longitude",
    "Latitude",
    "Initial Type Group",
    "At Scene Time"]
remove_field => ["message"]
}
  
mutate {
convert => {
"General Offense Number" => "integer"
"Event Clearance Group" => "string"
"District/Sector" => "string"
"Longitude" => "geopoint.lon"
"Latitude" => "geopoint.lat"
"Initial Type Group" => "string"
}
}

date {
match => ["Event Clearance Date", "MM/dd/YY HH:mm:ss"]
match => ["At Scene Time", "MM/dd/YY HH:mm:ss”]
}
}

output
{
elasticsearch
  {
    hosts => "http://localhost:9200"
      index => "test"
  }
stdout {}
}

Every time I load the .conf I get an error saying
"[2017-01-09T20:09:14,623][ERROR][logstash.agent ] fetched an invalid config {:......(edited)..... reason=>"Expected one of #, {, ,, ] at line 46, column 19 (byte 893) after filter

I am 100% lost. Any help or direction would be appreciated. Thanks!

4

Line 46 - hosts => "http://localhost:9200"

Yet the docs say it's an array - https://www.elastic.co/guide/en/logstash/5.1/plugins-outputs-elasticsearch.html#plugins-outputs-elasticsearch-hosts

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.