Documents not getting indexed with custom timestamp

Elastic Stack Logstash
1

HI All,

We are trying to index data using a custom timestamp and ended up with failure. Requesting for suggestions.

If i index the data through @timestamp or without time filter i'm getting all the documents indexed, but if i index using the field name Date( the field is there in my csv which i'm uploading)
I have created the mappings in kibana but still i'm not getting any data in the index, It says no results found.
Note: The date mentioned in the fields are backdated.

Any Advice please

Here is my config file

input {
  file {
    path => "/opt/installables/csv/BVD_Master_Data_final.csv"
    start_position => "beginning"
    sincedb_path => "/dev/null"
  }
}
filter {
  csv {
      separator => ","
      columns => ["Date","Hostname","Memory_GB","CPU_Usage_Perc","Mem_Util_Perc","CPU_Load_Perc","ibt_KBS","obt_bS","Swap_usage_Perc","Mem_Usage_GB","Process_Runing","Disk_Used_Perc","ServiceName","Latitude","Longitude","City","State"]
  }
      mutate {convert => ["Memory_GB", "integer"]}
      mutate {convert => ["CPU_Usage_Perc", "float"]}
      mutate {convert => ["Mem_Util_Perc", "float"]}
      mutate {convert => ["CPU_Load_Perc", "float"]}
      mutate {convert => ["ibt_KBS", "float"]}
      mutate {convert => ["obt_bS", "float"]}
      mutate {convert => ["Swap_usage_Perc", "float"]}
      mutate {convert => ["Mem_Usage_GB", "float"]}
      mutate {convert => ["Process_Runing", "integer"]}
      mutate {convert => ["Disk_Used_Perc", "float"]}
      mutate {convert => ["Latitude", "float"]}
      mutate {convert => ["Longitude", "float"]}
      date { match => ["Date", "dd/MM/YYYY HH:mm:ss"] }
      mutate {
      rename => {
          "Longitude" => "[location][lon]"
          "Latitude" => "[location][lat]"
      }
 }
}
output {
   elasticsearch {
     hosts => "1.3.1.3:9200"
     index => "bvd"
  }
#  stdout {
#    codec => rubydebug
#  }
}

Here is my mapping in kibana:

PUT bvd
{
  "mappings": {
    "doc": { 
      "properties": { 
  "Date":  {
          "type":   "date", 
          "format": "dd/MM/YYYY HH:mm:ss",
	  "ignore_malformed": true },
  "Memory_GB":  {
          "type":   "integer"},
  "CPU_Usage_Perc":  {
          "type":   "float"},
  "Mem_Util_Perc":  {
          "type":   "float"},
  "CPU_Load_Perc":  {
          "type":   "float"},
  "ibt_KBS":  {
          "type":   "float"},
  "obt_bS":  {
          "type":   "float"},
  "Swap_usage_Perc":  {
          "type":   "float"},
  "Mem_Usage_GB":  {
          "type":   "float"},
  "Process_Runing":  {
          "type":   "integer"},
  "Disk_Used_Perc":  {
          "type":   "float"},
  "location":  {
	  "type": "geo_point",
	  "ignore_malformed": true}
      } 
    }
  }
}

Thanks
Gauti

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.