Problem with elastic template in Logstash when indexing

Elastic Stack Logstash
1

i have a working index in Logstash version 5.x but its not working in version 6.3.2
date fields are not reading as date. it's reading as keyword

please help

    {
        "template" : "index_i",
        "settings": { "index.refresh_interval": "5s" },
        "mappings" : {
          "index_i" : {
             "properties": {         	
             	"Start Time":  { "type": "date" , "format" : "MM-dd-YYYY HH:mm" } ,
             	"End Time": { "type": "date" , "format" : "MM-dd-YYYY HH:mm" } ,
                "Process Name": { "type": "keyword" , "fielddata": true } ,
    		    "User Name": {"type":"keyword" , "fielddata": true }
             	}
        }
      }
    }

My logstash config file as below

    input {

	file {
			path => "C:\test.csv"                                
			start_position => "beginning"
			sincedb_path => "C:/Logstash/sincedb/*" 
		}
	}

filter {
csv {

    skip_empty_columns => true
    skip_header => true
    skip_empty_rows => true

    columns => ["Start Time","End Time","ProcessName","User Name"]
	separator => ","

   	}

}

output {
  elasticsearch {
    hosts => ["localhost:9200"]
    index => "index_i"
    template => "C:\test.json"                                        
    document_id => "%{Process Name}"
  }
   stdout {
    codec => rubydebug
  }
}
2

See the index template documentation for ES 6.3. Your template is missing an index_templates option.

3

i edited template like this, still not working, what is the issue ?

{
    "index_patterns" : "index_i",
    "settings": { "number_of_shards": 1 },
    "mappings" : {
      "_doc" : {
        "_source": {
        "enabled": false
      },
         "properties": {         	
         	"Start Time":  { "type": "date" , "format" : "MM/dd/YYYY HH:mm" } ,
         	"End Time": { "type": "date" , "format" : "MM/dd/YYYY HH:mm" } ,
            "Process Name": { "type": "keyword" , "fielddata": true } ,
		    "User Name": {"type":"keyword" , "fielddata": true }
         	}
    }
  }
}
4

I'd verify that the template has been properly stored in ES (use the get index template API), then delete the index_i index, recreated it, and inspect its mappings with the get mapping API. Does the recreated index still have the wrong mappings?

5

when i check GET template/index_i , i didn't get any output

but when i did GET index_i , i get same template with date fields saved as keyword

what am doing wrong ?

6

Since you haven't overridden the template_name option in your elasticsearch output the template will be saved under the name "logstash", so you should issue a GET /_template/logstash request.

7

but why is using logstash template , am specifying template through config file

8

i manually added template through kibana.

i edited my config as below to specify the template index_i

input {

	file {
			path => "C:\test.csv"                            
			start_position => "beginning"
			sincedb_path => "C:/Logstash/sincedb/*"                    
		}
	}

filter {
csv {

	skip_empty_columns => true
    skip_header => true
    skip_empty_rows => true

    columns => ["StartTime","EndTime","ProcessName","UserName"]
	separator => ","

   	}

}

output {
  elasticsearch {
    hosts => ["localhost:9200"]
    index => "index_i"
   template_name => "index_i"
    document_id => "%{ProcessName}"
  }
   stdout {
    codec => rubydebug
  }
}

still am having same issue it's not using the template i specified

Capture
Capture.png1588×562 489 KB
Capture2
Capture2.png561×731 96 KB

9

Now you're not specifying the path to your template so Logstash is overwriting whatever template you may have uploaded under the index_i name. Again:

  • Verify that the index_i template has been properly stored in ES (use the get index template API).
  • Delete the index_i index via the ES API.
  • Recreate it via the ES API.
  • Fetch the mappings with the get mapping API. Are they what you expect?
10
  • I deleted the index_i index from elastic through ES Delete API
  • Then i uploaded the template via ES PUT command
  • Then i run the logstash config file as above
  • I specified the template in config as template_name , not as a json file. because i already uploded it

Still i am having issue, in the logstash cmd window saying , it is using logstash-* template, not using the one i specified through template_name

after running logstash config , do i need to anything extra ?
Is there anything wrong with template or config file ?

11

If you want any further help you need to follow my instructions exactly.

12

Ok.
I uploaded template manually to elasticsearch through put command.

So what should I specify in template/template_name in logstash elasticsearch output section?

13

I uploaded template manually to elasticsearch through put command.

Yes...? And the template was applied when you created an index via the API?

So what should I specify in template/template_name in logstash elasticsearch output section?

The template_name option doesn't determine which index template is applied to the added document. It's the index pattern field in each template you should pay attention to. If you've saved templates under various names you should probably clean them up.

So, the template option should point to the path of the index template you want to store under the name given in the template_name option. It's up to you to make sure that the contents of that template (i.e. the template's index pattern) matches the indexes you're creating.

14

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.