Config problem with logstash


(ankit) #1

in loagstash choosing correct path alougth it's not config properly.
file given below.

input {
file {
path => "C:\Ankit\kibana_files\Client_4_May_2018_16_31.csv"
start_position => "beginning"
sincedb_path=>"c:\Ankit\kibana_files"
}
}
filter {
csv{
separator => ","
columns => ["log_id","msg string","ID","Version","date"]
}
date {
match => ["date", "MM/dd/yyyy hh:mm"]
target => "date"
}
mutate {
convert => {
"log_id" => "integer"
"ID" => "integer"
}
}
}
output {
elasticsearch {
hosts => "localhost"
index => "error"
}
stdout {codec => rubydebug}
}

In logstash:
logstash -f c:\ankit\kibana_files\test_error_1.conf

it's not working?
i can not creat proper index pattern?


(Magnus B├Ąck) #2

Comment out your elasticsearch output. Are you getting anything from your stdout output? Have you tried deleting the sincedb file?


(ankit) #3

this kindof issue i face during logstash install
confi file is above.

[2018-07-02T13:32:33,413][INFO ][logstash.setting.writabledirectory] Creating directory {:setting=>"path.queue", :path=>"C:/Ankit/tools/logstash-6.3.0/data/queue"}
[2018-07-02T13:32:33,413][INFO ][logstash.setting.writabledirectory] Creating directory {:setting=>"path.dead_letter_queue", :path=>"C:/Ankit/tools/logstash-6.3.0/data/dead_letter_queue"}
[2018-07-02T13:32:33,569][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2018-07-02T13:32:33,694][INFO ][logstash.agent ] No persistent UUID file found. Generating new UUID {:uuid=>"04aac045-97eb-4a5c-80cc-f3318e2cad38", :path=>"C:/Ankit/tools/logstash-6.3.0/data/uuid"}
[2018-07-02T13:32:34,442][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"6.3.0"}
[2018-07-02T13:32:39,559][INFO ][logstash.pipeline ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
[2018-07-02T13:32:40,636][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://localhost:9200/]}}
[2018-07-02T13:32:40,651][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://localhost:9200/, :path=>"/"}
[2018-07-02T13:32:41,213][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://localhost:9200/"}
[2018-07-02T13:32:41,759][INFO ][logstash.outputs.elasticsearch] ES Output version determined {:es_version=>6}
[2018-07-02T13:32:41,774][WARN ][logstash.outputs.elasticsearch] Detected a 6.x and above cluster: the type event field won't be used to determine the document _type {:es_version=>6}
[2018-07-02T13:32:41,790][INFO ][logstash.outputs.elasticsearch] Using mapping template from {:path=>nil}
[2018-07-02T13:32:41,852][INFO ][logstash.outputs.elasticsearch] Attempting to install template {:manage_template=>{"template"=>"logstash-", "version"=>60001, "settings"=>{"index.refresh_interval"=>"5s"}, "mappings"=>{"default"=>{"dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date"}, "@version"=>{"type"=>"keyword"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}}
[2018-07-02T13:32:42,102][INFO ][logstash.outputs.elasticsearch] Installing elasticsearch template to _template/logstash
[2018-07-02T13:32:44,066][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//localhost"]}
[2018-07-02T13:32:45,174][INFO ][logstash.pipeline ] Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x1628d06 sleep>"}
[2018-07-02T13:32:45,423][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2018-07-02T13:32:45,891][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
[2018-07-02T13:34:20,813][WARN ][logstash.runner ] SIGINT received. Shutting down.


(Krunal Kalaria) #4

hey @ankit211,

You have installed x-pack in your Elastic Stack ?

try with following in elasticsearch output part :slight_smile:

hosts => ["localhost:9200"]

and see that file has permission or not if its not their then give the permission to chmod 777.

and restart the logstash and run the config file tell me its working or not.

Thanks & Regards,
Krunal.


(ankit) #5

yes i have x-pack in elasticsearch.
i try this elasticsearch output.
restart logstash.

still not working. same thing happen with logstash.
kibana-6.3.0
logstash 6.3.0
elasticsearch-6.3.0
working on windows.


(Krunal Kalaria) #6

okk now add this following lines in your elasticsearch output

output
{
elasticsearch
{
hosts => ["localhost:9200"]
index => "error"
user => "elastic" #your elasticsearch username that you have mention while installed x-pack
password => "elastic" #your elasticsearch password that you have mention while installed x-pack
}
stdout { codec => rubydebug }
}

Note that your have to add the x-pack passwords of elasticsearch in your elasticsearch.yml file.

have you added that x-pack username and password of elasticsearch in your elasticsearch.yml file ?

Thanks & Regards,
Krunal.


(system) #8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.