Not able to create Index in ES with Logstash

rkhapre · May 22, 2015, 1:16pm

Continuing the discussion from Not able to create Index with Logstash:

input {
file {
path => "C:/TEMP/*.txt"
type => "core2"
start_position => "beginning"
}
}

filter {

}

output {
elasticsearch {
action => "index"
host => "localhost"
cluster => "clustertest"
node_name=> "test"
index => "market"
}
 stdout {
    codec => rubydebug
 }
}

warkolm · May 24, 2015, 2:16am

You haven't given much information.

What happens when you run this?
Have you checked your ES logs? Have you made sure your cluster is up and running ok?

rkhapre · May 24, 2015, 2:26pm

Hi

Yes the 'clustertest' cluster is up and running and also when it starts it is showing the node has started.

When i just create a index with a document in ES it works, but when i run the above config file through logstash it doesn't create a index.

The above code is in my config file- simple.conf for logstash, i run it through logstash command

logstash -f simple.conf

warkolm · May 24, 2015, 11:24pm

If you are always using the same log input then you are probably running into a sincedb issue. See if you have a .sincedb file and if you do delete and try running LS again.

rkhapre · May 25, 2015, 4:37am

Hi,

You seems to be correct as when i change the index name its working but not the next time
Where i can find .sincedb file to delete it

I do not see any, below is screenshot of my logstash home folder. In the documentation its written it will be available @$HOME/.sincedb*

magnusbaeck · May 25, 2015, 5:33am

That's the Logstash installation directory and I doubt you have the HOME variable pointing there. If you start Logstash with --debug it'll tell you what sincedb files it reads. See below for the (Logstash 1.5) logic used when deciding the directory for sincedb files (if the sincedb_path parameter isn't set).

github.com

logstash-plugins/logstash-input-file/blob/v0.1.10/lib/logstash/inputs/file.rb#L94-L104


if ENV["SINCEDB_DIR"].nil? && ENV["HOME"].nil?
  @logger.error("No SINCEDB_DIR or HOME environment variable set, I don't know where " \
                "to keep track of the files I'm watching. Either set " \
                "HOME or SINCEDB_DIR in your environment, or set sincedb_path in " \
                "in your Logstash config for the file input with " \
                "path '#{@path.inspect}'")
  raise # TODO(sissel): HOW DO I FAIL PROPERLY YO
end


#pick SINCEDB_DIR if available, otherwise use HOME
sincedb_dir = ENV["SINCEDB_DIR"] || ENV["HOME"]

Topic		Replies	Views
Logstash not creating an index in elasticsearch Logstash	12	11011	July 6, 2017
Logstash does not create index and doesn't give out any errors. Help Logstash	11	5565	July 6, 2017
Logstash doesn't create index in ES, no errors Logstash	13	4245	July 6, 2017
Not able to create Index in ES by using logstash Logstash	4	909	July 6, 2017
Logstash index is not creating Logstash	17	7363	July 6, 2017

Not able to create Index in ES with Logstash

Related topics