Logstash not creating an index in elasticsearch

Elastic Stack Logstash
1

Below is my config file -

input {  
      file {
          path => "/home/pavan/Downloads/sample.csv"
	  type => "Sample"
          start_position => "beginning"
      }
}
filter {  
    csv {
        columns => ['cdatetime', 'address', 'district', 'beat', 'grid', 'crimedescr', 'ucr_ncic_code', 'latitude', 'longitude']
        separator => ","
    }
}

output {  
    elasticsearch {
	
        action => "index"
        host => "localhost"
        protocol => "http"
        index => "sample"
	cluster => "pavan"        
    }
      stdout {
        codec => rubydebug
     }
}

The problem is when I run bin/logstash -f sample.conf from terminal I get the following response Logstash startup completed but when I see the list of indices in Elasticsearch, it remains the same.

2

It'll be a sincedb issue, delete that and you should be good (see the file input docs).

3

Can you please be more specific as I'm still a beginner with Logstash.

4

Check out https://www.elastic.co/guide/en/logstash/current/plugins-inputs-file.html#plugins-inputs-file-sincedb_path

5

Sorry. It didn't solve my problem. Do you have any alternatives?

6

I'm sorry about the inconvenience Warkolm. I re-installed Logstash and after that I located all the .sincedb files which were in HOME directory and deleted it. After that my first index got created in elasticsearch but then when I try to index a new csv file, it fails. I again need to explicitly delete those sincedb files and index it again. Is there any way to autodelete .sincedb files?

7

Are you doing it into the same file?

8

Yeah.I'm just changing the path in the same .config file. Is that the problem?

9

Hi

Hope it will solve your problem
add sincedb_path => "/home/pavan/Downloads/dbflile"
it will create one file with name dbfile and this file keeps track of the current position of monitored log files.

Exmple
input {
file {
path => "/home/pavan/Downloads/sample.csv"
type => "Sample"
start_position => "beginning"
sincedb_path => "/home/pavan/Downloads/dbflile"
}
}

//Lee

2 Likes
10

Thanks Lee. It solved my problem.

11

Have the same problem, deleted the sincedb files but index still won't be created. Run logtsash with path to a logfile and there's no error in that file. Kinda blind to what's going on. Any ideas?

12

If the file input's ignore_older option doesn't help you, please start a new thread for your problem.

1 Like