Unable to create elasticsearch index from logstash.conf file

Elastic Stack Elasticsearch
1

I am trying to create index of elasticsearch using logstash.conf file but I am not able to create index. There is not any error in logs of elasticsearch and logstash. After starting the logstash whenever I do url/_cat/indices it is not showing index in the list but whenever I am commenting index tag from conf file it is creating the default index with name logstash-%(date)
Pls help me I am new to this.
Following is the logstash.conf file

input {
  file {
    path => ["/var/lib/jenkins/jobs/settlement_poc/builds/124/log"]
    type => "syslog"
  }
}
output {
  elasticsearch {
    hosts => ["http://localhost:9200"]
    index => "syslog-%{+YYYY.MM.dd}"
    document_type => "system_logs"
  }
}
2

@sachinarora I'm not positive, but you should do two things:

  1. Try running your pipeline with output { stdout {} } to make sure you're getting records emitted at all
  2. Try running without setting document_type: that is a deprecated field and should be using the default value of _doc. See https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html#plugins-outputs-elasticsearch-document_type
3

yes i tried running with stdout it is printing logs and I also removed document type still same issue

4

Hey what I found it is creating index of one log file file but not for another
second log file does n't have any timestamp does it matter while creating index

5

and index doesn't need a timestamp or date field, no. But, if the second log file isn't in syslog format, you input codec might be causing the problem. Do you see any errors in Logstash logs?

6

no there is not any error in logstash and both log files are completely different , I did not created any format for any log file..I just used another log file for testing purpose. But it creating index for one file but not creating for the file I need.

7

I highly recommend that you give a close reading to the file input documentation. For example, are you in read mode or tail mode? Are you aware of the implications of sincedb? If using the default tail mode, is the second log file receiving new events or are you expecting Logstash to process it from the beginning, which means you need to set start_position?

For a sanity check, run a pipeline like:

input {
    file {
        path => "/var/lib/jenkins/jobs/settlement_poc/builds/124/log"
        start_position => "beginning"
        sincedb_path => "/dev/null"
    }
}
output {
    stdout {}
}

...and do it at TRACE level logging: logtash --log.level TRACE -f <your pipeline>

You should see TRACE message for every file it finds (look for log lines with "filewatch.discoverer").

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.