Could not index event to Elasticsearch. status: 404

Elastic Stack Logstash
1

refer to /etc/logstash/conf.d/pipeline.conf

input {
  file {
    path => "/var/log/secure"
    start_position => "beginning"
    sincedb_path => "/dev/null"
  }
}
filter {
  grok {
    match => { "message" => ""%{IPV4:ip} %{DATA} %{DATA} \[%{DATA:time} %{ISO8601_TIMEZONE:timeZone}\] "%{DATA:method} %{URIPATH:url} %{URIPROTO:protocol}/%{NUMBER:ver}" %{NUMBER} %{DATA} "%{GREEDYDATA:jdk}" %{NUMBER:micro_sec} %{NUMBER:res_code}""}
    }
}
output {
  elasticsearch {
    hosts => ["https://elk.myvfirst.com:9200"]
    user => "#####"
    password => "#####"
    cacert => ["/etc/logstash/ca.crt"]
    index => "testdata"
    ilm_enabled => false
  }
}

refer to logstash log below

[2023-01-02T19:15:02,434][WARN ][logstash.outputs.elasticsearch][main][96893d2aeea9e839ac0f0219e3d2f83164833c044c7b26e5892645181a6ee7b8] Could not index event to Elasticsearch. status: 404, action: ["index", {:_id=>nil, :_index=>"testdata", :routing=>nil}, {"month"=>"Jan", "message"=>["Jan  2 19:15:01 vl070073-app2-pd-a15-sms-aws-test-mum-in-vf crond[17732]: pam_env(crond:setcred): Unable to open env file: /etc/environment: No such file or directory", "No such file or directory"], "event"=>{"original"=>"Jan  2 19:15:01 vl070073-app2-pd-a15-sms-aws-test-mum-in-vf crond[17732]: pam_env(crond:setcred): Unable to open env file: /etc/environment: No such file or directory"}, "log"=>{"file"=>{"path"=>"/var/log/secure"}}, "host"=>{"name"=>"vl070073-app2-pd-a15-sms-aws-test-mum-in-vf.vfirst.local"}, "hostname"=>"vl070073-app2-pd-a15-sms-aws-test-mum-in-vf", "action"=>"crond[17732]: pam_env(crond:setcred): Unable to open env file: /etc/environment", "@version"=>"1", "date"=>"2", "time"=>"19:15:01", "@timestamp"=>2023-01-02T13:45:02.328131787Z}], response: {"index"=>{"_index"=>"testdata", "_id"=>nil, "status"=>404, "error"=>{"type"=>"index_not_found_exception", "reason"=>"no such index [testdata] and [action.auto_create_index] ([.monitoring*,.watches,.triggered_watches,.watcher-history*,.ml*,.apm*]) doesn't match", "index_uuid"=>"_na_", "index"=>"testdata"}}}
[2023-01-02T19:15:02,434][WARN ][logstash.outputs.elasticsearch][main][96893d2aeea9e839ac0f0219e3d2f83164833c044c7b26e5892645181a6ee7b8] Could not index event to Elasticsearch. status: 404, action: ["index", {:_id=>nil, :_index=>"testdata", :routing=>nil}, {"month"=>"Jan", "message"=>["Jan  2 19:15:01 vl070073-app2-pd-a15-sms-aws-test-mum-in-vf crond[17732]: pam_env(crond:setcred): Unable to open env file: /etc/environment: No such file or directory", "No such file or directory"], "event"=>{"original"=>"Jan  2 19:15:01 vl070073-app2-pd-a15-sms-aws-test-mum-in-vf crond[17732]: pam_env(crond:setcred): Unable to open env file: /etc/environment: No such file or directory"}, "log"=>{"file"=>{"path"=>"/var/log/secure"}}, "host"=>{"name"=>"vl070073-app2-pd-a15-sms-aws-test-mum-in-vf.vfirst.local"}, "hostname"=>"vl070073-app2-pd-a15-sms-aws-test-mum-in-vf", "action"=>"crond[17732]: pam_env(crond:setcred): Unable to open env file: /etc/environment", "@version"=>"1", "date"=>"2", "time"=>"19:15:01", "@timestamp"=>2023-01-02T13:45:02.327851201Z}], response: {"index"=>{"_index"=>"testdata", "_id"=>nil, "status"=>404, "error"=>{"type"=>"index_not_found_exception", "reason"=>"no such index [testdata] and [action.auto_create_index] ([.monitoring*,.watches,.triggered_watches,.watcher-history*,.ml*,.apm*]) doesn't match", "index_uuid"=>"_na_", "index"=>"testdata"}}}
[2023-01-02T19:15:02,436][WARN ][logstash.outputs.elasticsearch][main][96893d2aeea9e839ac0f0219e3d2f83164833c044c7b26e5892645181a6ee7b8] Could not index event to Elasticsearch. status: 404, action: ["index", {:_id=>nil, :_index=>"testdata", :routing=>nil}, {"month"=>"Jan", "message"=>["Jan  2 19:15:01 vl070073-app2-pd-a15-sms-aws-test-mum-in-vf CROND[17732]: pam_env(crond:setcred): Unable to open env file: /etc/environment: No such file or directory", "No such file or directory"], "event"=>{"original"=>"Jan  2 19:15:01 vl070073-app2-pd-a15-sms-aws-test-mum-in-vf CROND[17732]: pam_env(crond:setcred): Unable to open env file: /etc/environment: No such file or directory"}, "log"=>{"file"=>{"path"=>"/var/log/secure"}}, "host"=>{"name"=>"vl070073-app2-pd-a15-sms-aws-test-mum-in-vf.vfirst.local"}, "hostname"=>"vl070073-app2-pd-a15-sms-aws-test-mum-in-vf", "action"=>"CROND[17732]: pam_env(crond:setcred): Unable to open env file: /etc/environment", "@version"=>"1", "date"=>"2", "time"=>"19:15:01", "@timestamp"=>2023-01-02T13:45:02.328294019Z}], response: {"index"=>{"_index"=>"testdata", "_id"=>nil, "status"=>404, "error"=>{"type"=>"index_not_found_exception", "reason"=>"no such index [testdata] and [action.auto_create_index] ([.monitoring*,.watches,.triggered_watches,.watcher-history*,.ml*,.apm*]) doesn't match", "index_uuid"=>"_na_", "index"=>"testdata"}}}
2

By default, elasticsearch will create any index that you try to write to. It looks like someone has disabled that option.

1 Like
3

How can I enable that option and also can I create the index manually

4

@warkolm I understand the option can be set, but I do not run elasticsearch, so I do not have a test-bed for this. Can you answer? Thanks in advance!

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.