Missing index issue

I have a java based application which reads an index sys_username_mapping. This index gets update once every day. My steps are :

  1. initially load the index sys_username_mapping using filebeat+logstash for first time with a csv file.
  2. Start my java application that reads sys_username_mapping index and it works well reading the index sys_username_mapping.
  3. On 2nd day of data load into same index sys_username_mapping, the java application start giving me this error in the log file :

"ESWriter.initDependentData(ESWriter.java:258) Dependent index [sys_username_mapping] is
missing."

This java application has to read data from sys_username_mapping index and write to another index. However it starts giving this error only after 2nd load of sys_username_mapping index.

Here is my logstash.conf file :
input {
beats {
port => 5043
client_inactivity_timeout => 432000
}
}

filter {

dissect {
mapping => { "message" => "%{user_name}:%{email}/%{site}/%{group}/%{division}/%{cad_cc}/%{ldap_cc}/%{acc_type}" }
}

grok {

    match => ["@timestamp", "%{YEAR:al_year}-%{MONTHNUM:al_month}-%{MONTHDAY:al_day}T%{HOUR:al_hr}:%{MINUTE:al_mn}:%{SECOND:al_ss}"]
    add_field => ["last_modified_utctimestamp", "%{al_year}-%{al_month}-%{al_day} %{al_hr}:%{al_mn}:%{al_ss}"]
    remove_field => "al_year"
    remove_field => "al_month"
    remove_field => "al_day"
    remove_field => "al_hr"
    remove_field => "al_mn"
    remove_field => "al_ss"
    }

mutate {
convert => {
"user_name" => "string"
"email" => "string"
"site" => "string"
"group" => "string"
"division" => "string"
"cad_cc" => "string"
"ldap_cc" => "string"
"acc_type" => "string"
}
gsub => [ "last_modified_utctimestamp", ":\d\d.\d\d\d$", ":00" ]
}
date {
match => [ "last_modified_utctimestamp", "yyyy-MM-dd HH:mm:ss" ]
timezone => "Europe/Paris"
target => "last_modified_time_stamp"

}

ruby {
code => "try = DateTime.parse(event.get('last_modified_utctimestamp'))
event.set('last_modified_utc',try.strftime('%Q')) "
}
}

output {
elasticsearch {
hosts => ["http://xxxx.xxxx.xxxx.xxxx:9200"]
index => "sys_username_mapping"
document_id => "%{user_name}"
}
}

I am able to list sys_username_mapping index and see it's contents using curl but the java application is not able to see it. I suspect that java application is not able to reload the state with new data? Should I use action => "update" ? It's really difficult to debug

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.