I have a java based application which reads an index sys_username_mapping. This index gets update once every day. My steps are :
- initially load the index sys_username_mapping using filebeat+logstash for first time with a csv file.
- Start my java application that reads sys_username_mapping index and it works well reading the index sys_username_mapping.
- On 2nd day of data load into same index sys_username_mapping, the java application start giving me this error in the log file :
"ESWriter.initDependentData(ESWriter.java:258) Dependent index [sys_username_mapping] is
missing."
This java application has to read data from sys_username_mapping index and write to another index. However it starts giving this error only after 2nd load of sys_username_mapping index.
Here is my logstash.conf file :
input {
beats {
port => 5043
client_inactivity_timeout => 432000
}
}
filter {
dissect {
mapping => { "message" => "%{user_name}:%{email}/%{site}/%{group}/%{division}/%{cad_cc}/%{ldap_cc}/%{acc_type}" }
}
grok {
match => ["@timestamp", "%{YEAR:al_year}-%{MONTHNUM:al_month}-%{MONTHDAY:al_day}T%{HOUR:al_hr}:%{MINUTE:al_mn}:%{SECOND:al_ss}"]
add_field => ["last_modified_utctimestamp", "%{al_year}-%{al_month}-%{al_day} %{al_hr}:%{al_mn}:%{al_ss}"]
remove_field => "al_year"
remove_field => "al_month"
remove_field => "al_day"
remove_field => "al_hr"
remove_field => "al_mn"
remove_field => "al_ss"
}
mutate {
convert => {
"user_name" => "string"
"email" => "string"
"site" => "string"
"group" => "string"
"division" => "string"
"cad_cc" => "string"
"ldap_cc" => "string"
"acc_type" => "string"
}
gsub => [ "last_modified_utctimestamp", ":\d\d.\d\d\d$", ":00" ]
}
date {
match => [ "last_modified_utctimestamp", "yyyy-MM-dd HH:mm:ss" ]
timezone => "Europe/Paris"
target => "last_modified_time_stamp"
}
ruby {
code => "try = DateTime.parse(event.get('last_modified_utctimestamp'))
event.set('last_modified_utc',try.strftime('%Q')) "
}
}
output {
elasticsearch {
hosts => ["http://xxxx.xxxx.xxxx.xxxx:9200"]
index => "sys_username_mapping"
document_id => "%{user_name}"
}
}
I am able to list sys_username_mapping index and see it's contents using curl but the java application is not able to see it. I suspect that java application is not able to reload the state with new data? Should I use action => "update" ? It's really difficult to debug