Problem: When trying to create a new index pattern the error "The index pattern you've entered does not match any indices". Really the problem is my index's will not load from a config / schema.
Index Details: This index is associated with the schema and config file which are located in /etc/logstash/conf.d/rjdns.conf, and /etc/logstash/rjdns.json.
Trouble shooting:
- Tried checking Stack Management > Index Management, index does not show. Tried "Reload indicies", nothing.
- Tried to create the index under "Index Mangement", hence the problem above, the "next step" is greyed out when trying to use "rjdns*"
- Checked syslog, I keep seeing this error but not sure if related to my problem or for some other file, there is no context to what is causing this. Anyway I can tell?
[2020-12-23T15:34:52,994][ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of [ \\t\\r\\n], \"#\", \"if\", [A-Za-z0-9_-], '\"', \"'\", \"}\" at line 6, column 1 (byte 41) after input {\n beats {\n port => 5044\n }\n\n", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:58:in `compile_imperative'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:66:in `compile_graph'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:28:in `block in compile_sources'", "org/jruby/RubyArray.java:2577:in `map'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:27:in `compile_sources'", "org/logstash/execution/AbstractPipelineExt.java:181:in `initialize'", "org/logstash/execution/JavaBasePipelineExt.java:67:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:43:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:52:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:342:in `block in converge_state'"]
- Tried running the following commands:
-
/usr/share/logstash/bin/logstash --config.test_and_exit -f conf.d/rjdns.conf
after running I get "Configuration OK" -
/usr/share/logstash/bin/logstash --debug -f rjflow-schema.jsonafter running I get the following (Only including Errors and warnings):
-
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path
[WARN ] 2020-12-23 15:32:09.959 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified
[ERROR] 2020-12-23 15:32:10.374 [Agent thread] sourceloader - No configuration found in the configured sources.
Here is the config file:
input {
file {
path => "/etc/logstash/redjack-data/sensor-dnstap/*.json"
mode => "read"
codec => "json"
exit_after_read => true
}
}
output {
elasticsearch {
hosts => ["10.10.10.10:9200"]
index => "rjdns-%{+yyyy.MM.dd}"
manage_template => true
template => "/etc/logstash/rjdns-schema.json"
template_name => "rjdns_template"
}
}
Really not sure how to trouble shoot the problem at this point. Why wont Elastic load my config / schema as an index? Is this a permissions issue? Is there anyway I can check some log or debug to tell me why the index wont be recognized?