I have some data in the index of Elasticsearch. I want to copy the data from the existing index in to the new index in Elasticsearch using logstash. I have used Elasticsearch input plugin and output plugin to load it to the new index.
Here's the config file i used:
input {
elasticsearch {
hosts => ["localhost:9200"]
index => "new_apps"
document_type =>"sample_apps"
query => '{ "query": {"match_all": {}} }'
size => 2000
scroll => "5m"
docinfo => true
}
}
output {
stdout {}
elasticsearch {
index => "newindex"
hosts => ["localhost:9200"]
}
}
But it's not getting data from the Elasticsearch index. I don't know where i am doing wrong. How to fix it....... Is the configuration file is correct????
Error i got in terminal:
Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.headius.backport9.modules.Modules (file:/usr/share/logstash/logstash-core/lib/jars/jruby-complete-9.2.7.0.jar) to field java.io.FileDescriptor.fd
WARNING: Please consider reporting this to the maintainers of com.headius.backport9.modules.Modules
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[WARN ] 2019-06-07 10:43:31.437 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified
[INFO ] 2019-06-07 10:43:31.458 [LogStash::Runner] runner - Starting Logstash {"logstash.version"=>"6.8.0"}
[ERROR] 2019-06-07 10:43:35.665 [Converge PipelineAction::Create] elasticsearch - Unknown setting 'document_type' for elasticsearch
[ERROR] 2019-06-07 10:43:35.685 [Converge PipelineAction::Create] agent - Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Something is wrong with your configuration.", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/config/mixin.rb:86:in `config_init'", "/usr/share/logstash/logstash-core/lib/logstash/inputs/base.rb:60:in `initialize'", "org/logstash/plugins/PluginFactoryExt.java:255:in `plugin'", "org/logstash/plugins/PluginFactoryExt.java:184:in `plugin'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:71:in `plugin'", "(eval):8:in `initialize'", "org/jruby/RubyKernel.java:1061:in `eval'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:49:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:90:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:43:in `block in execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:96:in `block in exclusive'", "org/jruby/ext/thread/Mutex.java:165:in `synchronize'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:96:in `exclusive'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:39:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:334:in `block in converge_state'"]}
[INFO ] 2019-06-07 10:43:36.323 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600}
[INFO ] 2019-06-07 10:43:40.905 [LogStash::Runner] runner - Logstash shut down.