i am trying to create logstash job that have input from elasticsearch index pattern and to take a specific logs then to save them in a historical index so the configuration as below,
input{
elasticsearch {
hosts => ['http://xxxxxx:9200']
user => xxxx
password => "xxxxx"
index => "iiblogs-*"
}
}
output{
if 'ENQ0000' in [Service]{
stdout{}
}
}
and when try to start the job it gives me this error message
[ERROR] 2023-11-15 09:26:41.552 [[main]<elasticsearch] javapipeline - A plugin had an unrecoverable error. Will restart this plugin.
Pipeline_id:main
Plugin: <LogStash::Inputs::Elasticsearch index=>"xxxxxx", password=><password>, id=>"a0dec4855e6d3299f0afdce84e6cb3feb58a99afbed10fcf132e5c348be99392", user=>"xxxx", hosts=>["http://xxxxxxxx:9200"], enable_metric=>true, codec=><LogStash::Codecs::JSON id=>"json_21a4b3f1-d2bf-4f91-b903-941f4d05bb7c", enable_metric=>true, charset=>"UTF-8">, query=>"{ \"sort\": [ \"_doc\" ] }", size=>1000, scroll=>"1m", docinfo=>false, docinfo_target=>"@metadata", docinfo_fields=>["_index", "_type", "_id"], connect_timeout_seconds=>10, request_timeout_seconds=>60, socket_timeout_seconds=>60, ssl=>false>
Error: [401] {"error":{"root_cause":[{"type":"security_exception","reason":"missing authentication credentials for REST request [/iiblogs-%2A/_search?scroll=1m&size=1000]","header":{"WWW-Authenticate":"Basic realm=\"security\" charset=\"UTF-8\""}}],"type":"security_exception","reason":"missing authentication credentials for REST request [/iiblogs-%2A/_search?scroll=1m&size=1000]","header":{"WWW-Authenticate":"Basic realm=\"security\" charset=\"UTF-8\""}},"status":401}
Exception: Elasticsearch::Transport::Transport::Errors::Unauthorized
noting that I am sure that the user and password are correct and tried to curl on the index with the user and password and the index settings retrieved also they used in other logstash jobs.
since there is no answer after 8 days and this is an important task in my work I used a walk around way to save historical data by creating a transform with a multiple unique ids and the data saved in another index .
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.