Logstash elasticsearch input plugin

Elastic Stack Logstash
1

Hello dears,

i am trying to create logstash job that have input from elasticsearch index pattern and to take a specific logs then to save them in a historical index so the configuration as below,

input{
	elasticsearch	{
  						hosts => ['http://xxxxxx:9200']
  						user => xxxx
  						password => "xxxxx"
  						index => "iiblogs-*"
						}
}

output{
  if 'ENQ0000' in [Service]{
    stdout{}
  } 
}

and when try to start the job it gives me this error message

[ERROR] 2023-11-15 09:26:41.552 [[main]<elasticsearch] javapipeline - A plugin had an unrecoverable error. Will restart this plugin.
  Pipeline_id:main
  Plugin: <LogStash::Inputs::Elasticsearch index=>"xxxxxx", password=><password>, id=>"a0dec4855e6d3299f0afdce84e6cb3feb58a99afbed10fcf132e5c348be99392", user=>"xxxx", hosts=>["http://xxxxxxxx:9200"], enable_metric=>true, codec=><LogStash::Codecs::JSON id=>"json_21a4b3f1-d2bf-4f91-b903-941f4d05bb7c", enable_metric=>true, charset=>"UTF-8">, query=>"{ \"sort\": [ \"_doc\" ] }", size=>1000, scroll=>"1m", docinfo=>false, docinfo_target=>"@metadata", docinfo_fields=>["_index", "_type", "_id"], connect_timeout_seconds=>10, request_timeout_seconds=>60, socket_timeout_seconds=>60, ssl=>false>
  Error: [401] {"error":{"root_cause":[{"type":"security_exception","reason":"missing authentication credentials for REST request [/iiblogs-%2A/_search?scroll=1m&size=1000]","header":{"WWW-Authenticate":"Basic realm=\"security\" charset=\"UTF-8\""}}],"type":"security_exception","reason":"missing authentication credentials for REST request [/iiblogs-%2A/_search?scroll=1m&size=1000]","header":{"WWW-Authenticate":"Basic realm=\"security\" charset=\"UTF-8\""}},"status":401}
  Exception: Elasticsearch::Transport::Transport::Errors::Unauthorized

noting that I am sure that the user and password are correct and tried to curl on the index with the user and password and the index settings retrieved also they used in other logstash jobs.

2

since there is no answer after 8 days and this is an important task in my work I used a walk around way to save historical data by creating a transform with a multiple unique ids and the data saved in another index .

3

Which version are you using. Shouldn't it be https instead of http?

4

Tried with https and the output as below, the version is 7.14.0

i don't think we have to use the https because we dont use certificate

[2023-11-28T12:34:33,001][ERROR][logstash.javapipeline    ][main][691e4dd7a344720d3a28f9d1f072bd34a4bd64ec27f02cac891b60967d67abd5] A pl                         ugin had an unrecoverable error. Will restart this plugin.
  Pipeline_id:main
  Plugin: <LogStash::Inputs::Elasticsearch index=>"iib-*", password=><password>, id=>"691e4dd7a344720d3a28f9d1f072bd34a4bd64ec27f02cac89                         1b60967d67abd5", user=>"logstash_internal", ssl=>false, hosts=>["https://xxxxxxx:xxxx"], enable_metric=>true, codec=><LogStash::Codecs                         ::JSON id=>"json_5724c5f8-c99a-4bc0-a442-b2acea905a1b", enable_metric=>true, charset=>"UTF-8">, query=>"{ \"sort\": [ \"_doc\" ] }", siz                         e=>1000, scroll=>"1m", docinfo=>false, docinfo_target=>"@metadata", docinfo_fields=>["_index", "_type", "_id"], connect_timeout_seconds=                         >10, request_timeout_seconds=>60, socket_timeout_seconds=>60>
  Error: https
  Exception: Manticore::ResolutionFailure
  Stack: /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/manticore-0.7.0-java/lib/manticore/response.rb:37:in `block in initialize'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/manticore-0.7.0-java/lib/manticore/response.rb:79:in `call'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/manticore-0.7.0-java/lib/manticore/response.rb:274:in `call_once'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/manticore-0.7.0-java/lib/manticore/response.rb:158:in `code'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/elasticsearch-transport-7.5.0/lib/elasticsearch/transport/transport/http/manticore.rb                         :89:in `block in perform_request'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/elasticsearch-transport-7.5.0/lib/elasticsearch/transport/transport/base.rb:274:in `p                         erform_request'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/elasticsearch-transport-7.5.0/lib/elasticsearch/transport/transport/http/manticore.rb                         :71:in `perform_request'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/elasticsearch-transport-7.5.0/lib/elasticsearch/transport/client.rb:152:in `perform_r                         equest'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/elasticsearch-api-7.5.0/lib/elasticsearch/api/actions/search.rb:91:in `search'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-elasticsearch-4.9.1/lib/logstash/inputs/elasticsearch.rb:349:in `searc                         h_request'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-elasticsearch-4.9.1/lib/logstash/inputs/elasticsearch.rb:272:in `do_ru                         n_slice'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-elasticsearch-4.9.1/lib/logstash/inputs/elasticsearch.rb:250:in `do_ru                         n'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-elasticsearch-4.9.1/lib/logstash/inputs/elasticsearch.rb:238:in `run'
/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:405:in `inputworker'
/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:396:in `block in start_input'
5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.