Unable to retreive data from index using elasticsearch input plugin in logstash

Elastic Stack Logstash
1 
    input {
  elasticsearch {
    hosts => ["172.25.8.45:9200"]
    index => ["heartbeat-*"]
   # query => '{ "query": { "match_all": {} } }'
   # query => '{ "query": { "bool": {"must" : { "term" : { "monitor.status" : "down" } } } } }'
   # query => '{ "query": { "query_string": { "query": "*" } } }'
   # size => 500
   # scroll => "5m"
    #docinfo => true
  }
}
output {
   # if "_grokparsefailure" in [tags] {
  stdout  { codec => "rubydebug" }
 # }
    elasticsearch {
    hosts => ["172.25.8.45:9200"]
    manage_template => false
    index => "tp"
   # document_type => "new-type"
     }
}

This is my logstash configuration

Im getting following error:

[2019-05-16T16:15:59,334][ERROR][logstash.pipeline        ] A plugin had an unrecoverable error. Will restart this plugin.
  Pipeline_id:main
  Plugin: <LogStash::Inputs::Elasticsearch index=>"heartbeat-*", hosts=>["172.25.8.45:9200"], id=>"49529869826fbe6db5a4ea023f4022b6ebb73e17e4ff1830d2ffb45b9b292152", enable_metric=>true, codec=><LogStash::Codecs::JSON id=>"json_ef14a6ee-dfb5-4070-9a52-faa253c049c8", enable_metric=>true, charset=>"UTF-8">, query=>"{ \"sort\": [ \"_doc\" ] }", size=>1000, scroll=>"1m", docinfo=>false, docinfo_target=>"@metadata", docinfo_fields=>["_index", "_type", "_id"], ssl=>false>
  Error: [411] <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
<HTML><HEAD><META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=iso-8859-1">
<TITLE>ERROR: The requested URL could not be retrieved</TITLE>
<STYLE type="text/css"><!--BODY{background-color:#ffffff;font-family:verdana,sans-serif}PRE{font-family:sans-serif}--></STYLE>
</HEAD><BODY>
<H1>ERROR</H1>
<H2>The requested URL could not be retrieved</H2>
<HR noshade size="1px">
<P>
While trying to process the request:
<PRE>
GET /heartbeat-%2A/_search?scroll=1m&amp;size=1000 HTTP/1.1
Content-Type: application/json
User-Agent: Faraday v0.9.2
Accept-Encoding: gzip;q=1.0,deflate;q=0.6,identity;q=0.3
Accept: */*
Connection: close
Host: 172.25.8.45:9200
Content-Length: 22

</PRE>
<P>
The following error was encountered:
<UL>
<LI>
<STRONG>
Invalid Request
</STRONG>
</UL>

<P>
Some aspect of the HTTP Request is invalid.  Possible problems:
<UL>
<LI>Missing or unknown request method
<LI>Missing URL
<LI>Missing HTTP Identifier (HTTP/1.0)
<LI>Request is too large
<LI>Content-Length missing for POST or PUT requests
<LI>Illegal character in hostname; underscores are not allowed
</UL>
<P>Your cache administrator is <A HREF="mailto:root">root</A>. 

<BR clear="all">
<HR noshade size="1px">
<ADDRESS>
Generated Thu, 16 May 2019 09:00:09 GMT by TGPROXY-Backup (squid/2.6.STABLE6)
</ADDRESS>
</BODY></HTML>

  Exception: Elasticsearch::Transport::Transport::Errors::LengthRequired
  Stack: /home/abc/elk/logstash-6.5.3/vendor/bundle/jruby/2.3.0/gems/elasticsearch-transport-5.0.5/lib/elasticsearch/transport/transport/base.rb:202:in `__raise_transport_error'
/home/abc/elk/logstash-6.5.3/vendor/bundle/jruby/2.3.0/gems/elasticsearch-transport-5.0.5/lib/elasticsearch/transport/transport/base.rb:319:in `perform_request'
/home/abc/elk/logstash-6.5.3/vendor/bundle/jruby/2.3.0/gems/elasticsearch-transport-5.0.5/lib/elasticsearch/transport/transport/http/faraday.rb:20:in `perform_request'
/home/abc/elk/logstash-6.5.3/vendor/bundle/jruby/2.3.0/gems/elasticsearch-transport-5.0.5/lib/elasticsearch/transport/client.rb:131:in `perform_request'
/home/abc/elk/logstash-6.5.3/vendor/bundle/jruby/2.3.0/gems/elasticsearch-api-5.0.5/lib/elasticsearch/api/actions/search.rb:183:in `search'
/home/abc/elk/logstash-6.5.3/vendor/bundle/jruby/2.3.0/gems/logstash-input-elasticsearch-4.2.1/lib/logstash/inputs/elasticsearch.rb:200:in `do_run'
/home/abc/elk/logstash-6.5.3/vendor/bundle/jruby/2.3.0/gems/logstash-input-elasticsearch-4.2.1/lib/logstash/inputs/elasticsearch.rb:188:in `run'
/home/abc/elk/logstash-6.5.3/logstash-core/lib/logstash/pipeline.rb:409:in `inputworker'
/home/abc/elk/logstash-6.5.3/logstash-core/lib/logstash/pipeline.rb:403:in `block in start_input'
2

There is a squid proxy between you and elasticsearch. Is there any indication in the elasticsearch logs that the request is reaching it?

3

No there is no indication in elasticsearch logs

4

That makes me think the problem is in the proxy. Can you not bypass the proxy?

5

Thanking for you reply but i didnt understand what bypassing proxy mean

6

You are connecting to 172.25.8.45:9200 and your post suggests that is a Squid proxy, not an elasticsearch instance. I am suggesting you should connect directly to elasticsearch, not to squid.

7

Ok understood thank you so much

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.