Could not retrieve remote IP address for beats input

gotjoshua · October 22, 2017, 6:36pm

"Could not retrieve remote IP address for beats input."

logstash-plugins/logstash-input-beats/blob/master/lib/logstash/inputs/beats/message_listener.rb#L34




  @nocodec_transformer = RawEventTransform.new(@input)
  @codec_transformer = DecodedEventTransform.new(@input)
end


def onNewMessage(ctx, message)
  hash = message.getData
  ip_address = ip_address(ctx)


  hash['@metadata']['ip_address'] = ip_address unless ip_address.nil? || hash['@metadata'].nil?
  target_field = extract_target_field(hash)


  if target_field.nil?
    event = LogStash::Event.new(hash)
    @nocodec_transformer.transform(event)
    @queue << event
  else
    codec(ctx).accept(CodecCallbackListener.new(target_field,
                                                hash,
                                                message.getIdentityStream(),
                                                @codec_transformer,

I see this warning regularly. (even though the comment says it should never happen)

My setup is 6.0.0-rc1
beats are running on a remote server in docker containers.

ELK stack is deployed with this docker-compose repo:
decomposed/elk
It uses the official elastic docker images.

Any advice how to troubleshoot this error message?

gotjoshua · October 23, 2017, 3:47pm

More details:
In bursts of less than 0.1 seconds, I get around 30 of these errors:

[2017-10-23T14:59:17,540][WARN ][logstash.inputs.beats ] Could not retrieve remote IP address for beats input.
... 30 more...
[2017-10-23T14:59:17,549][WARN ][logstash.inputs.beats ] Could not retrieve remote IP address for beats input.

There are 3 events that were stashed at 14:59:17.019 - one ajax request (got a 200) and two gitlab runner requests (got 204s) All three have remote_ip fields with valid ips (one is a 172 address from another docker container on the same box)

Again I ask for help to troubleshoot/investigate, as the error does not explicitly tie itself to a specific request...

rzr_mallox · November 16, 2017, 12:50am

After updating to version 6 of the ELK stack I've had similar issues, along with documents not being able to be indexed because they have 2 document types (this seems to be related to old plugins not having been properly updated).

Have you found any fix for this?

lcorsini · November 16, 2017, 12:42pm

Both errors happen for me too

[2017-11-16T13:38:41,327][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"filebeat-2017.11.15", :_type=>"doc", :_routing=>nil}, #<LogStash::Event:0x3f81832e>], :response=>{"index"=>{"_index"=>"filebeat-2017.11.15", "_type"=>"doc", "_id"=>"uH7WxF8BDa1ley8Gmc9y", "status"=>400, "error"=>{"type"=>"illegal_argument_exception", "reason"=>"Rejecting mapping update to [filebeat-2017.11.15] as the final mapping would have more than 1 type: [log, doc]"}}}}
[2017-11-16T13:38:41,328][WARN ][logstash.inputs.beats    ] Could not retrieve remote IP address for beats input.

Hush077 · November 17, 2017, 1:40am

Same issue opened a ticket here: https://github.com/logstash-plugins/logstash-input-beats/issues/269

Hush077 · November 17, 2017, 2:43am

Going to: vim /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-beats-5.0.2-java/lib/logstash/inputs/beats/message_listener.rb

And deleting:

begin
    hash.get("@metadata").put("ip_address", ctx.channel().remoteAddress().getAddress().getHostAddress())
  rescue #should never happen, but don't allow an error here to stop beats input
    input.logger.warn("Could not retrieve remote IP address for beats input.")
end

Fixes the issue.

system · December 15, 2017, 2:43am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.