Head malformed HTTP response

Elastic Stack Beats
1

Hi,

I tried filebeat version 1.0.0-rc2 (amd64), running on a Mac and pointed to ES 1.7.x.

It looks like it's reading the log file because it prints nice JSON to console at debug logging level.
However, I cannot get it to publish anything to ES, even though ES is configured as the output.

So then I used nc -k -l 1234 and pointed Filebeat to localhost:1234 to see what Filebeat sends to ES. Here's what I got:
$ nc -k -l 1234
HEAD / HTTP/1.1
Host: localhost:1234
User-Agent: Go-http-client/1.1

And after that - nothing.

Then at some point I noticed this:
2015/11/18 03:06:40.459154 single.go:121: INFO Connecting error publishing events (retrying): Head http://localhost:1234: malformed HTTP response ""
2015/11/18 03:06:41.060248 single.go:143: INFO send fail

So sounds like some specific HTTP response is expected by Filebeat for a HEAD request? Can anyone point me to what exactly is expected?

I also managed to get this output:

2015/11/18 03:09:33.126235 client.go:55: DBG send event
2015/11/18 03:09:33.126294 preprocess.go:37: DBG Start Preprocessing
2015/11/18 03:09:33.126504 publish.go:100: DBG Publish: {
.... JSON here
}
2015/11/18 03:09:33.126718 preprocess.go:94: DBG Forward preprocessed events
2015/11/18 03:09:33.126762 output.go:103: DBG output worker: publish 5 events

This implied that some events were sent. But I don't see any POSTs, PUTs, or GETs or anything being sent to my nc -k -l 1234 -- just that HEAD.

Is there any way to debug what's going on?

Thanks,
Otis

2

@otisg I tried to reproduce this with elasticsearch 1.7.3 and 1.0.0-rc2 on Mac OS X. Could you share your the config file you are using? We changed the config option for the output with rc2, means the enabled flag was removed. To disable an output, it must now be commented out. As you say it is printing out packages to the console, it could be that you have other outputs enabled? Of course, it then should still send to ES.

It looks like you already enabled debugging, which is normally the best way to figure out what is going on.

3

During the connect phase Filebeat sends a HEAD request as a ping mechanism to check if Elasticsearch is up before sending it data. So the behaviour with nc is expected, I think.

For debugging I usually just use Elasticsearch and use ngrep to see what requests/responses are sent. Something like: ngrep -d lo -Wbyline "" "port 9200".

4

right, HEAD is used as ping to see if elasticsearch server will respond. here elasticsearch should response with a message containing some headers only, just like HTTP protocol demands. You have elasticsearch running behind a firewall/proxy? Sometimes these filter out non GET/PUT messages.

The debug messages about to be published events are internal to libbeat the events entering the publisher pipeline. In last phase events are forwarded to output plugin (elasticsearch).

5

Can you try to point your browser to http://localhost:1234 (port elasticsearch is listening on)? You get a JSON response?