Haproxy.source not being logged

[RainTown (Kevin Maguire)]

I am using latest (6.7) ES, Kibana, and Filebeat.

The grok patterns in pipeline.json for http log format contain

%{IPORHOST:haproxy.source}

and my log files seem to match OK, but this field is not being logged into my ES? is this intended? I see the field as searchable string field in filebeat-* index-patterns.

I am running filebeat on saved log files piping straight into ES, no logstash.

I have many haproxy instances and it's important to be able to distinguish them somehow in visualisations.

Ideas?

[RainTown (Kevin Maguire)]

So, using the 7.0.0rc2 version of filebeat and elasticsearch I got it to work with my files using this grok:

"(%{IPORHOST:haproxy.source} %{NOTSPACE:process.name}\\[%{NUMBER:process.pid:long}\\]: )?%{IP:source.address}:%{NUMBER:source.port:long} \\[%{NOTSPACE:haproxy.request_date}\\] %{NOTSPACE:haproxy.frontend_name} %{NOTSPACE:haproxy.backend_name}/%{NOTSPACE:haproxy.server_name} %{NUMBER:haproxy.http.request.time_wait_ms:long}/%{NUMBER:haproxy.total_waiting_time_ms:long}/%{NUMBER:haproxy.connection_wait_time_ms:long}/%{NUMBER:haproxy.http.request.time_wait_without_data_ms:long}/%{NUMBER:temp.duration:long} %{NUMBER:haproxy.http.response.status_code:long} %{NUMBER:haproxy.bytes_read:long} %{NOTSPACE:haproxy.http.request.captured_cookie} %{NOTSPACE:haproxy.http.response.captured_cookie} %{NOTSPACE:haproxy.termination_state} %{NUMBER:haproxy.connections.active:long}/%{NUMBER:haproxy.connections.frontend:long}/%{NUMBER:haproxy.connections.backend:long}/%{NUMBER:haproxy.connections.server:long}/%{NUMBER:haproxy.connections.retries:long} %{NUMBER:haproxy.server_queue:long}/%{NUMBER:haproxy.backend_queue:long} (\\{%{DATA:haproxy.http.request.captured_headers}\\} \\{%{DATA:haproxy.http.response.captured_headers}\\} |\\{%{DATA}\\} )?\"%{GREEDYDATA:haproxy.http.request.raw_request_line}\""

Might help someone else down the line ...