Filebeat panics on CentOS 7.2 when stdin is defined as an input

Elastic Stack Beats
1

I can't seem to get stdin working as an input for filebeat when it's running as service under systemd.

[root@x.x.x.x log]# hostnamectl
   Static hostname:xxxx
Transient hostname:xxxx
         Icon name: computer-vm
           Chassis: vm
        Machine ID: f32e0af35637b5dfcbedcb0a1de8dca1
           Boot ID: 480bcc8e58e6492b92927c8528700be7
    Virtualization: xen
  Operating System: CentOS Linux 7 (Core)
       CPE OS Name: cpe:/o:centos:centos:7
            Kernel: Linux 3.10.0-327.18.2.el7.x86_64
      Architecture: x86-64
[root@x.x.x.x log]# filebeat -version
filebeat version 1.2.3 (amd64)

For reference, here is my configuration, two log prospectors and a stdin prospector:

############################# Filebeat ######################################
filebeat:
  prospectors:
    - input_type: stdin
    -
      input_type: log
      paths:
        - /var/lib/mesos/slave/slaves/*/frameworks/*/executors/*/runs/latest/stdout
      fields:
        log_level: stdout
    -
      input_type: log
      paths:
        - /var/lib/mesos/slave/slaves/*/frameworks/*/executors/*/runs/latest/stderr
      fields:
        log_level: stderr
  registry_file: /var/lib/filebeat/registry

###############################################################################
############################# Libbeat Config ##################################
# Base config file used by all other beats for using libbeat features

############################# Output ##########################################

# Configure what outputs to use when sending the data collected by the beat.
# Multiple outputs may be used.
output:

  ### Elasticsearch as output
  elasticsearch:
    # Array of hosts to connect to.
    # Scheme and port can be left out and will be set to the default (http and 9200)
    # In case you specify and additional path, the scheme is required: http://localhost:9200/path
    # IPv6 addresses should always be defined as: https://[2001:db8::1]:9200
    hosts: ["x.x.x.x:9200"]

    index: "filebeat"

    # A template is used to set the mapping in Elasticsearch
    # By default template loading is disabled and no template is loaded.
    # These settings can be adjusted to load your own template or overwrite existing ones
    # template:
    #  name: "filebeat"
    #  path: "/etc/filebeat/filebeat.template.json"
    #  overwrite: true

############################# Shipper #########################################

shipper:

############################# Logging #########################################

logging:
  to_syslog: true

Running this in debug mode throws no errors: https://gist.github.com/gaahrdner/0929a773337801c01a18251f7a1bfdc9

However, when restarting the service, filebeat panics: https://gist.github.com/gaahrdner/d29d15cacbb809f12dd5aa20ec305814

2

running filebeat as service fails, due to stdin being closed by systemd. Never use stdin, when running filebeat as standalone service. stdin is some special mode (inherited from logstash-forwarder) used to push generated content from scripts via filebeat to logstash/elasticsearch.

3

Ok, thanks for your response Steffen. I was attempting to follow the DC/OS documents for logging which specified using logstash-forwarder and stdin as input to accept entries from jouranlctl, so I'll need to figure out how to do this with filebeat or switch back to logstash-forwarder. I'd rather use filebeat, as you an imagine.

https://docs.mesosphere.com/1.7/administration/logging/elk/

4

How do you run filebeat? The document referenced by you says, write a logstash.sh script piping journald output to logstash-forwarder. And run logstash.sh script as a service (create systemd service file for custom script and so on). This can be done with filebeat too.In your script just run:

$ journalctl <all-your-opts> | filebeat -c /etc/filebeat/filebeat.yml

But you can not pipe to any process running somewhere in background unless you use a fifo, but no idea fifo files are supported by filebeat (most likely not).

1 Like
5

This topic was automatically closed after 21 days. New replies are no longer allowed.