Elastic, Kibana, filebeat failed sending data to elastic

Elastic Stack Beats
1

I have a problem with sending data from filebeat to elasticsearch

filebeat ver. filebeat-6.6.2-linux-x86_64

./filebeat setup

Loaded index template
Loading dashboards (Kibana must be running and reachable)
Loaded dashboards
Loaded machine learning job configurations

./filebeat test output

elasticsearch: http://xx.xx.xx.xx:9200...
parse url... OK
connection...
parse host... OK
dns lookup... OK
addresses: xx.xx.xx.xx
dial up... OK
TLS... WARN secure connection disabled
talk to server... OK
version: 6.5.1

./filebeat modules list

Enabled:
apache2

Disabled:
auditd
elasticsearch
haproxy
icinga
iis
kafka
kibana
logstash
mongodb
mysql
nginx
osquery
postgresql
redis
suricata
system
traefik

./filebeat export config

filebeat:
  config:
    modules:
      path: /xxx/xxx/filebeat-6.6.2-linux-x86_64/modules.d/*.yml
      reload:
        enabled: false
  inputs:
  - enabled: false
    paths:
    - /home/xxx/logs/*.log
    type: log
output:
  elasticsearch:
    hosts:
    - xx.xx.xx.xx:9200
path:
  config: /xxx/xxx/filebeat-6.6.2-linux-x86_64
  data: /xxx/xxx/filebeat-6.6.2-linux-x86_64/data
  home: /xxx/xxx/filebeat-6.6.2-linux-x86_64
  logs: /xxx/xxx/filebeat-6.6.2-linux-x86_64/logs
processors:
- add_host_metadata: null
- add_cloud_metadata: null
setup:
  kibana:
    host: xx.xx.xx.xx:8086
  template:
    settings:
      index:
        number_of_shards: 3

configuration apache module

- module: apache2
  # Access logs
  access:
    enabled: true

    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    var.paths: ["/xxx/xxx/logs/http_access.log"]

  # Error logs
  error:
    enabled: true

    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    var.paths: ["/xxx/xxx/logs/http_error.log"]

When i start filebeat i have an error
./filebeat -e

2019-03-28T09:58:08.354+0100    INFO    crawler/crawler.go:72   Loading Inputs: 1
2019-03-28T09:58:08.358+0100    INFO    log/input.go:138        Configured paths: [/xxx/xxx/logs/http_error.log]
2019-03-28T09:58:08.359+0100    INFO    log/input.go:138        Configured paths: [/xxx/xxx/logs/http_access.log]
2019-03-28T09:58:08.359+0100    INFO    crawler/crawler.go:106  Loading and starting Inputs completed. Enabled inputs: 0
2019-03-28T09:58:08.359+0100    INFO    cfgfile/reload.go:150   Config reloader started
2019-03-28T09:58:08.363+0100    INFO    log/input.go:138        Configured paths: [/xxx/xxx/logs/http_access.log]
2019-03-28T09:58:08.364+0100    INFO    log/input.go:138        Configured paths: [/xxx/xxx/logs/http_error.log]
2019-03-28T09:58:08.364+0100    INFO    elasticsearch/client.go:165     Elasticsearch url: http://xx.xx.xx.xx:9200
2019-03-28T09:58:08.366+0100    INFO    elasticsearch/client.go:721     Connected to Elasticsearch version 6.5.1
2019-03-28T09:58:08.367+0100    ERROR   fileset/factory.go:142  Error loading pipeline: Error loading pipeline for fileset apache2/access: This module requires the following Elasticsearch plugins: ingest-user-agent, ingest-geoip. You can install them by running the following commands on all the Elasticsearch nodes:
    sudo bin/elasticsearch-plugin install ingest-user-agent
    sudo bin/elasticsearch-plugin install ingest-geoip

I also install in elasticsearch this modules
./bin/elasticsearch-plugin list

ingest-geoip
ingest-user-agent

And I have still this problem. What I have to do with this.

2

Hi All,

I am using filebeat to forward data coming from an external splunk server to logstash. Can someone please share a good configuration which can include the meta data information of the logs as well.

==

filebeat.inputs:

  • type: tcp

max_message_size: 10MiB

host: "X.X.X.X.X:9200"

#since log is coming from a splunk server this is working fine, however the meta data info is not being carried over to my elastic search.

output.elasticsearch:

hosts: ["http://X.X.X.X:9200/"]

output.elasticsearch.index: "filebeat-%{[beat.version]}-%{+yyyy.MM.dd}"

setup.template.name: "filebeat"

setup.template.pattern: "filebeat-*"

===

3

This is an ES problem, please post your question in the appropriate subforum.

4

ok I change it.

5

I moved your question to #beats:filebeat.

1 Like
6

Can someone can help me to solve this problem?

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.