Filebeat not trasnferring messages to Logstash

Elastic Stack Beats
1

HI Team,
I have completed ELK setup completed and successful in creating many visualtisation, however, offlate my filebeat is not sending any messages to my logstash system. I am getting below message when trying to publish

  ./filebeat -e -c filebeat.yml -d "publish"
2018-07-23T23:46:15.212-0700    INFO    instance/beat.go:468    Home path: [/opt/bea/ELKSTACK/filebeat-6.2.4-linux-x86_64] Config path: [/opt/bea/ELKSTACK/filebeat-6.2.4-linux-x86_64] Data path: [/opt/bea/ELKSTACK/filebeat-6.2.4-linux-x86_64/data] Logs path: [/opt/bea/ELKSTACK/filebeat-6.2.4-linux-x86_64/logs]
2018-07-23T23:46:15.212-0700    INFO    instance/beat.go:475    Beat UUID: 57d1129b-8370-49a0-80ca-15f15e15e377
2018-07-23T23:46:15.212-0700    INFO    instance/beat.go:213    Setup Beat: filebeat; Version: 6.2.4
2018-07-23T23:46:15.213-0700    INFO    pipeline/module.go:76   Beat name: STOXXXXX.corp.prk.com
2018-07-23T23:46:15.213-0700    INFO    [monitoring]    log/log.go:97   Starting metrics logging every 30s
2018-07-23T23:46:15.213-0700    INFO    instance/beat.go:301    filebeat start running.
2018-07-23T23:46:15.213-0700    INFO    registrar/registrar.go:73       No registry file found under: /opt/bea/ELKSTACK/filebeat-6.2.4-linux-x86_64/data/registry. Creating a new registry file.
2018-07-23T23:46:15.216-0700    INFO    registrar/registrar.go:110      Loading registrar data from /opt/bea/ELKSTACK/filebeat-6.2.4-linux-x86_64/data/registry
2018-07-23T23:46:15.216-0700    INFO    registrar/registrar.go:121      States Loaded from registrar: 0
2018-07-23T23:46:15.216-0700    WARN    beater/filebeat.go:261  Filebeat is unable to load the Ingest Node pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the Ingest Node pipelines or are using Logstash pipelines, you can ignore this warning.
2018-07-23T23:46:15.216-0700    INFO    crawler/crawler.go:48   Loading Prospectors: 1
2018-07-23T23:46:15.218-0700    INFO    log/prospector.go:111   Configured paths: [/opt/bea/Logwarhouse/Genacess/messages]
2018-07-23T23:46:15.218-0700    INFO    crawler/crawler.go:82   Loading and starting Prospectors completed. Enabled prospectors: 1
2018-07-23T23:46:15.218-0700    INFO    cfgfile/reload.go:127   Config reloader started
2018-07-23T23:46:15.218-0700    INFO    cfgfile/reload.go:219   Loading of config files completed.
2018-07-23T23:46:45.216-0700    INFO    [monitoring]    log/log.go:124  Non-zero metrics in the last 30s        {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":10,"time":11},"total":{"ticks":20,"time":26,"value":20},"user":{"ticks":10,"time":15}},"info":{"ephemeral_id":"e80fa1bc-b620-4750-bd55-e78defa06843","uptime":{"ms":30009}},"memstats":{"gc_next":4194304,"memory_alloc":1394592,"memory_total":2918704,"rss":14323712}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0},"reloads":1},"output":{"type":"logstash"},"pipeline":{"clients":1,"events":{"active":0}}},"registrar":{"states":{"current":0},"writes":1},"system":{"cpu":{"cores":4},"load":{"1":0.54,"15":0.25,"5":0.37,"norm":{"1":0.135,"15":0.0625,"5":0.0925}}}}}}
------------------------------------------------------------------------------------

Filebeat Input:
 -----------------

paths:
    - /opt/bea/Logwarhouse/Genacess/messages
include_lines: ['^Apigee']
close_inactive: 20m
multiline.pattern: '^\w{3}\s\s\d{2}'
multiline.negate: true
multiline.match: after

output.logstash:
  # The Logstash hosts
  hosts: ["localhost:5089"]
2

Tried to format your above config but I don't think I was successful. Could you paste your exact config? If the above is correct, there are issues with the indentation.

3

Yes the above details are right, can you help me by highlighting the mistakes(indentation misatkes)

4

My log file looks below

< Jul 2 02:30:03 10.66.9.201 [] 2018-07-02T09:30:03.003+0000 fez8b5754-125c-47g-850c-21ae56f6c080 Apigee-Edge - - 1530523803518|US-Common-Accounts|/v1-log/us/common/accounts/middlewareSOAP|POST|Debug|Response-Target|[Connection: close][Co
ntent-Length: 1174][Content-Type: text/xml;charset=utf-8][Date: Mon, 02 Jul 2018 09:30:03 GMT],|200|x4Jt3ak41J|GetAccount|10x.2xx.1xx.xx|c7373307-3457-6bff-a5a6-3e4ad69171e1|PostmanRuntime/6.1.6|<soapenv:Envelope xmlns:soapenv="http://sc
hemas.xmlsoap.org/soap/envelope/" xmlns:oas="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-secext-1.0.xsd" xmlns:add="http://schemas.xmlsoap.org/ws/2004/08/addressing" xmlns:v260="http://middleware.prk.com/XMLSchemas
/AccountGetDetails/v260">soapenv:Headeradd:Actionhttp://sharedservices.prk.com/AccountGetDetailsv260/sendSynchronousResponse</add:Action>add:Tohttp://test.GWSBETA.frk.com/</add:To><add:RelatesTo RelationshipType="wsa:Reply">c737330
7-3457-4bff-a5a6-3e4ad69171e1</add:RelatesTo>add:MessageID7fb537427dda11e8a0210a415f66000013185c3277000000</add:MessageID></soapenv:Header>soapenv:Bodysoapenv:FaultReceiverSystem error</faultstri
ng>COMN0025Account Not FoundAccount.GetDetailsHigh2018-07-02
T02:30:03.374-07:00FdNbr-AccNbr111-000000234512345</soapenv:Fault></soapenv:Body></soapenv:Envelope> />

Since i am looking for messages which as Apigee in them to be indexed, i have mentioned include pattern.

5

Another important point, i am working on two other log patterns and filebeat is still not sending the input.

Also everything was fine tine till last night, i created one index pattern.

6

You find a few valid examples with the correct indentation here in the getting started guide: https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-configuration.html

7

Hi,
I dont think i have made any intendation errors this time, but i still get the same error.

<
filebeat.prospectors:

Each - is a prospector. Most options can be set at the prospector level, so

you can use different prospectors for various configurations.

Below are the prospector specific configurations.

  • type: log

    Change to true to enable this prospector configuration.

    enabled: true

    Paths that should be crawled and fetched. Glob based paths.

    paths:

    • /home/pttartc/messages
      include_lines: ['^Apigee']
8

Could you format your config above with 3 ticks to make readable?

Also please run the beat with -d option which will enable debug output. This should tell us more.

9

Hi Ruflin,
I am able to fix it by myself, big thanks for identifying the problem with indentation though!

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.