Logstash not receiving beats input

Elastic Stack Logstash
1

Hello,

i have no issues importing data from beats to elasticsearch, however after changing output to logstash in the beats config, logstash do not import data to elasticsearch. No stdout is printed in the shell running logstash.
Please help,thanks.

Filebeat config file:

#==========================  Modules configuration ============================
filebeat.modules:




#------------------------------- Osquery Module ------------------------------
- module: osquery
  result:
    enabled: true

   






#=========================== Filebeat prospectors =============================

   



#------------------------------ Log prospector --------------------------------
- type: log

  
  enabled: true

   paths:
- /var/log/*.log
- C:\Users\Downloads\data\Test\*

output.logstash:
  
  hosts: ["localhost:5044"]

Logstash config file:

input {
  beats {
    port => 5044
  }
}



output {
  elasticsearch {
    hosts => ["localhost:9200"]
	manage_template => false
    index => "beatslogstashsample"
	
  }
  stdout { codec => rubydebug }
}
2

Is Filebeat able to connect to logstash? You should look into filebeat and logstash logs to see if that is the case.

3

seems like logstash receive filebit input but the index is not found in elasticsearch.
last few lines of Output in shell running logstash:

[2018-07-05T17:04:18,793][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://localhost:9200/"}
[2018-07-05T17:04:18,934][INFO ][logstash.outputs.elasticsearch] ES Output version determined {:es_version=>6}
[2018-07-05T17:04:18,934][WARN ][logstash.outputs.elasticsearch] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>6}
[2018-07-05T17:04:19,012][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//localhost:9200"]}
[2018-07-05T17:04:20,100][INFO ][logstash.inputs.beats    ] Beats inputs: Starting input listener {:address=>"0.0.0.0:5044"}
[2018-07-05T17:04:20,407][INFO ][logstash.pipeline        ] Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x2d4ceacd run>"}
[2018-07-05T17:04:20,594][INFO ][org.logstash.beats.Server] Starting server on port: 5044
[2018-07-05T17:04:20,795][INFO ][logstash.agent           ] Pipelines running {:count=>1, :pipelines=>["main"]}
4

This tells that your logstash pipeline has been correctly instantiated, but do filebeat logs also tell that filebeat can reach logstash?
Filebeat logs are normally in /var/log/filebeat.

6

Have made some changes in the filebeats config file,logs are now shown in filebeats debugging but index is not shown in elasticsearch. Stdout are still not printed in shell running logstash.Please advice.Thanks.

Filebeats config:

#=========================== Filebeat inputs =============================

filebeat.inputs:

# Each - is an input. Most options can be set at the input level, so
# you can use different inputs for various configurations.
# Below are the input specific configurations.

- type: log

  # Change to true to enable this input configuration.
  enabled: true

  # Paths that should be crawled and fetched. Glob based paths.
  paths:
    #- /var/log/*.log
    - C:\Users\Downloads\data\Test\*.log


#============================= Filebeat modules ===============================

filebeat.config.modules:
  # Glob pattern for configuration loading
  path: ${path.config}/modules.d/*.yml

  # Set to true to enable config reloading
  reload.enabled: false

#----------------------------- Logstash output --------------------------------
output.logstash:
  hosts: ["localhost:5044"]
7

filebeats debugging output:

2018-07-09T17:52:40.787+0800    INFO    instance/beat.go:492    Home path: [C:\Users \Downloads\ELK\filebeat-6.3.0-windows-x86_64] Config path: [C:\Users \Downloads\ELK\filebeat-6.3.0-windows-x86_64] Data path: [C:\Users \Downloads\ELK\filebeat-6.3.0-windows-x86_64\data] Logs path: [C:\Users \Downloads\ELK\filebeat-6.3.0-windows-x86_64\logs]
2018-07-09T17:52:40.945+0800    INFO    instance/beat.go:499    Beat UUID: b0025e4f-fa74-4346-881e-c8fea05855fd
2018-07-09T17:52:40.954+0800    INFO    [beat]  instance/beat.go:716    Beat info       {"system_info": {"beat": {"path": {"config": "C:\\Users \\Downloads\\ELK\\filebeat-6.3.0-windows-x86_64", "data": "C:\\Users \\Downloads\\ELK\\filebeat-6.3.0-windows-x86_64\\data", "home": "C:\\Users \\Downloads\\ELK\\filebeat-6.3.0-windows-x86_64", "logs": "C:\\Users \\Downloads\\ELK\\filebeat-6.3.0-windows-x86_64\\logs"}, "type": "filebeat", "uuid": "b0025e4f-fa74-4346-881e-c8fea05855fd"}}}
2018-07-09T17:52:40.956+0800    INFO    [beat]  instance/beat.go:725    Build info      {"system_info": {"build": {"commit": "a04cb664d5fbd4b1aab485d1766f3979c138fd38", "libbeat": "6.3.0", "time": "2018-06-11T22:34:03.000Z", "version": "6.3.0"}}}
2018-07-09T17:52:40.957+0800    INFO    [beat]  instance/beat.go:728    Go runtime info {"system_info": {"go": {"os":"windows","arch":"amd64","max_procs":4,"version":"go1.9.4"}}}
2018-07-09T17:52:41.031+0800    INFO    instance/beat.go:225    Setup Beat: filebeat; Version: 6.3.0
2018-07-09T17:52:41.036+0800    DEBUG   [publish]       pipeline/consumer.go:120        start pipeline event consumer
2018-07-09T17:52:41.036+0800    INFO    pipeline/module.go:81   Beat name: CPX-YLR7N6AXN6H
2018-07-09T17:52:41.058+0800    INFO    instance/beat.go:315    filebeat start running.
2018-07-09T17:52:41.067+0800    INFO    registrar/registrar.go:75       No registry file found under: C:\Users \Downloads\ELK\filebeat-6.3.0-windows-x86_64\data\registry. Creating a new registry file.
2018-07-09T17:52:41.058+0800    INFO    [monitoring]    log/log.go:97   Starting metrics logging every 30s
2018-07-09T17:52:41.132+0800    INFO    registrar/registrar.go:112      Loading registrar data from C:\Users \Downloads\ELK\filebeat-6.3.0-windows-x86_64\data\registry
2018-07-09T17:52:41.139+0800    INFO    registrar/registrar.go:123      States Loaded from registrar: 0
2018-07-09T17:52:41.164+0800    WARN    beater/filebeat.go:354  Filebeat is unable to load the Ingest Node pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the Ingest Node pipelines or are using Logstash pipelines, you can ignore this warning.
2018-07-09T17:52:41.171+0800    INFO    crawler/crawler.go:48   Loading Inputs: 1
2018-07-09T17:52:41.192+0800    INFO    log/input.go:111        Configured paths: [C:\Users \Downloads\data\Test\*.log]
2018-07-09T17:52:41.202+0800    INFO    input/input.go:87       Starting input of type: log; ID: 7110464235258394879
2018-07-09T17:52:41.224+0800    INFO    cfgfile/reload.go:122   Config reloader started
2018-07-09T17:52:41.228+0800    INFO    crawler/crawler.go:82   Loading and starting Inputs completed. Enabled inputs: 1
2018-07-09T17:52:41.232+0800    INFO    log/harvester.go:228    Harvester started for file: C:\Users\ Downloads\data\Test\yoyoyo2.log
2018-07-09T17:52:41.232+0800    INFO    log/harvester.go:228    Harvester started for file: C:\Users \Downloads\data\Test\yoyoyo.log
2018-07-09T17:52:41.243+0800    INFO    cfgfile/reload.go:214   Loading of config files completed.
2018-07-09T17:52:41.280+0800    DEBUG   [publish]       pipeline/processor.go:291       Publish event: {
  "@timestamp": "2018-07-09T09:52:41.278Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "doc",
    "version": "6.3.0"
  },
  "offset": 184,
  "message": "WebContainer : 312 - 2018-06-27 11:00:52.443 DEBUG c.i.g.s.s.c.l.PerfLogger:24 - com.ida.gov.sg.spcp.client.gemalto.BaseGemaltoConnector | delete token | Time taken (in milli Sec):10",
  "prospector": {
    "type": "log"
  },
  "input": {
    "type": "log"
  },
  "beat": {
    "name": "CPX-YLR7N6AXN6H",
    "hostname": "CPX-YLR7N6AXN6H",
    "version": "6.3.0"
  },
  "host": {
    "name": "CPX-YLR7N6AXN6H"
  },
  "source": "C:\\Users \\Downloads\\data\\Test\\yoyoyo2.log"
}
2018-07-09T17:53:11.494+0800    INFO    [monitoring]    log/log.go:124  Non-zero metrics in the last 30s        {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":593,"time":{"ms":593}},"total":{"ticks":843,"time":{"ms":843},"value":843},"user":{"ticks":250,"time":{"ms":250}}},"info":{"ephemeral_id":"b8d4ea34-08f1-4554-ad20-b304ff846b56","uptime":{"ms":30529}},"memstats":{"gc_next":5794656,"memory_alloc":2950496,"memory_total":5153512,"rss":23564288}},"filebeat":{"events":{"added":16,"done":16},"harvester":{"open_files":2,"running":2,"started":2}},"libbeat":{"config":{"module":{"running":0},"reloads":1},"output":{"events":{"acked":14,"batches":1,"total":14},"read":{"bytes":6},"type":"logstash","write":{"bytes":1025}},"pipeline":{"clients":1,"events":{"active":0,"filtered":2,"published":14,"retry":14,"total":16},"queue":{"acked":14}}},"registrar":{"states":{"current":2,"update":16},"writes":4},"system":{"cpu":{"cores":4}}}}}
2018-07-09T17:53:41.108+0800    INFO    [monitoring]    log/log.go:124  Non-zero metrics in the last 30s        {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":625,"time":{"ms":32}},"total":{"ticks":890,"time":{"ms":47},"value":890},"user":{"ticks":265,"time":{"ms":15}}},"info":{"ephemeral_id":"b8d4ea34-08f1-4554-ad20-b304ff846b56","uptime":{"ms":60531}},"memstats":{"gc_next":5794656,"memory_alloc":3069128,"memory_total":5272144,"rss":2207744}},"filebeat":{"harvester":{"open_files":2,"running":2}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":1,"events":{"active":0}}},"registrar":{"states":{"current":2}}}}}
8

Deleted old registry files in the filebeats data folder each time before running filebeats,this seems to solve the issue.

1 Like
9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.