# Logstash not receiving beats input

**URL:** <https://discuss.elastic.co/t/logstash-not-receiving-beats-input/138652>\
**Category:** Logstash\
**Created:** [July 5, 2018, 7:36am UTC](https://discuss.elastic.co/t/logstash-not-receiving-beats-input/138652 "2018-07-05T07:36:01Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![bobbytan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bobbytan/32/32720_2.png) [@bobbytan](https://discuss.elastic.co/u/bobbytan)\
**Post date:** [July 5, 2018, 7:36am UTC](https://discuss.elastic.co/t/logstash-not-receiving-beats-input/138652/1 "2018-07-05T07:36:01Z")

</div>

Hello,

i have no issues importing data from beats to elasticsearch, however after changing output to logstash in the beats config, logstash do not import data to elasticsearch. No stdout is printed in the shell running logstash.  
Please help,thanks.

**Filebeat config file:**

```
#========================== Modules configuration ============================
filebeat.modules:

#------------------------------- Osquery Module ------------------------------
- module: osquery
  result:
    enabled: true

   

#=========================== Filebeat prospectors =============================

   

#------------------------------ Log prospector --------------------------------
- type: log

  
  enabled: true

   paths:
- /var/log/*.log
- C:\Users\Downloads\data\Test\*

output.logstash:
  
  hosts: ["localhost:5044"]

```

**Logstash config file:**

```
input {
  beats {
    port => 5044
  }
}

output {
  elasticsearch {
    hosts => ["localhost:9200"]
	manage_template => false
    index => "beatslogstashsample"
	
  }
  stdout { codec => rubydebug }
}
```

---

<div class="post-metadata">

**Author:** ![Dheeraj\_Gupta](https://avatars.discourse-cdn.com/v4/letter/d/49beb7/32.png) [@Dheeraj\_Gupta](https://discuss.elastic.co/u/Dheeraj_Gupta)\
**Post date:** [July 5, 2018, 8:53am UTC](https://discuss.elastic.co/t/logstash-not-receiving-beats-input/138652/2 "2018-07-05T08:53:13Z")

</div>

Is Filebeat able to connect to logstash? You should look into filebeat and logstash logs to see if that is the case.

---

<div class="post-metadata">

**Author:** ![bobbytan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bobbytan/32/32720_2.png) [@bobbytan](https://discuss.elastic.co/u/bobbytan)\
**Post date:** [July 5, 2018, 9:08am UTC](https://discuss.elastic.co/t/logstash-not-receiving-beats-input/138652/3 "2018-07-05T09:08:54Z")

</div>

seems like logstash receive filebit input but the index is not found in elasticsearch.  
**last few lines of Output in shell running logstash:**

```
[2018-07-05T17:04:18,793][WARN][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://localhost:9200/"}
[2018-07-05T17:04:18,934][INFO][logstash.outputs.elasticsearch] ES Output version determined {:es_version=>6}
[2018-07-05T17:04:18,934][WARN][logstash.outputs.elasticsearch] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>6}
[2018-07-05T17:04:19,012][INFO][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//localhost:9200"]}
[2018-07-05T17:04:20,100][INFO][logstash.inputs.beats] Beats inputs: Starting input listener {:address=>"0.0.0.0:5044"}
[2018-07-05T17:04:20,407][INFO][logstash.pipeline] Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x2d4ceacd run>"}
[2018-07-05T17:04:20,594][INFO][org.logstash.beats.Server] Starting server on port: 5044
[2018-07-05T17:04:20,795][INFO][logstash.agent] Pipelines running {:count=>1, :pipelines=>["main"]}
```

---

<div class="post-metadata">

**Author:** ![Dheeraj\_Gupta](https://avatars.discourse-cdn.com/v4/letter/d/49beb7/32.png) [@Dheeraj\_Gupta](https://discuss.elastic.co/u/Dheeraj_Gupta)\
**Post date:** [July 5, 2018, 10:04am UTC](https://discuss.elastic.co/t/logstash-not-receiving-beats-input/138652/4 "2018-07-05T10:04:35Z")

</div>

This tells that your logstash pipeline has been correctly instantiated, but do filebeat logs also tell that filebeat can reach logstash?  
Filebeat logs are normally in `/var/log/filebeat`.

---

<div class="post-metadata">

**Author:** ![bobbytan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bobbytan/32/32720_2.png) [@bobbytan](https://discuss.elastic.co/u/bobbytan)\
**Post date:** [July 9, 2018, 3:37am UTC](https://discuss.elastic.co/t/logstash-not-receiving-beats-input/138652/6 "2018-07-09T03:37:00Z")

</div>

Have made some changes in the filebeats config file,logs are now shown in filebeats debugging but index is not shown in elasticsearch. Stdout are still not printed in shell running logstash.Please advice.Thanks.

**Filebeats config:**

```
#=========================== Filebeat inputs =============================

filebeat.inputs:

# Each - is an input. Most options can be set at the input level, so
# you can use different inputs for various configurations.
# Below are the input specific configurations.

- type: log

  # Change to true to enable this input configuration.
  enabled: true

  # Paths that should be crawled and fetched. Glob based paths.
  paths:
    #- /var/log/*.log
    - C:\Users\Downloads\data\Test\*.log

#============================= Filebeat modules ===============================

filebeat.config.modules:
  # Glob pattern for configuration loading
  path: ${path.config}/modules.d/*.yml

  # Set to true to enable config reloading
  reload.enabled: false

#----------------------------- Logstash output --------------------------------
output.logstash:
  hosts: ["localhost:5044"]
```

---

<div class="post-metadata">

**Author:** ![bobbytan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bobbytan/32/32720_2.png) [@bobbytan](https://discuss.elastic.co/u/bobbytan)\
**Post date:** [July 9, 2018, 10:46am UTC](https://discuss.elastic.co/t/logstash-not-receiving-beats-input/138652/7 "2018-07-09T10:46:39Z")

</div>

**filebeats debugging output:**

```
2018-07-09T17:52:40.787+0800 INFO instance/beat.go:492 Home path: [C:\Users \Downloads\ELK\filebeat-6.3.0-windows-x86_64] Config path: [C:\Users \Downloads\ELK\filebeat-6.3.0-windows-x86_64] Data path: [C:\Users \Downloads\ELK\filebeat-6.3.0-windows-x86_64\data] Logs path: [C:\Users \Downloads\ELK\filebeat-6.3.0-windows-x86_64\logs]
2018-07-09T17:52:40.945+0800 INFO instance/beat.go:499 Beat UUID: b0025e4f-fa74-4346-881e-c8fea05855fd
2018-07-09T17:52:40.954+0800 INFO [beat] instance/beat.go:716 Beat info {"system_info": {"beat": {"path": {"config": "C:\\Users \\Downloads\\ELK\\filebeat-6.3.0-windows-x86_64", "data": "C:\\Users \\Downloads\\ELK\\filebeat-6.3.0-windows-x86_64\\data", "home": "C:\\Users \\Downloads\\ELK\\filebeat-6.3.0-windows-x86_64", "logs": "C:\\Users \\Downloads\\ELK\\filebeat-6.3.0-windows-x86_64\\logs"}, "type": "filebeat", "uuid": "b0025e4f-fa74-4346-881e-c8fea05855fd"}}}
2018-07-09T17:52:40.956+0800 INFO [beat] instance/beat.go:725 Build info {"system_info": {"build": {"commit": "a04cb664d5fbd4b1aab485d1766f3979c138fd38", "libbeat": "6.3.0", "time": "2018-06-11T22:34:03.000Z", "version": "6.3.0"}}}
2018-07-09T17:52:40.957+0800 INFO [beat] instance/beat.go:728 Go runtime info {"system_info": {"go": {"os":"windows","arch":"amd64","max_procs":4,"version":"go1.9.4"}}}
2018-07-09T17:52:41.031+0800 INFO instance/beat.go:225 Setup Beat: filebeat; Version: 6.3.0
2018-07-09T17:52:41.036+0800 DEBUG [publish] pipeline/consumer.go:120 start pipeline event consumer
2018-07-09T17:52:41.036+0800 INFO pipeline/module.go:81 Beat name: CPX-YLR7N6AXN6H
2018-07-09T17:52:41.058+0800 INFO instance/beat.go:315 filebeat start running.
2018-07-09T17:52:41.067+0800 INFO registrar/registrar.go:75 No registry file found under: C:\Users \Downloads\ELK\filebeat-6.3.0-windows-x86_64\data\registry. Creating a new registry file.
2018-07-09T17:52:41.058+0800 INFO [monitoring] log/log.go:97 Starting metrics logging every 30s
2018-07-09T17:52:41.132+0800 INFO registrar/registrar.go:112 Loading registrar data from C:\Users \Downloads\ELK\filebeat-6.3.0-windows-x86_64\data\registry
2018-07-09T17:52:41.139+0800 INFO registrar/registrar.go:123 States Loaded from registrar: 0
2018-07-09T17:52:41.164+0800 WARN beater/filebeat.go:354 Filebeat is unable to load the Ingest Node pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the Ingest Node pipelines or are using Logstash pipelines, you can ignore this warning.
2018-07-09T17:52:41.171+0800 INFO crawler/crawler.go:48 Loading Inputs: 1
2018-07-09T17:52:41.192+0800 INFO log/input.go:111 Configured paths: [C:\Users \Downloads\data\Test\*.log]
2018-07-09T17:52:41.202+0800 INFO input/input.go:87 Starting input of type: log; ID: 7110464235258394879
2018-07-09T17:52:41.224+0800 INFO cfgfile/reload.go:122 Config reloader started
2018-07-09T17:52:41.228+0800 INFO crawler/crawler.go:82 Loading and starting Inputs completed. Enabled inputs: 1
2018-07-09T17:52:41.232+0800 INFO log/harvester.go:228 Harvester started for file: C:\Users\ Downloads\data\Test\yoyoyo2.log
2018-07-09T17:52:41.232+0800 INFO log/harvester.go:228 Harvester started for file: C:\Users \Downloads\data\Test\yoyoyo.log
2018-07-09T17:52:41.243+0800 INFO cfgfile/reload.go:214 Loading of config files completed.
2018-07-09T17:52:41.280+0800 DEBUG [publish] pipeline/processor.go:291 Publish event: {
  "@timestamp": "2018-07-09T09:52:41.278Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "doc",
    "version": "6.3.0"
  },
  "offset": 184,
  "message": "WebContainer : 312 - 2018-06-27 11:00:52.443 DEBUG c.i.g.s.s.c.l.PerfLogger:24 - com.ida.gov.sg.spcp.client.gemalto.BaseGemaltoConnector | delete token | Time taken (in milli Sec):10",
  "prospector": {
    "type": "log"
  },
  "input": {
    "type": "log"
  },
  "beat": {
    "name": "CPX-YLR7N6AXN6H",
    "hostname": "CPX-YLR7N6AXN6H",
    "version": "6.3.0"
  },
  "host": {
    "name": "CPX-YLR7N6AXN6H"
  },
  "source": "C:\\Users \\Downloads\\data\\Test\\yoyoyo2.log"
}
2018-07-09T17:53:11.494+0800 INFO [monitoring] log/log.go:124 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":593,"time":{"ms":593}},"total":{"ticks":843,"time":{"ms":843},"value":843},"user":{"ticks":250,"time":{"ms":250}}},"info":{"ephemeral_id":"b8d4ea34-08f1-4554-ad20-b304ff846b56","uptime":{"ms":30529}},"memstats":{"gc_next":5794656,"memory_alloc":2950496,"memory_total":5153512,"rss":23564288}},"filebeat":{"events":{"added":16,"done":16},"harvester":{"open_files":2,"running":2,"started":2}},"libbeat":{"config":{"module":{"running":0},"reloads":1},"output":{"events":{"acked":14,"batches":1,"total":14},"read":{"bytes":6},"type":"logstash","write":{"bytes":1025}},"pipeline":{"clients":1,"events":{"active":0,"filtered":2,"published":14,"retry":14,"total":16},"queue":{"acked":14}}},"registrar":{"states":{"current":2,"update":16},"writes":4},"system":{"cpu":{"cores":4}}}}}
2018-07-09T17:53:41.108+0800 INFO [monitoring] log/log.go:124 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":625,"time":{"ms":32}},"total":{"ticks":890,"time":{"ms":47},"value":890},"user":{"ticks":265,"time":{"ms":15}}},"info":{"ephemeral_id":"b8d4ea34-08f1-4554-ad20-b304ff846b56","uptime":{"ms":60531}},"memstats":{"gc_next":5794656,"memory_alloc":3069128,"memory_total":5272144,"rss":2207744}},"filebeat":{"harvester":{"open_files":2,"running":2}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":1,"events":{"active":0}}},"registrar":{"states":{"current":2}}}}}
```

---

<div class="post-metadata">

**Author:** ![bobbytan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bobbytan/32/32720_2.png) [@bobbytan](https://discuss.elastic.co/u/bobbytan)\
**Post date:** [July 12, 2018, 9:15am UTC](https://discuss.elastic.co/t/logstash-not-receiving-beats-input/138652/8 "2018-07-12T09:15:44Z")

</div>

Deleted old registry files in the filebeats data folder each time before running filebeats,this seems to solve the issue.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [August 9, 2018, 9:15am UTC](https://discuss.elastic.co/t/logstash-not-receiving-beats-input/138652/9 "2018-08-09T09:15:48Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.