# Filebeat to Logstash to ES

**URL:** <https://discuss.elastic.co/t/filebeat-to-logstash-to-es/247767>\
**Category:** Logstash\
**Created:** [September 7, 2020, 11:39am UTC](https://discuss.elastic.co/t/filebeat-to-logstash-to-es/247767 "2020-09-07T11:39:05Z")\
**Posts on this page:** 17\
**Page:** 1

<div class="post-metadata">

**Author:** ![slash665](https://avatars.discourse-cdn.com/v4/letter/s/c2a13f/32.png) [@slash665](https://discuss.elastic.co/u/slash665)\
**Post date:** [September 7, 2020, 11:39am UTC](https://discuss.elastic.co/t/filebeat-to-logstash-to-es/247767/1 "2020-09-07T11:39:05Z")

</div>

Hello,

I am trying to set up a pipeline from Filebeat to Logstash to ES, following the guide from Elastic with the tutorial logs they provide, and for the love of me I just can't get it to work.

This is what I've got in my filebeat.yml:

```
 # Paths that should be crawled and fetched. Glob based paths.
  paths:
    # - /var/log/*.log
    # - c:\programdata\elasticsearch\logs\*
   - /home/admin/Downloads/logstash-tutorial.log

#output.logstash:
      # The Logstash hosts
      hosts: ["localhost:5044"]

```

And this is my logstash conf file:

```
##Input section
input {
        beats {
                port => "5044"
        }
}

##Filter section
filter {
        grok {
                match => { "message" => "%{COMBINEDAPACHELOG}" }
        }
        date {
                match => ["timestamp" , "dd/MMM/yyyy:HH:mm:ss Z"]
        }
}

## Output section
output {
        elasticsearch { hosts => ["localhost:9200"] }
        stdout { codec => rubydebug }
}

```

Testing the logstash conf file comes back as OK, but this just doesn't work, and I have no idea why not. Filebeat, Logstash and ES are all running on the same system. Is there anything else I need to configure to make it work?

Thanks for any advice!

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [September 7, 2020, 1:48pm UTC](https://discuss.elastic.co/t/filebeat-to-logstash-to-es/247767/2 "2020-09-07T13:48:34Z")

</div>

> [@slash665](#):
>
> ```auto
> #output.logstash:
> # The Logstash hosts
> hosts: ["localhost:5044"]
> 
> ```

If this is how it looks in your file, your `logstash` output is commented, you need to uncomment it to enable.

Also, what do you have in Filebeat and Logstash logs? Any errors?

---

<div class="post-metadata">

**Author:** ![slash665](https://avatars.discourse-cdn.com/v4/letter/s/c2a13f/32.png) [@slash665](https://discuss.elastic.co/u/slash665)\
**Post date:** [September 7, 2020, 2:00pm UTC](https://discuss.elastic.co/t/filebeat-to-logstash-to-es/247767/3 "2020-09-07T14:00:18Z")

</div>

Thank you, that was definitely part of it! I uncommented that, and commented in output.elasticsearch, so that part should be fine now.

It's still not working however, and there are no errors in the logstash logs:

```
[2020-09-07T15:48:10,452][INFO][logstash.outputs.elasticsearch][main] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//localhost:9200"]}
[2020-09-07T15:48:10,500][INFO][logstash.outputs.elasticsearch][main] Using a default mapping template {:es_version=>7, :ecs_compatibility=>:disabled}
[2020-09-07T15:48:10,619][INFO][logstash.outputs.elasticsearch][main] Attempting to install template {:manage_template=>{"index_patterns"=>"logstash-*", "version"=>60001, "settings"=>{"index.refresh_interval"=>"5s", "number_of_shards"=>1, "index.lifecycle.name"=>"logstash-policy", "index.lifecycle.rollover_alias"=>"logstash"}, "mappings"=>{"dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"*", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date"}, "@version"=>{"type"=>"keyword"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}
[2020-09-07T15:48:10,806][INFO][logstash.javapipeline][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>500, "pipeline.sources"=>["/etc/logstash/conf.d/logstash.conf"], :thread=>"#<Thread:0x2ec7349c run>"}
[2020-09-07T15:48:11,976][INFO][logstash.javapipeline][main] Pipeline Java execution initialization time {"seconds"=>1.16}
[2020-09-07T15:48:12,013][INFO][logstash.inputs.beats][main] Beats inputs: Starting input listener {:address=>"0.0.0.0:5044"}
[2020-09-07T15:48:12,037][INFO][logstash.javapipeline][main] Pipeline started {"pipeline.id"=>"main"}
[2020-09-07T15:48:12,196][INFO][logstash.agent] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2020-09-07T15:48:12,335][INFO][org.logstash.beats.Server][main][3e42ac5b0e5ab0e7354421a4d00dba9353a94c8ecf339bec7e2ef6c635472daf] Starting server on port: 5044
[2020-09-07T15:48:12,578][INFO][logstash.agent] Successfully started Logstash API endpoint {:port=>9600}

```

This is all there is in the filebeat logs:

```
|2020-09-07T13:55:39.063+0200|INFO|instance/beat.go:640|Home path: [/usr/share/filebeat] Config path: [/etc/filebeat] Data path: [/var/lib/filebeat] Logs path: [/var/log/filebeat]|
|---|---|---|---|
|2020-09-07T13:55:39.063+0200|INFO|instance/beat.go:648|Beat ID: fd69b29a-f42c-4cc3-b176-864c43325556|

```

Unless I'm looking in the wrong place? (/var/log/filebeat/filebeat)

---

<div class="post-metadata">

**Author:** ![slash665](https://avatars.discourse-cdn.com/v4/letter/s/c2a13f/32.png) [@slash665](https://discuss.elastic.co/u/slash665)\
**Post date:** [September 7, 2020, 2:14pm UTC](https://discuss.elastic.co/t/filebeat-to-logstash-to-es/247767/4 "2020-09-07T14:14:30Z")

</div>

i did set the logging level to debug in the filebeat.yml. No change in the logs, though.

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [September 7, 2020, 2:17pm UTC](https://discuss.elastic.co/t/filebeat-to-logstash-to-es/247767/5 "2020-09-07T14:17:23Z")

</div>

How is the rest of your `filebeat.yml`? Share the file here, you can remove the commented lines.

Make sure that your input is enabled, in the reference file the `filebeat.inputs` example is disabled by default (`enabled: false`)

Your config should look something like this.

```auto
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /home/admin/Downloads/logstash-tutorial.log

```

Also, how are you running Filebeat? The user running Filebeat needs to have read write to the file in your `path`.

---

<div class="post-metadata">

**Author:** ![slash665](https://avatars.discourse-cdn.com/v4/letter/s/c2a13f/32.png) [@slash665](https://discuss.elastic.co/u/slash665)\
**Post date:** [September 7, 2020, 2:49pm UTC](https://discuss.elastic.co/t/filebeat-to-logstash-to-es/247767/6 "2020-09-07T14:49:29Z")

</div>

This is what I've got after removing everything that's commented out ( I left the section titles though) :

```
# ============================== Filebeat inputs ===============================

filebeat.inputs:

- type: log

  enabled: true
  paths:
   - /home/admin/Downloads/logstash-tutorial.log

# ============================== Filebeat modules ==============================

filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false

# ======================= Elasticsearch template setting =======================

setup.template.settings:
  index.number_of_shards: 1

  # ================================== General ===================================

# ================================= Dashboards =================================

# =================================== Kibana ===================================

setup.kibana:

# =============================== Elastic Cloud ================================

# ================================== Outputs ===================================

# ---------------------------- Elasticsearch Output ----------------------------

# ------------------------------ Logstash Output -------------------------------
output.logstash:
 
  hosts: ["localhost:5044"]

# ================================= Processors =================================
processors:
  - add_host_metadata:
      when.not.contains.tags: forwarded
  - add_cloud_metadata: ~
  - add_docker_metadata: ~
  - add_kubernetes_metadata: ~

# ================================== Logging ===================================

logging.level: debug
logging.selectors: ["*"]

# ============================= X-Pack Monitoring ==============================

# ============================== Instrumentation ===============================

# ================================= Migration ==================================

```

I started Filebeat as root, if that's what you mean, and I set rw globally on the file.

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [September 7, 2020, 3:01pm UTC](https://discuss.elastic.co/t/filebeat-to-logstash-to-es/247767/7 "2020-09-07T15:01:37Z")

</div>

It could be that filebeat already read the file, but your output wasn't configured before and now it will only read new lines.

Stop filebeat and remove the `registry` file, located in `/var/lib/filebeat/registry`, then start filebeat again to see if the logs are working.

---

<div class="post-metadata">

**Author:** ![slash665](https://avatars.discourse-cdn.com/v4/letter/s/c2a13f/32.png) [@slash665](https://discuss.elastic.co/u/slash665)\
**Post date:** [September 8, 2020, 9:25am UTC](https://discuss.elastic.co/t/filebeat-to-logstash-to-es/247767/8 "2020-09-08T09:25:35Z")

</div>

i just tried that, but it didnt work ☹

---

<div class="post-metadata">

**Author:** ![d.silwon](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/d.silwon/32/65853_2.png) [@d.silwon](https://discuss.elastic.co/u/d.silwon)\
**Post date:** [September 8, 2020, 11:32am UTC](https://discuss.elastic.co/t/filebeat-to-logstash-to-es/247767/9 "2020-09-08T11:32:37Z")

</div>

Please try change logging section in your config file to this and restart service:

```auto
logging.level: debug
logging.to_files: true
logging.files:
  path: /var/log/filebeat
  name: filebeat
  keepfiles: 7
  permissions: 0644

```

and check log file after service restarting. Thanks

---

<div class="post-metadata">

**Author:** ![slash665](https://avatars.discourse-cdn.com/v4/letter/s/c2a13f/32.png) [@slash665](https://discuss.elastic.co/u/slash665)\
**Post date:** [September 8, 2020, 1:39pm UTC](https://discuss.elastic.co/t/filebeat-to-logstash-to-es/247767/10 "2020-09-08T13:39:15Z")

</div>

That definitely helped to get more information in the logs, but I don't see any errors. What I do see are scans of files that are no longer configured as inputs in the filebeat.yml. (at one point they were, i've been unsuccesfully trying to get this going with several different log files...)

This seems strange to me, why would it keep scanning files that are no longer configured?

---

<div class="post-metadata">

**Author:** ![slash665](https://avatars.discourse-cdn.com/v4/letter/s/c2a13f/32.png) [@slash665](https://discuss.elastic.co/u/slash665)\
**Post date:** [September 8, 2020, 1:45pm UTC](https://discuss.elastic.co/t/filebeat-to-logstash-to-es/247767/11 "2020-09-08T13:45:06Z")

</div>

Oh damn, almost overlooked this:

`2020-09-08T15:35:49.852+0200	ERROR	[publisher_pipeline_output]	pipeline/output.go:154	Failed to connect to backoff(async(tcp://localhost:5044)): dial tcp [::1]:5044: connect: connection refused`

I think this might be a firewall issue, and if so I'll feel pretty dumb..

---

<div class="post-metadata">

**Author:** ![slash665](https://avatars.discourse-cdn.com/v4/letter/s/c2a13f/32.png) [@slash665](https://discuss.elastic.co/u/slash665)\
**Post date:** [September 8, 2020, 2:04pm UTC](https://discuss.elastic.co/t/filebeat-to-logstash-to-es/247767/12 "2020-09-08T14:04:51Z")

</div>

I don't think it's a firewall issue..

---

<div class="post-metadata">

**Author:** ![d.silwon](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/d.silwon/32/65853_2.png) [@d.silwon](https://discuss.elastic.co/u/d.silwon)\
**Post date:** [September 8, 2020, 2:09pm UTC](https://discuss.elastic.co/t/filebeat-to-logstash-to-es/247767/13 "2020-09-08T14:09:18Z")

</div>

Please show us the result of this command:

```auto
firewall-cmd --list-port

```

---

<div class="post-metadata">

**Author:** ![d.silwon](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/d.silwon/32/65853_2.png) [@d.silwon](https://discuss.elastic.co/u/d.silwon)\
**Post date:** [September 8, 2020, 2:11pm UTC](https://discuss.elastic.co/t/filebeat-to-logstash-to-es/247767/14 "2020-09-08T14:11:28Z")

</div>

and result of this command:

```auto
systemctl status logstash -l

```

and

```auto
journalctl -u logstash -xe

```

Thanks

---

<div class="post-metadata">

**Author:** ![slash665](https://avatars.discourse-cdn.com/v4/letter/s/c2a13f/32.png) [@slash665](https://discuss.elastic.co/u/slash665)\
**Post date:** [September 8, 2020, 2:47pm UTC](https://discuss.elastic.co/t/filebeat-to-logstash-to-es/247767/15 "2020-09-08T14:47:13Z")

</div>

I got it working! Turns out port forwarding needed to be turned on on the virtualbox side, since the vm was NATed..

Thanks for the tip for the debugging level, that actually got me the visibility I needed to figure this out!

What's still weird is that filebeat is still scanning files that are no longer configured as inputs in filebeat.yml, though. Probably best to open a new topic for that..

Thank you to everyone who tried to help, I learned a lot!

---

<div class="post-metadata">

**Author:** ![d.silwon](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/d.silwon/32/65853_2.png) [@d.silwon](https://discuss.elastic.co/u/d.silwon)\
**Post date:** [September 8, 2020, 3:22pm UTC](https://discuss.elastic.co/t/filebeat-to-logstash-to-es/247767/16 "2020-09-08T15:22:15Z")

</div>

Good job 🙂

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 6, 2020, 3:22pm UTC](https://discuss.elastic.co/t/filebeat-to-logstash-to-es/247767/17 "2020-10-06T15:22:24Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.