Filebeat to Logstash to ES

Elastic Stack Logstash
1

Hello,

I am trying to set up a pipeline from Filebeat to Logstash to ES, following the guide from Elastic with the tutorial logs they provide, and for the love of me I just can't get it to work.

This is what I've got in my filebeat.yml:

 # Paths that should be crawled and fetched. Glob based paths.
  paths:
    # - /var/log/*.log
    # - c:\programdata\elasticsearch\logs\*
   - /home/admin/Downloads/logstash-tutorial.log


#output.logstash:
      # The Logstash hosts
      hosts: ["localhost:5044"]

And this is my logstash conf file:

##Input section
input {
        beats {
                port => "5044"
        }
}

##Filter section
filter {
        grok {
                match => { "message" => "%{COMBINEDAPACHELOG}" }
        }
        date {
                match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z"]
        }
}


## Output section
output {
        elasticsearch { hosts => ["localhost:9200"] }
        stdout { codec => rubydebug }
}

Testing the logstash conf file comes back as OK, but this just doesn't work, and I have no idea why not. Filebeat, Logstash and ES are all running on the same system. Is there anything else I need to configure to make it work?

Thanks for any advice!

2

If this is how it looks in your file, your logstash output is commented, you need to uncomment it to enable.

Also, what do you have in Filebeat and Logstash logs? Any errors?

3

Thank you, that was definitely part of it! I uncommented that, and commented in output.elasticsearch, so that part should be fine now.

It's still not working however, and there are no errors in the logstash logs:

[2020-09-07T15:48:10,452][INFO ][logstash.outputs.elasticsearch][main] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//localhost:9200"]}
[2020-09-07T15:48:10,500][INFO ][logstash.outputs.elasticsearch][main] Using a default mapping template {:es_version=>7, :ecs_compatibility=>:disabled}
[2020-09-07T15:48:10,619][INFO ][logstash.outputs.elasticsearch][main] Attempting to install template {:manage_template=>{"index_patterns"=>"logstash-*", "version"=>60001, "settings"=>{"index.refresh_interval"=>"5s", "number_of_shards"=>1, "index.lifecycle.name"=>"logstash-policy", "index.lifecycle.rollover_alias"=>"logstash"}, "mappings"=>{"dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"*", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date"}, "@version"=>{"type"=>"keyword"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}
[2020-09-07T15:48:10,806][INFO ][logstash.javapipeline    ][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>500, "pipeline.sources"=>["/etc/logstash/conf.d/logstash.conf"], :thread=>"#<Thread:0x2ec7349c run>"}
[2020-09-07T15:48:11,976][INFO ][logstash.javapipeline    ][main] Pipeline Java execution initialization time {"seconds"=>1.16}
[2020-09-07T15:48:12,013][INFO ][logstash.inputs.beats    ][main] Beats inputs: Starting input listener {:address=>"0.0.0.0:5044"}
[2020-09-07T15:48:12,037][INFO ][logstash.javapipeline    ][main] Pipeline started {"pipeline.id"=>"main"}
[2020-09-07T15:48:12,196][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2020-09-07T15:48:12,335][INFO ][org.logstash.beats.Server][main][3e42ac5b0e5ab0e7354421a4d00dba9353a94c8ecf339bec7e2ef6c635472daf] Starting server on port: 5044
[2020-09-07T15:48:12,578][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}

This is all there is in the filebeat logs:

|2020-09-07T13:55:39.063+0200|INFO|instance/beat.go:640|Home path: [/usr/share/filebeat] Config path: [/etc/filebeat] Data path: [/var/lib/filebeat] Logs path: [/var/log/filebeat]|
|---|---|---|---|
|2020-09-07T13:55:39.063+0200|INFO|instance/beat.go:648|Beat ID: fd69b29a-f42c-4cc3-b176-864c43325556|

Unless I'm looking in the wrong place? (/var/log/filebeat/filebeat)

4

i did set the logging level to debug in the filebeat.yml. No change in the logs, though.

5

How is the rest of your filebeat.yml? Share the file here, you can remove the commented lines.

Make sure that your input is enabled, in the reference file the filebeat.inputs example is disabled by default (enabled: false)

Your config should look something like this.

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /home/admin/Downloads/logstash-tutorial.log

Also, how are you running Filebeat? The user running Filebeat needs to have read write to the file in your path.

6

This is what I've got after removing everything that's commented out ( I left the section titles though) :

# ============================== Filebeat inputs ===============================

filebeat.inputs:


- type: log

  enabled: true
  paths:
   - /home/admin/Downloads/logstash-tutorial.log

# ============================== Filebeat modules ==============================

filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false

# ======================= Elasticsearch template setting =======================

setup.template.settings:
  index.number_of_shards: 1

  # ================================== General ===================================

# ================================= Dashboards =================================

# =================================== Kibana ===================================

setup.kibana:

# =============================== Elastic Cloud ================================

# ================================== Outputs ===================================

# ---------------------------- Elasticsearch Output ----------------------------

# ------------------------------ Logstash Output -------------------------------
output.logstash:
 
  hosts: ["localhost:5044"]



# ================================= Processors =================================
processors:
  - add_host_metadata:
      when.not.contains.tags: forwarded
  - add_cloud_metadata: ~
  - add_docker_metadata: ~
  - add_kubernetes_metadata: ~

# ================================== Logging ===================================


logging.level: debug
logging.selectors: ["*"]

# ============================= X-Pack Monitoring ==============================

# ============================== Instrumentation ===============================

# ================================= Migration ==================================

I started Filebeat as root, if that's what you mean, and I set rw globally on the file.

7

It could be that filebeat already read the file, but your output wasn't configured before and now it will only read new lines.

Stop filebeat and remove the registry file, located in /var/lib/filebeat/registry, then start filebeat again to see if the logs are working.

8

i just tried that, but it didnt work :frowning:

9

Please try change logging section in your config file to this and restart service:

logging.level: debug
logging.to_files: true
logging.files:
  path: /var/log/filebeat
  name: filebeat
  keepfiles: 7
  permissions: 0644

and check log file after service restarting. Thanks

10

That definitely helped to get more information in the logs, but I don't see any errors. What I do see are scans of files that are no longer configured as inputs in the filebeat.yml. (at one point they were, i've been unsuccesfully trying to get this going with several different log files...)

This seems strange to me, why would it keep scanning files that are no longer configured?

11

Oh damn, almost overlooked this:

2020-09-08T15:35:49.852+0200 ERROR [publisher_pipeline_output] pipeline/output.go:154 Failed to connect to backoff(async(tcp://localhost:5044)): dial tcp [::1]:5044: connect: connection refused

I think this might be a firewall issue, and if so I'll feel pretty dumb..

12

I don't think it's a firewall issue..

13

Please show us the result of this command:

firewall-cmd --list-port
14

and result of this command:

systemctl status logstash -l

and

journalctl -u logstash -xe

Thanks

15

I got it working! Turns out port forwarding needed to be turned on on the virtualbox side, since the vm was NATed..

Thanks for the tip for the debugging level, that actually got me the visibility I needed to figure this out!

What's still weird is that filebeat is still scanning files that are no longer configured as inputs in filebeat.yml, though. Probably best to open a new topic for that..

Thank you to everyone who tried to help, I learned a lot!

16

Good job :slight_smile:

17

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.