Difference between "filebeat setup" and "filebeat --setup"

Elastic Stack Beats
1

According to the docs, filebeat setup is the same as filebeat --setup except that filebeat --setup keeps running while filebeat setup runs the setup and exits.

filebeat setup

Sets up the initial environment, including the index template, Kibana dashboards (when available), and machine learning jobs (when available).

filebeat --setup

Loads the sample Kibana dashboards. If you want to load the dashboards without running Filebeat, use the setup command instead.

However, it seems the ingest nodes are only loaded when running filebeat --setup. This will not load the pipelines as seen by "GET /_ingest/pipeline/" in Kibana's console.

filebeat setup -e -v \
-E 'setup.template.overwrite=true' \
-E 'setup.kibana.host="kibana.example.com:5601"' \
-E 'output.logstash.enabled=false' \
-E 'output.elasticsearch.hosts=["elasticsearch.example.com:9200"]'

2018/01/27 13:58:14.232586 beat.go:436: INFO Home path: [/usr/share/filebeat] Config path: [/etc/filebeat] Data path: [/var/lib/filebeat] Logs path: [/var/log/filebeat]
2018/01/27 13:58:14.232661 metrics.go:23: INFO Metrics logging every 30s
2018/01/27 13:58:14.232675 beat.go:443: INFO Beat UUID: aff84948-7a36-4d9d-8418-50e711c18258
2018/01/27 13:58:14.232691 beat.go:203: INFO Setup Beat: filebeat; Version: 6.1.2
2018/01/27 13:58:14.232867 client.go:123: INFO Elasticsearch url: http://elasticsearch.example.com:9200
2018/01/27 13:58:14.233130 module.go:76: INFO Beat name: LXD1
2018/01/27 13:58:14.233281 client.go:123: INFO Elasticsearch url: http://elasticsearch.example.com:9200
2018/01/27 13:58:14.235110 client.go:651: INFO Connected to Elasticsearch version 6.1.2
2018/01/27 13:58:14.236008 load.go:55: INFO Loading template for Elasticsearch version: 6.1.2
2018/01/27 13:58:14.236019 load.go:58: INFO Existing template will be overwritten, as overwrite is enabled.
2018/01/27 13:58:14.261007 load.go:89: INFO Elasticsearch template with name 'filebeat-6.1.2' loaded
Loaded index template
2018/01/27 13:58:14.261136 client.go:123: INFO Elasticsearch url: http://elasticsearch.example.com:9200
2018/01/27 13:58:14.262468 client.go:651: INFO Connected to Elasticsearch version 6.1.2
2018/01/27 13:58:14.262514 client.go:69: INFO Kibana url: http://kibana.example.com:5601
2018/01/27 13:58:38.307364 beat.go:551: INFO Kibana dashboards successfully loaded.
Loaded dashboards
2018/01/27 13:58:38.307508 client.go:123: INFO Elasticsearch url: http://elasticsearch.example.com:9200
2018/01/27 13:58:38.308961 client.go:651: INFO Connected to Elasticsearch version 6.1.2
2018/01/27 13:58:38.309459 modules.go:459: WARN Xpack Machine Learning is not enabled
2018/01/27 13:58:38.310399 modules.go:459: WARN Xpack Machine Learning is not enabled
2018/01/27 13:58:38.311140 modules.go:459: WARN Xpack Machine Learning is not enabled
Loaded machine learning job configurations

This command will load the pipelines as seen by "Elasticsearch pipeline with ID ... loaded" in the output.

filebeat --setup -e -v \
-E 'setup.template.overwrite=true' \
-E 'setup.kibana.host="kibana.example.com:5601"' \
-E 'output.logstash.enabled=false' \
-E 'output.elasticsearch.hosts=["elasticsearch.example.com:9200"]'

... (output same as above)
2018/01/27 13:59:29.463309 crawler.go:82: INFO Loading and starting Prospectors completed. Enabled prospectors: 1
2018/01/27 13:59:29.463345 reload.go:127: INFO Config reloader started
2018/01/27 13:59:29.485863 reload.go:258: INFO Starting 2 runners ...
2018/01/27 13:59:29.486034 client.go:123: INFO Elasticsearch url: http://elasticsearch.example.com:9200
2018/01/27 13:59:29.487848 client.go:651: INFO Connected to Elasticsearch version 6.1.2
2018/01/27 13:59:29.502610 modules.go:384: INFO Elasticsearch pipeline with ID 'filebeat-6.1.2-auditd-log-pipeline' loaded
2018/01/27 13:59:29.502630 prospector.go:87: INFO Starting prospector of type: log; ID: 9601487749450928370
2018/01/27 13:59:29.502712 client.go:123: INFO Elasticsearch url: http://elasticsearch.example.com:9200
2018/01/27 13:59:29.504498 client.go:651: INFO Connected to Elasticsearch version 6.1.2
2018/01/27 13:59:29.530594 modules.go:384: INFO Elasticsearch pipeline with ID 'filebeat-6.1.2-system-auth-pipeline' loaded
2018/01/27 13:59:29.550989 modules.go:384: INFO Elasticsearch pipeline with ID 'filebeat-6.1.2-system-syslog-pipeline' loaded
2018/01/27 13:59:29.551011 prospector.go:87: INFO Starting prospector of type: log; ID: 18044925928469627648
2018/01/27 13:59:29.551018 prospector.go:87: INFO Starting prospector of type: log; ID: 8734454681295277600
2018/01/27 13:59:29.551036 reload.go:219: INFO Loading of config files completed.
2018/01/27 13:59:30.434150 client.go:651: INFO Connected to Elasticsearch version 6.1.2
2018/01/27 13:59:30.435186 load.go:55: INFO Loading template for Elasticsearch version: 6.1.2
2018/01/27 13:59:30.435198 load.go:58: INFO Existing template will be overwritten, as overwrite is enabled.
2018/01/27 13:59:30.460249 load.go:89: INFO Elasticsearch template with name 'filebeat-6.1.2' loaded

IHMO pipelines are part of the setup. Why are they only included when actually running filebeat?

2

Hi,

As the documentation you pasted here says, setup only loads

  • index template
  • Kibana dashboards
  • ML jobs

Ingest pipelines are only loaded after everything is set up and messages are sent to ES. As --setup continues running after setting up dashboards, index templates and possible ML jobs, messages are started to be sent and the pipeline is loaded.

It is the way it supposed to work. But I do understand that loading Ingest pipelines after actual messages are sent is counterintuitive.

3

What happens when you add Logstash to the equation? The general consensus seems to be that ingest nodes should not be used with Logstash.

  1. Setup - Need to run filebeat connected directly to ES to load the ingest nodes, then kill filebeat and start it connected to Logstash.
  2. Configuration - The pipeline can be specified in the elasticsearch output but it gets complicated real fast when managing multiple beats.
  3. Maintaining - The ingest pipeline might need to be updated every time filebeat is updated. This means possibly running filebeat to connect directly to ES after each update, as well as updating the Logstash pipeline.
  4. Support - Are the Logstash configuration examples supported? What about other log formats (modules) that filebeat supports?
  5. Filebeat modules - The filebeat modules break when filebeat is connected to Logstash. This is primarily because the Logstash doesn't run the ingest node to process the data.

It seems logical that the ingest nodes are considered part of the filebeat processing, not setup, and if you want to use Logstash, you need to convert the ingest node to a Logstash pipeline or use the configuration examples. The documentation doesn't outright state this, and when searching the answers generally point towards Logstash pipelines (using files) OR filebeat ingest nodes in Elasticsearch.

4

The problem you are describing is a real problem; we could do a better job of documenting this transition. Currently, we don't have a better story other than the links you have provided.

Would you mind creating an issue on both projects repositories to help move it forward?

Concerning the Logstash pipeline and the ingest pipeline, we have some discussion in beats to create a shared definition that will allow us to target both software. Both products can do similar things, but they diverge in the how.

5

Would you mind creating an issue on both projects repositories to help move it forward?

Sure! It may take a day to two since this isn't a high priority.

this transition

Is the future ingest nodes? I.e. transitioning from Logstash pipelines to ingest nodes?

6

@NiceGuyIT Logstash and Ingest will live together, ingest wont replace logstash , they just target different needs

7

Thank you for the information Pier. It's greatly appreciated to have a candid reply.

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.