Elasticsearch ingest pipelines not configured

A_B · March 24, 2020, 3:47pm

Hello all,

not entirely sure if this should be classified as Filebeat, Elasticsearch or Kibana but as I'm executing a Filebeat command, I will start here.

I'm trying to setup Elasticsearch ingest pipelines per this documentation

I have ran this

# /usr/share/filebeat/bin/filebeat -c /tmp/filebeat.yml setup --pipelines --modules nginx,haproxy
Loaded Ingest pipelines

As I usually ship logs via Logstash I created /tmp/filebeat.yml with a Elasticsearch output.

But when I check ingest pipelines in Kibana/Elasticsearch, I do not see any new pipelines

GET /_ingest/pipeline
apm-this
xpack-that

Should I see those pipelines? How can I confirm they are configured?

All tips and suggestions are welcome

shaunak · March 24, 2020, 3:54pm

Hi @A_B,

A few questions to get started:

What version of Filebeat are you using?
Could you please post the contents of /tmp/filebeat.yml here (with any sensitive information redacted)?
Could you re-run the Filebeat command with -d"*"? This will print out debug level logs. Could you please post them here (again, with any sensitive information redacted)?

Thanks,

Shaunak

A_B · March 24, 2020, 4:20pm

Hi @shaunak,

Sure, no problem

All Elastic components are version 7.5.2

Content of /tmp/filebeat.yml

---
shutdown_timeout: '0'
name: my_host.example.com
tags: []
fields: {}
fields_under_root: false
filebeat:
  config.inputs:
    enabled: true
    path: "/etc/filebeat/conf.d/*.yml"
  config.modules:
    enabled: true
    path: "/etc/filebeat/puppet_managed_modules/*.yml"
  shutdown_timeout: '0'
  modules: []
http: {}
output:
  elasticsearch:
    hosts:
    - 10.10.10.10:9200
shipper: {}
logging: {}
runoptions: {}
processors: []
setup: {}

here you go

# /usr/share/filebeat/bin/filebeat -e -c /tmp/filebeat.yml -d "*" setup --pipelines --modules nginx,haproxy
2020-03-24T16:15:10.270Z	INFO	instance/beat.go:610	Home path: [/usr/share/filebeat/bin] Config path: [/usr/share/filebeat/bin] Data path: [/usr/share/filebeat/bin/data] Logs path: [/usr/share/filebeat/bin/logs]
2020-03-24T16:15:10.270Z	DEBUG	[beat]	instance/beat.go:662	Beat metadata path: /usr/share/filebeat/bin/data/meta.json
2020-03-24T16:15:10.271Z	INFO	instance/beat.go:618	Beat ID: a41d7d9d-4d0f-4fe5-869f-1705f9929bf6
2020-03-24T16:15:10.271Z	INFO	[beat]	instance/beat.go:941	Beat info	{"system_info": {"beat": {"path": {"config": "/usr/share/filebeat/bin", "data": "/usr/share/filebeat/bin/data", "home": "/usr/share/filebeat/bin", "logs": "/usr/share/filebeat/bin/logs"}, "type": "filebeat", "uuid": "a41d7d9d-4d0f-4fe5-869f-1705f9929bf6"}}}
2020-03-24T16:15:10.272Z	INFO	[beat]	instance/beat.go:950	Build info	{"system_info": {"build": {"commit": "a9c141434cd6b25d7a74a9c770be6b70643dc767", "libbeat": "7.5.2", "time": "2020-01-15T11:13:22.000Z", "version": "7.5.2"}}}
2020-03-24T16:15:10.273Z	INFO	[beat]	instance/beat.go:953	Go runtime info	{"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":4,"version":"go1.12.12"}}}
2020-03-24T16:15:10.274Z	INFO	[beat]	instance/beat.go:957	Host info	{"system_info": {"host": {"architecture":"x86_64","boot_time":"2020-01-29T11:56:35Z","containerized":false,"name":"metrics-proxy2","ip":["127.0.0.1/8","::1/128","172.16.0.52/24","fe80::f816:3eff:fe40:632d/64"],"kernel_version":"4.19.0-6-cloud-amd64","mac":["fa:16:3e:40:63:2d"],"os":{"family":"debian","platform":"debian","name":"Debian GNU/Linux","version":"10 (buster)","major":10,"minor":0,"patch":0,"codename":"buster"},"timezone":"UTC","timezone_offset_sec":0,"id":"8a527801253942fe9150a0449a82dd42"}}}
2020-03-24T16:15:10.275Z	INFO	[beat]	instance/beat.go:986	Process info	{"system_info": {"process": {"capabilities": {"inheritable":null,"permitted":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read"],"effective":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read"],"bounding":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read"],"ambient":null}, "cwd": "/etc/filebeat", "exe": "/usr/share/filebeat/bin/filebeat", "name": "filebeat", "pid": 170146, "ppid": 148959, "seccomp": {"mode":"disabled","no_new_privs":false}, "start_time": "2020-03-24T16:15:10.070Z"}}}
2020-03-24T16:15:10.275Z	INFO	instance/beat.go:297	Setup Beat: filebeat; Version: 7.5.2
2020-03-24T16:15:10.276Z	DEBUG	[beat]	instance/beat.go:323	Initializing output plugins
2020-03-24T16:15:10.276Z	INFO	[index-management]	idxmgmt/std.go:182	Set output.elasticsearch.index to 'filebeat-7.5.2' as ILM is enabled.
2020-03-24T16:15:10.277Z	INFO	elasticsearch/client.go:171	Elasticsearch url: http://10.10.10.10:9200
2020-03-24T16:15:10.278Z	DEBUG	[publisher]	pipeline/consumer.go:137	start pipeline event consumer
2020-03-24T16:15:10.278Z	INFO	[publisher]	pipeline/module.go:97	Beat name: my_host.example.com
2020-03-24T16:15:10.278Z	ERROR	fileset/modules.go:125	Not loading modules. Module directory not found: /usr/share/filebeat/bin/module
2020-03-24T16:15:10.279Z	INFO	elasticsearch/client.go:171	Elasticsearch url: http://10.10.10.10:9200
2020-03-24T16:15:10.279Z	DEBUG	[elasticsearch]	elasticsearch/client.go:729	ES Ping(url=http://10.10.10.10:9200)
2020-03-24T16:15:10.283Z	DEBUG	[elasticsearch]	elasticsearch/client.go:752	Ping status code: 200
2020-03-24T16:15:10.283Z	INFO	elasticsearch/client.go:753	Attempting to connect to Elasticsearch version 7.5.2
2020-03-24T16:15:10.283Z	DEBUG	[elasticsearch]	elasticsearch/client.go:771	GET http://10.10.10.10:9200/_xpack?human=false  <nil>
2020-03-24T16:15:10.332Z	DEBUG	[cfgfile]	cfgfile/reload.go:248	Scan for config files
2020-03-24T16:15:10.333Z	DEBUG	[cfgfile]	cfgfile/cfgfile.go:193	Load config from file: /etc/filebeat/puppet_managed_modules/haproxy.yml
2020-03-24T16:15:10.333Z	DEBUG	[cfgfile]	cfgfile/reload.go:257	Number of module configs found: 1
2020-03-24T16:15:10.333Z	DEBUG	[load]	cfgfile/list.go:62	Starting reload procedure, current runners: 0
2020-03-24T16:15:10.333Z	DEBUG	[load]	cfgfile/list.go:80	Start list: 1, Stop list: 0
2020-03-24T16:15:10.334Z	ERROR	fileset/modules.go:125	Not loading modules. Module directory not found: /usr/share/filebeat/bin/module
2020-03-24T16:15:10.334Z	DEBUG	[load]	cfgfile/list.go:101	Starting runner:
2020-03-24T16:15:10.334Z	DEBUG	[fileset]	fileset/setup.go:66	Loading ingest pipelines for modules from modules.d
2020-03-24T16:15:10.334Z	INFO	elasticsearch/client.go:171	Elasticsearch url: http://10.10.10.10:9200
2020-03-24T16:15:10.334Z	DEBUG	[elasticsearch]	elasticsearch/client.go:729	ES Ping(url=http://10.10.10.10:9200)
2020-03-24T16:15:10.336Z	DEBUG	[elasticsearch]	elasticsearch/client.go:752	Ping status code: 200
2020-03-24T16:15:10.336Z	INFO	elasticsearch/client.go:753	Attempting to connect to Elasticsearch version 7.5.2
2020-03-24T16:15:10.336Z	DEBUG	[elasticsearch]	elasticsearch/client.go:771	GET http://10.10.10.10:9200/_xpack?human=false  <nil>
2020-03-24T16:15:10.386Z	INFO	cfgfile/reload.go:264	Loading of config files completed.
2020-03-24T16:15:10.386Z	INFO	[load]	cfgfile/list.go:118	Stopping 1 runners ...
2020-03-24T16:15:10.386Z	DEBUG	[load]	cfgfile/list.go:129	Stopping runner:
2020-03-24T16:15:10.386Z	DEBUG	[load]	cfgfile/list.go:131	Stopped runner:
Loaded Ingest pipelines

So, some modules are missing?

A_B · March 25, 2020, 12:23pm

Turns out this was the problem. Running Debian 10 with Elastic Debian repos (from Puppet module elastic_stack)

There was indeed no /usr/share/filebeat/bin/module. There was /usr/share/filebeat/module so I created a symbolic link and now I can setup the ingest pipelines in Elasticsearch.

/usr/share/filebeat/bin # ls -l
total 75112
drwxr-x--- 2 root root     4096 Mar 23 13:33 data
drwx------ 2 root root     4096 Mar 24 16:33 logs
-rwxr-xr-x 1 root root 75842832 Jan 15 11:14 filebeat
-rwxr-xr-x 1 root root  1057147 Jan 15 11:13 filebeat-god
lrwxrwxrwx 1 root root        9 Mar 25 12:15 module -> ../module

Now I see

GET /_ingest/pipeline/filebeat-7.5.2-haproxy-log-pipeline
{
  "filebeat-7.5.2-haproxy-log-pipeline" : {
...
  }
}

system · April 22, 2020, 12:23pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Filebeat is not using custom pipeline mentioned in the output.elasticsearch Beats filebeat , ingest-pipeline	5	507	July 28, 2021
Cannot seem to get ingest pipelines to work, please help! Elasticsearch	12	970	April 6, 2021
Filebeat dashboards are not loaded in kibana---filebeat version 7.10 Beats filebeat	5	636	January 5, 2021
Logstash is not creating ingest pipelines Logstash	3	457	July 13, 2021
Right way to make ingest pipeline work Beats filebeat	1	596	October 16, 2020

Elasticsearch ingest pipelines not configured

Related topics