Elasticsearch ingest pipelines not configured

A_B · March 24, 2020, 3:47pm

Hello all,

not entirely sure if this should be classified as Filebeat, Elasticsearch or Kibana but as I'm executing a Filebeat command, I will start here.

I'm trying to setup Elasticsearch ingest pipelines per this documentation

I have ran this

# /usr/share/filebeat/bin/filebeat -c /tmp/filebeat.yml setup --pipelines --modules nginx,haproxy
Loaded Ingest pipelines

As I usually ship logs via Logstash I created /tmp/filebeat.yml with a Elasticsearch output.

But when I check ingest pipelines in Kibana/Elasticsearch, I do not see any new pipelines

GET /_ingest/pipeline
apm-this
xpack-that

Should I see those pipelines? How can I confirm they are configured?

All tips and suggestions are welcome

shaunak · March 24, 2020, 3:54pm

Hi @A_B,

A few questions to get started:

What version of Filebeat are you using?
Could you please post the contents of /tmp/filebeat.yml here (with any sensitive information redacted)?
Could you re-run the Filebeat command with -d"*"? This will print out debug level logs. Could you please post them here (again, with any sensitive information redacted)?

Thanks,

Shaunak

A_B · March 24, 2020, 4:20pm

Hi @shaunak,

Sure, no problem

All Elastic components are version 7.5.2

Content of /tmp/filebeat.yml

---
shutdown_timeout: '0'
name: my_host.example.com
tags: []
fields: {}
fields_under_root: false
filebeat:
  config.inputs:
    enabled: true
    path: "/etc/filebeat/conf.d/*.yml"
  config.modules:
    enabled: true
    path: "/etc/filebeat/puppet_managed_modules/*.yml"
  shutdown_timeout: '0'
  modules: []
http: {}
output:
  elasticsearch:
    hosts:
    - 10.10.10.10:9200
shipper: {}
logging: {}
runoptions: {}
processors: []
setup: {}

here you go

# /usr/share/filebeat/bin/filebeat -e -c /tmp/filebeat.yml -d "*" setup --pipelines --modules nginx,haproxy
2020-03-24T16:15:10.270Z	INFO	instance/beat.go:610	Home path: [/usr/share/filebeat/bin] Config path: [/usr/share/filebeat/bin] Data path: [/usr/share/filebeat/bin/data] Logs path: [/usr/share/filebeat/bin/logs]
2020-03-24T16:15:10.270Z	DEBUG	[beat]	instance/beat.go:662	Beat metadata path: /usr/share/filebeat/bin/data/meta.json
2020-03-24T16:15:10.271Z	INFO	instance/beat.go:618	Beat ID: a41d7d9d-4d0f-4fe5-869f-1705f9929bf6
2020-03-24T16:15:10.271Z	INFO	[beat]	instance/beat.go:941	Beat info	{"system_info": {"beat": {"path": {"config": "/usr/share/filebeat/bin", "data": "/usr/share/filebeat/bin/data", "home": "/usr/share/filebeat/bin", "logs": "/usr/share/filebeat/bin/logs"}, "type": "filebeat", "uuid": "a41d7d9d-4d0f-4fe5-869f-1705f9929bf6"}}}
2020-03-24T16:15:10.272Z	INFO	[beat]	instance/beat.go:950	Build info	{"system_info": {"build": {"commit": "a9c141434cd6b25d7a74a9c770be6b70643dc767", "libbeat": "7.5.2", "time": "2020-01-15T11:13:22.000Z", "version": "7.5.2"}}}
2020-03-24T16:15:10.273Z	INFO	[beat]	instance/beat.go:953	Go runtime info	{"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":4,"version":"go1.12.12"}}}
2020-03-24T16:15:10.274Z	INFO	[beat]	instance/beat.go:957	Host info	{"system_info": {"host": {"architecture":"x86_64","boot_time":"2020-01-29T11:56:35Z","containerized":false,"name":"metrics-proxy2","ip":["127.0.0.1/8","::1/128","172.16.0.52/24","fe80::f816:3eff:fe40:632d/64"],"kernel_version":"4.19.0-6-cloud-amd64","mac":["fa:16:3e:40:63:2d"],"os":{"family":"debian","platform":"debian","name":"Debian GNU/Linux","version":"10 (buster)","major":10,"minor":0,"patch":0,"codename":"buster"},"timezone":"UTC","timezone_offset_sec":0,"id":"8a527801253942fe9150a0449a82dd42"}}}
2020-03-24T16:15:10.275Z	INFO	[beat]	instance/beat.go:986	Process info	{"system_info": {"process": {"capabilities": {"inheritable":null,"permitted":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read"],"effective":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read"],"bounding":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read"],"ambient":null}, "cwd": "/etc/filebeat", "exe": "/usr/share/filebeat/bin/filebeat", "name": "filebeat", "pid": 170146, "ppid": 148959, "seccomp": {"mode":"disabled","no_new_privs":false}, "start_time": "2020-03-24T16:15:10.070Z"}}}
2020-03-24T16:15:10.275Z	INFO	instance/beat.go:297	Setup Beat: filebeat; Version: 7.5.2
2020-03-24T16:15:10.276Z	DEBUG	[beat]	instance/beat.go:323	Initializing output plugins
2020-03-24T16:15:10.276Z	INFO	[index-management]	idxmgmt/std.go:182	Set output.elasticsearch.index to 'filebeat-7.5.2' as ILM is enabled.
2020-03-24T16:15:10.277Z	INFO	elasticsearch/client.go:171	Elasticsearch url: http://10.10.10.10:9200
2020-03-24T16:15:10.278Z	DEBUG	[publisher]	pipeline/consumer.go:137	start pipeline event consumer
2020-03-24T16:15:10.278Z	INFO	[publisher]	pipeline/module.go:97	Beat name: my_host.example.com
2020-03-24T16:15:10.278Z	ERROR	fileset/modules.go:125	Not loading modules. Module directory not found: /usr/share/filebeat/bin/module
2020-03-24T16:15:10.279Z	INFO	elasticsearch/client.go:171	Elasticsearch url: http://10.10.10.10:9200
2020-03-24T16:15:10.279Z	DEBUG	[elasticsearch]	elasticsearch/client.go:729	ES Ping(url=http://10.10.10.10:9200)
2020-03-24T16:15:10.283Z	DEBUG	[elasticsearch]	elasticsearch/client.go:752	Ping status code: 200
2020-03-24T16:15:10.283Z	INFO	elasticsearch/client.go:753	Attempting to connect to Elasticsearch version 7.5.2
2020-03-24T16:15:10.283Z	DEBUG	[elasticsearch]	elasticsearch/client.go:771	GET http://10.10.10.10:9200/_xpack?human=false  <nil>
2020-03-24T16:15:10.332Z	DEBUG	[cfgfile]	cfgfile/reload.go:248	Scan for config files
2020-03-24T16:15:10.333Z	DEBUG	[cfgfile]	cfgfile/cfgfile.go:193	Load config from file: /etc/filebeat/puppet_managed_modules/haproxy.yml
2020-03-24T16:15:10.333Z	DEBUG	[cfgfile]	cfgfile/reload.go:257	Number of module configs found: 1
2020-03-24T16:15:10.333Z	DEBUG	[load]	cfgfile/list.go:62	Starting reload procedure, current runners: 0
2020-03-24T16:15:10.333Z	DEBUG	[load]	cfgfile/list.go:80	Start list: 1, Stop list: 0
2020-03-24T16:15:10.334Z	ERROR	fileset/modules.go:125	Not loading modules. Module directory not found: /usr/share/filebeat/bin/module
2020-03-24T16:15:10.334Z	DEBUG	[load]	cfgfile/list.go:101	Starting runner:
2020-03-24T16:15:10.334Z	DEBUG	[fileset]	fileset/setup.go:66	Loading ingest pipelines for modules from modules.d
2020-03-24T16:15:10.334Z	INFO	elasticsearch/client.go:171	Elasticsearch url: http://10.10.10.10:9200
2020-03-24T16:15:10.334Z	DEBUG	[elasticsearch]	elasticsearch/client.go:729	ES Ping(url=http://10.10.10.10:9200)
2020-03-24T16:15:10.336Z	DEBUG	[elasticsearch]	elasticsearch/client.go:752	Ping status code: 200
2020-03-24T16:15:10.336Z	INFO	elasticsearch/client.go:753	Attempting to connect to Elasticsearch version 7.5.2
2020-03-24T16:15:10.336Z	DEBUG	[elasticsearch]	elasticsearch/client.go:771	GET http://10.10.10.10:9200/_xpack?human=false  <nil>
2020-03-24T16:15:10.386Z	INFO	cfgfile/reload.go:264	Loading of config files completed.
2020-03-24T16:15:10.386Z	INFO	[load]	cfgfile/list.go:118	Stopping 1 runners ...
2020-03-24T16:15:10.386Z	DEBUG	[load]	cfgfile/list.go:129	Stopping runner:
2020-03-24T16:15:10.386Z	DEBUG	[load]	cfgfile/list.go:131	Stopped runner:
Loaded Ingest pipelines

So, some modules are missing?

A_B · March 25, 2020, 12:23pm

Turns out this was the problem. Running Debian 10 with Elastic Debian repos (from Puppet module elastic_stack)

There was indeed no /usr/share/filebeat/bin/module. There was /usr/share/filebeat/module so I created a symbolic link and now I can setup the ingest pipelines in Elasticsearch.

/usr/share/filebeat/bin # ls -l
total 75112
drwxr-x--- 2 root root     4096 Mar 23 13:33 data
drwx------ 2 root root     4096 Mar 24 16:33 logs
-rwxr-xr-x 1 root root 75842832 Jan 15 11:14 filebeat
-rwxr-xr-x 1 root root  1057147 Jan 15 11:13 filebeat-god
lrwxrwxrwx 1 root root        9 Mar 25 12:15 module -> ../module

Now I see

GET /_ingest/pipeline/filebeat-7.5.2-haproxy-log-pipeline
{
  "filebeat-7.5.2-haproxy-log-pipeline" : {
...
  }
}

system · April 22, 2020, 12:23pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.