# Start FileBeat using publish

**URL:** <https://discuss.elastic.co/t/start-filebeat-using-publish/200710>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [September 23, 2019, 2:44pm UTC](https://discuss.elastic.co/t/start-filebeat-using-publish/200710 "2019-09-23T14:44:26Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![Ghaith\_Haddad](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ghaith_haddad/32/49288_2.png) [@Ghaith\_Haddad](https://discuss.elastic.co/u/Ghaith_Haddad)\
**Post date:** [September 23, 2019, 2:44pm UTC](https://discuss.elastic.co/t/start-filebeat-using-publish/200710/1 "2019-09-23T14:44:27Z")

</div>

Hello everyone,

Im not sure what the difference is between starting filebeat using: service start filebeat AND ./filebeat -e -c /etc/filebeat/filebeat.yml -d "publish"

the latter seems to work better than the first one tho (to me atleast) any idea why and how can i make it run in the background?

Any help or guidance is appreciated, thanks in advance.

Cheers!

---

<div class="post-metadata">

**Author:** ![Michael\_Madden](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/michael_madden/32/46640_2.png) [@Michael\_Madden](https://discuss.elastic.co/u/Michael_Madden)\
**Post date:** [September 23, 2019, 3:21pm UTC](https://discuss.elastic.co/t/start-filebeat-using-publish/200710/2 "2019-09-23T15:21:57Z")

</div>

Hello, thanks for reaching out about filebeat. Could you let us know some more information about your filebeat implementation?

- Which version of filebeat are you using?
- Which distribution and version of Linux are you using?
- Do you get any errors when starting filebeat as service? filebeat defaults to logging to `/var/log/filebeat`.
- Which user do you using when running `service filebeat start`?

Thanks.

---

<div class="post-metadata">

**Author:** ![Ghaith\_Haddad](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ghaith_haddad/32/49288_2.png) [@Ghaith\_Haddad](https://discuss.elastic.co/u/Ghaith_Haddad)\
**Post date:** [September 23, 2019, 3:34pm UTC](https://discuss.elastic.co/t/start-filebeat-using-publish/200710/3 "2019-09-23T15:34:32Z")

</div>

Hello @Michael_Madden,

im using the latest version 7.3.2  
ubuntu 18.04  
no errors while running filebeat as a service  
im running it as root user.

should i be using a different user, aka regular user?

Thank you

---

<div class="post-metadata">

**Author:** ![Michael\_Madden](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/michael_madden/32/46640_2.png) [@Michael\_Madden](https://discuss.elastic.co/u/Michael_Madden)\
**Post date:** [September 23, 2019, 3:39pm UTC](https://discuss.elastic.co/t/start-filebeat-using-publish/200710/4 "2019-09-23T15:39:39Z")

</div>

Hi,

I wonder if the logs are getting stored in `journald`? After starting the service with `service filebeat start`, could you try the following commands?

```auto
ps auxw | grep filebeat
systemctl status filebeat
journalctl -u filebeat.service

```

---

<div class="post-metadata">

**Author:** ![Ghaith\_Haddad](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ghaith_haddad/32/49288_2.png) [@Ghaith\_Haddad](https://discuss.elastic.co/u/Ghaith_Haddad)\
**Post date:** [September 23, 2019, 3:49pm UTC](https://discuss.elastic.co/t/start-filebeat-using-publish/200710/5 "2019-09-23T15:49:14Z")

</div>

> [@Michael\_Madden](#):
>
> ps auxw | grep filebeat

should i run them as root user or regular user?

---

<div class="post-metadata">

**Author:** ![Ghaith\_Haddad](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ghaith_haddad/32/49288_2.png) [@Ghaith\_Haddad](https://discuss.elastic.co/u/Ghaith_Haddad)\
**Post date:** [September 23, 2019, 3:52pm UTC](https://discuss.elastic.co/t/start-filebeat-using-publish/200710/6 "2019-09-23T15:52:39Z")

</div>

> [@Michael\_Madden](#):
>
> ps auxw | grep filebeat

i ran them as root user on one of the servers and i got a huge list of logs for the journalctl command.

---

<div class="post-metadata">

**Author:** ![Ghaith\_Haddad](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ghaith_haddad/32/49288_2.png) [@Ghaith\_Haddad](https://discuss.elastic.co/u/Ghaith_Haddad)\
**Post date:** [September 23, 2019, 3:57pm UTC](https://discuss.elastic.co/t/start-filebeat-using-publish/200710/7 "2019-09-23T15:57:31Z")

</div>

My main goal is to send the log files using filebeat to logstash. but if i run the filebeat and logstash as a service instead of the whole command it doesnt work so i use the whole command instead: ./filebeat -e -c /etc/filebeat/filebeat.yml -d "publish" and ./logstash -f /etc/logstash/conf.d/logs.conf

is there any way to not have to use the whole command?

---

<div class="post-metadata">

**Author:** ![Michael\_Madden](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/michael_madden/32/46640_2.png) [@Michael\_Madden](https://discuss.elastic.co/u/Michael_Madden)\
**Post date:** [September 23, 2019, 3:58pm UTC](https://discuss.elastic.co/t/start-filebeat-using-publish/200710/8 "2019-09-23T15:58:09Z")

</div>

You can run the `ps` command as a regular user. I wanted to see if any `filebeat` processes were running.

---

<div class="post-metadata">

**Author:** ![Michael\_Madden](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/michael_madden/32/46640_2.png) [@Michael\_Madden](https://discuss.elastic.co/u/Michael_Madden)\
**Post date:** [September 23, 2019, 4:00pm UTC](https://discuss.elastic.co/t/start-filebeat-using-publish/200710/9 "2019-09-23T16:00:00Z")

</div>

Does the output of `journalctl -u filebeat.service` seem related to filebeat? Are there any indicators why the service cannot be started?

Also, is logstash also running on the same virtual machine / server as filebeat?

---

<div class="post-metadata">

**Author:** ![Ghaith\_Haddad](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ghaith_haddad/32/49288_2.png) [@Ghaith\_Haddad](https://discuss.elastic.co/u/Ghaith_Haddad)\
**Post date:** [September 23, 2019, 5:02pm UTC](https://discuss.elastic.co/t/start-filebeat-using-publish/200710/10 "2019-09-23T17:02:22Z")

</div>

no logstash is on a different server. And yeah the output of journalctl seems related to filebeat. i see this warning in the output tho not sure if its related:

Filebeat is unable to load the Ingest Node pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the Ingest Node pipelines or are using Logstash pipelines, you can ignore this warning

in my filebeat.yml file i commented out the elasticsearch output because i am using logstash instead.

---

<div class="post-metadata">

**Author:** ![Ghaith\_Haddad](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ghaith_haddad/32/49288_2.png) [@Ghaith\_Haddad](https://discuss.elastic.co/u/Ghaith_Haddad)\
**Post date:** [September 23, 2019, 5:06pm UTC](https://discuss.elastic.co/t/start-filebeat-using-publish/200710/11 "2019-09-23T17:06:53Z")

</div>

i know this may sound ignorant but is there anyway to configure the service start filebeat command to run in a certain way that i want it to?

---

<div class="post-metadata">

**Author:** ![Michael\_Madden](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/michael_madden/32/46640_2.png) [@Michael\_Madden](https://discuss.elastic.co/u/Michael_Madden)\
**Post date:** [September 23, 2019, 6:51pm UTC](https://discuss.elastic.co/t/start-filebeat-using-publish/200710/12 "2019-09-23T18:51:43Z")

</div>

Hello, I spun up a docker container for Ubuntu 18.04 to ensure the configuration shipped with filebeat 7.3.2 allowed you to run filebeat as a service.

I'd recommend starting with a default configuration to ensure filebeat starts as as a service. It looks like the service should be started with `service filebeat start` instead of `service start filebeat`.

```auto
root@8acca7f35df6:~# curl https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-7.3.2-amd64.deb -O -s
root@8acca7f35df6:~# dpkg -i filebeat-7.3.2-amd64.deb
Selecting previously unselected package filebeat.
(Reading database ... 4570 files and directories currently installed.)
Preparing to unpack filebeat-7.3.2-amd64.deb ...
Unpacking filebeat (7.3.2) ...
Setting up filebeat (7.3.2) ..

```

```auto
root@8acca7f35df6:~# service filebeat start
2019-09-23T18:44:57.833Z	INFO	instance/beat.go:607	Home path: [/usr/share/filebeat] Config path: [/etc/filebeat] Data path: [/var/lib/filebeat] Logs path: [/var/log/filebeat]
2019-09-23T18:44:57.836Z	INFO	instance/beat.go:615	Beat ID: 141a9a46-88fe-4032-a912-9991f19613f6
2019-09-23T18:44:57.838Z	INFO	[beat]	instance/beat.go:903	Beat info	{"system_info": {"beat": {"path": {"config": "/etc/filebeat", "data": "/var/lib/filebeat", "home": "/usr/share/filebeat", "logs": "/var/log/filebeat"}, "type": "filebeat", "uuid": "141a9a46-88fe-4032-a912-9991f19613f6"}}}
2019-09-23T18:44:57.838Z	INFO	[beat]	instance/beat.go:912	Build info	{"system_info": {"build": {"commit": "5b046c5a97fe1e312f22d40a1f05365621aad621", "libbeat": "7.3.2", "time": "2019-09-06T13:49:32.000Z", "version": "7.3.2"}}}
2019-09-23T18:44:57.838Z	INFO	[beat]	instance/beat.go:915	Go runtime info	{"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":4,"version":"go1.12.4"}}}
2019-09-23T18:44:57.840Z	INFO	[beat]	instance/beat.go:919	Host info	{"system_info": {"host": {"architecture":"x86_64","boot_time":"2019-09-23T17:33:30Z","containerized":true,"name":"8acca7f35df6","ip":["127.0.0.1/8","172.17.0.2/16"],"kernel_version":"4.9.184-linuxkit","mac":["02:42:ac:11:00:02"],"os":{"family":"debian","platform":"ubuntu","name":"Ubuntu","version":"18.04.3 LTS (Bionic Beaver)","major":18,"minor":4,"patch":3,"codename":"bionic"},"timezone":"UTC","timezone_offset_sec":0}}}
2019-09-23T18:44:57.841Z	INFO	[beat]	instance/beat.go:948	Process info	{"system_info": {"process": {"capabilities": {"inheritable":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"permitted":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"effective":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"bounding":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"ambient":null}, "cwd": "/", "exe": "/usr/share/filebeat/bin/filebeat", "name": "filebeat", "pid": 2803, "ppid": 2802, "seccomp": {"mode":"filter"}, "start_time": "2019-09-23T18:44:57.359Z"}}}
2019-09-23T18:44:57.841Z	INFO	instance/beat.go:292	Setup Beat: filebeat; Version: 7.3.2
2019-09-23T18:44:57.841Z	INFO	[index-management]	idxmgmt/std.go:178	Set output.elasticsearch.index to 'filebeat-7.3.2' as ILM is enabled.
2019-09-23T18:44:57.841Z	INFO	elasticsearch/client.go:170	Elasticsearch url: http://localhost:9200
2019-09-23T18:44:57.842Z	INFO	[publisher]	pipeline/module.go:97	Beat name: 8acca7f35df6
Config OK

```

```auto
root@8acca7f35df6:~# ps auxw | grep filebeat
root 2817 0.0 0.0 9300 636 pts/0 S 18:44 0:00 /usr/share/filebeat/bin/filebeat-god -r / -n -p /var/run/filebeat.pid -- /usr/share/filebeat/bin/filebeat -c /etc/filebeat/filebeat.yml -path.home /usr/share/filebeat -path.config /etc/filebeat -path.data /var/lib/filebeat -path.logs /var/log/filebeat
root 2818 0.4 0.3 967152 30724 pts/0 Sl 18:44 0:00 /usr/share/filebeat/bin/filebeat -c /etc/filebeat/filebeat.yml -path.home /usr/share/filebeat -path.config /etc/filebeat -path.data /var/lib/filebeat -path.logs /var/log/filebeat

```

---

<div class="post-metadata">

**Author:** ![Ghaith\_Haddad](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ghaith_haddad/32/49288_2.png) [@Ghaith\_Haddad](https://discuss.elastic.co/u/Ghaith_Haddad)\
**Post date:** [September 23, 2019, 6:54pm UTC](https://discuss.elastic.co/t/start-filebeat-using-publish/200710/13 "2019-09-23T18:54:14Z")

</div>

yeah you see when i type service filebeat start i dont get the same lines that you do, it just doesnt show anything in a way indicating that it started and when i check the status service filebeat status it says that its running fine

---

<div class="post-metadata">

**Author:** ![Ghaith\_Haddad](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ghaith_haddad/32/49288_2.png) [@Ghaith\_Haddad](https://discuss.elastic.co/u/Ghaith_Haddad)\
**Post date:** [September 23, 2019, 7:07pm UTC](https://discuss.elastic.co/t/start-filebeat-using-publish/200710/14 "2019-09-23T19:07:55Z")

</div>

ok here is the output of ps auxw | grep filebeat on a fresh filebeat installation on a new server without changing the configs whatsoever

```
root 10502 0.1 0.8 751800 32776 ? Ssl 19:06 0:00 /usr/share/filebeat/bin/filebeat -e -c /etc/filebeat/filebeat.yml -path.home /usr/share/filebeat -path.config /etc/filebeat -path.data /var/lib/filebeat -path.logs /var/log/filebeat
root 10534 0.0 0.0 14856 1108 pts/1 S+ 19:06 0:00 grep --color=auto filebeat
```

---

<div class="post-metadata">

**Author:** ![Michael\_Madden](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/michael_madden/32/46640_2.png) [@Michael\_Madden](https://discuss.elastic.co/u/Michael_Madden)\
**Post date:** [September 23, 2019, 7:09pm UTC](https://discuss.elastic.co/t/start-filebeat-using-publish/200710/15 "2019-09-23T19:09:05Z")

</div>

Thanks. When you issue `service filebeat start` are you using the default `filebeat.yml` or a customized version?

I could be possible that the startup process is different in a docker container versus a virtual machine or physical server.

---

<div class="post-metadata">

**Author:** ![Ghaith\_Haddad](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ghaith_haddad/32/49288_2.png) [@Ghaith\_Haddad](https://discuss.elastic.co/u/Ghaith_Haddad)\
**Post date:** [September 23, 2019, 7:09pm UTC](https://discuss.elastic.co/t/start-filebeat-using-publish/200710/16 "2019-09-23T19:09:47Z")

</div>

no im using the default filebeat.yml file, this server is an AWS EC2 instance btw, could that be why?

---

<div class="post-metadata">

**Author:** ![Ghaith\_Haddad](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ghaith_haddad/32/49288_2.png) [@Ghaith\_Haddad](https://discuss.elastic.co/u/Ghaith_Haddad)\
**Post date:** [September 23, 2019, 7:11pm UTC](https://discuss.elastic.co/t/start-filebeat-using-publish/200710/17 "2019-09-23T19:11:53Z")

</div>

and if i run `service filebeat status` it shows me the last 20 lines of logs or so showing what has been done and to what server it trying to connect to (and it does) but thats it, it doesnt send any log files to the logstash server

---

<div class="post-metadata">

**Author:** ![Michael\_Madden](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/michael_madden/32/46640_2.png) [@Michael\_Madden](https://discuss.elastic.co/u/Michael_Madden)\
**Post date:** [September 23, 2019, 7:19pm UTC](https://discuss.elastic.co/t/start-filebeat-using-publish/200710/18 "2019-09-23T19:19:03Z")

</div>

Hello,

Perhaps the filebeat instance is having issues communicating to Logstash? I'd refer to the following faq to debug the communication between filebeat and logstash.

[https://www.elastic.co/guide/en/beats/filebeat/7.1/faq.html#connection-problem](https://www.elastic.co/guide/en/beats/filebeat/7.1/faq.html#connection-problem)

---

<div class="post-metadata">

**Author:** ![Ghaith\_Haddad](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ghaith_haddad/32/49288_2.png) [@Ghaith\_Haddad](https://discuss.elastic.co/u/Ghaith_Haddad)\
**Post date:** [September 23, 2019, 7:27pm UTC](https://discuss.elastic.co/t/start-filebeat-using-publish/200710/19 "2019-09-23T19:27:51Z")

</div>

to be honest i dont think thats the issue, because, if run filebeat using this command `./filebeat -e -c /etc/filebeat/filebeat.yml -d "publish"` everything works fine, but one thing that i noticed when i ran the `ps auxw | grep filebeat` at the end i see this: `-path.logs /var/log/filebeat` but thats not the log files i had configured in the filebeat.yml file any idea oh how to fix this?

---

<div class="post-metadata">

**Author:** ![Ghaith\_Haddad](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ghaith_haddad/32/49288_2.png) [@Ghaith\_Haddad](https://discuss.elastic.co/u/Ghaith_Haddad)\
**Post date:** [September 23, 2019, 7:44pm UTC](https://discuss.elastic.co/t/start-filebeat-using-publish/200710/20 "2019-09-23T19:44:30Z")

</div>

FIXED IT!  
so apparently after i changed the filebeat.references.yml file to fetch the log files that i wanted then start filebeat normally using `service filebeat start` it actually sent the files and everything was working perfectly!

Thank you so very much good sir, your awesome!

[Next page](https://discuss.elastic.co/t/start-filebeat-using-publish/200710.md?page=2)