# Does filebeat cache?

**URL:** <https://discuss.elastic.co/t/does-filebeat-cache/35369>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [November 23, 2015, 11:43pm UTC](https://discuss.elastic.co/t/does-filebeat-cache/35369 "2015-11-23T23:43:09Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![sijis](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sijis/32/5662_2.png) [@sijis](https://discuss.elastic.co/u/sijis)\
**Post date:** [November 23, 2015, 11:43pm UTC](https://discuss.elastic.co/t/does-filebeat-cache/35369/1 "2015-11-23T23:43:09Z")

</div>

We bake our ami in AWS packer which includes filebeat.

However, we've noticed that new instances are showing the ip of of the instance used to create the ami (aka wrong ip) as shipper in logstash.

Currently, if we just restart filebeat, the shipper details are correct.

Is there a cache file that handles this which i'd have to remove?

Thanks,

Sijis

---

<div class="post-metadata">

**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\
**Post date:** [November 24, 2015, 7:08am UTC](https://discuss.elastic.co/t/does-filebeat-cache/35369/2 "2015-11-24T07:08:53Z")

</div>

It seems like if you boot up your new instance, filebeat is already running and not started. Is this intended? The shipper ip is loaded once during starting filebeat and reading the config and from then on cached as it assumes it will not change during running.

---

<div class="post-metadata">

**Author:** ![sijis](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sijis/32/5662_2.png) [@sijis](https://discuss.elastic.co/u/sijis)\
**Post date:** [November 24, 2015, 10:31pm UTC](https://discuss.elastic.co/t/does-filebeat-cache/35369/3 "2015-11-24T22:31:27Z")

</div>

The new instance is built from an ami which had filebeat installed. When the instance first boots, it should start filebeat (and it does).  
So where is the cache location of this config? I think if i delete that cache file pre the ami 'sealing', then i should be OK.  
My speculation is that somehow filebeat is reading the cache instead of the new hostname.

As an added note, i do not have a 'shipper' section in my filebeat.yml file.

---

<div class="post-metadata">

**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\
**Post date:** [November 25, 2015, 7:35am UTC](https://discuss.elastic.co/t/does-filebeat-cache/35369/4 "2015-11-25T07:35:40Z")

</div>

Which OS and Version are you using? I have to make some tests myself here. Filebeat itself does not have a cache location, but I have to check in detail where Golang reads the value from.

---

<div class="post-metadata">

**Author:** ![sijis](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sijis/32/5662_2.png) [@sijis](https://discuss.elastic.co/u/sijis)\
**Post date:** [November 25, 2015, 8:51pm UTC](https://discuss.elastic.co/t/does-filebeat-cache/35369/5 "2015-11-25T20:51:08Z")

</div>

@ruflin All systems are Fedora22.

On server (forward to logstash):

- filebeat-1.0.0-1.x86\_64

On logstash:

- logstash-2.1.0-1.noarch

---

<div class="post-metadata">

**Author:** ![sijis](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sijis/32/5662_2.png) [@sijis](https://discuss.elastic.co/u/sijis)\
**Post date:** [November 25, 2015, 9:07pm UTC](https://discuss.elastic.co/t/does-filebeat-cache/35369/6 "2015-11-25T21:07:09Z")

</div>

These are the configs i'm using

## filebeat config on server:

```
logging:
  level: info

  # enable file rotation with default configuration
  to_files: true

  # do not log to syslog
  to_syslog: false

  files:
    path: /var/log/filebeat
    name: filebeat.log
    keepfiles: 3

filebeat:

  # General filebeat configuration options
  #spool_size: 1024
  #idle_timeout: 5s
  registry_file: /var/lib/filebeat/registry
  config_dir: /etc/filebeat/conf.d

  prospectors:
    # Each - is a prospector. Below are the prospector specific configurations
    -
      paths:
        - /var/log/filebeat/*.log

      #encoding: plain
      type: log
      #ignore_older: 24h
      #scan_frequency: 10s
      #harvester_buffer_size: 16384
      #tail_on_rotate: false

output:
  logstash:
    enabled: true

    hosts: ["logstash:1001"]

    # index configures '@metadata.beat' field to be used by Logstash for
    # indexing. By Default the beat name is used (e.g. filebeat, topbeat, packetbeat)
    index: mybeat
```

## logstash config:

```
input {
  beats {
    port => 1001
  }
}
output {
  stdout {
    codec => rubydebug
  }
}
```

---

<div class="post-metadata">

**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\
**Post date:** [November 26, 2015, 12:39pm UTC](https://discuss.elastic.co/t/does-filebeat-cache/35369/7 "2015-11-26T12:39:51Z")

</div>

@sijis I did some tests on my side with fedora and the hostname change seems to be depend on timing and where / how do you change it. Which command do you use the change the hostname? At what stage is the hostname of the new machine set related to the start of filebeat?

For golang the file that seems to matter seems to be: `/proc/sys/kernel/hostname`

Here is the code reading it: [https://github.com/golang/go/blob/master/src/os/sys\_linux.go#L10](https://github.com/golang/go/blob/master/src/os/sys_linux.go#L10)

As you write after restarting filebeat it works as expected, I assume it has more to do with timing of setting the hostname. But be aware that I'm not a fedora expert 🙂

---

<div class="post-metadata">

**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\
**Post date:** [November 26, 2015, 2:49pm UTC](https://discuss.elastic.co/t/does-filebeat-cache/35369/8 "2015-11-26T14:49:03Z")

</div>

@sijis can you post the output of 'ls /etc/rc\<level\>.d' ? Maybe filebeat is started before network/hostname is configured.

---

<div class="post-metadata">

**Author:** ![sijis](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sijis/32/5662_2.png) [@sijis](https://discuss.elastic.co/u/sijis)\
**Post date:** [November 27, 2015, 3:40am UTC](https://discuss.elastic.co/t/does-filebeat-cache/35369/9 "2015-11-27T03:40:49Z")

</div>

@steffens I think there is an order issue going on.

This is the output of `ls /etc/rc3.d/`

```
[fedora@ip-xx-xx-xx-xx ~]$ ls /etc/rc3.d
K50netconsole S10network S50logstash S95jexec S98filebeat
```

That output seems a little misleading, as Fedora22 uses systemd. This is the output of that is the following:

```
[fedora@ip-xx-xx-xx-xx ~]$ systemctl list-dependencies
default.target
● ├─auditd.service
● ├─cloud-config.service
● ├─cloud-final.service
● ├─cloud-init-local.service
● ├─cloud-init.service
● ├─crond.service
● ├─dbus.service
● ├─filebeat.service
● ├─logstash.service
● ├─network.service
● ├─plymouth-quit-wait.service
● ├─plymouth-quit.service
● ├─sshd.service
● ├─systemd-ask-password-wall.path
● ├─systemd-logind.service
● ├─systemd-update-utmp-runlevel.service
● ├─systemd-user-sessions.service
● ├─tailon.service
● ├─basic.target
● │ ├─dnf-makecache.timer
● │ ├─fedora-autorelabel-mark.service
● │ ├─fedora-autorelabel.service
● │ ├─fedora-loadmodules.service
● │ ├─paths.target
● │ ├─slices.target
● │ │ ├─-.slice
● │ │ └─system.slice
● │ ├─sockets.target
● │ │ ├─dbus.socket
● │ │ ├─systemd-initctl.socket
● │ │ ├─systemd-journald-audit.socket
● │ │ ├─systemd-journald-dev-log.socket
● │ │ ├─systemd-journald.socket
● │ │ ├─systemd-shutdownd.socket
● │ │ ├─systemd-udevd-control.socket
● │ │ └─systemd-udevd-kernel.socket
● │ ├─sysinit.target
● │ │ ├─dev-hugepages.mount
● │ │ ├─dev-mqueue.mount
● │ │ ├─dracut-shutdown.service
● │ │ ├─kmod-static-nodes.service
● │ │ ├─ldconfig.service
● │ │ ├─plymouth-read-write.service
● │ │ ├─plymouth-start.service
● │ │ ├─proc-sys-fs-binfmt_misc.automount
● │ │ ├─sys-fs-fuse-connections.mount
● │ │ ├─sys-kernel-config.mount
● │ │ ├─sys-kernel-debug.mount
● │ │ ├─systemd-ask-password-console.path
● │ │ ├─systemd-binfmt.service
● │ │ ├─systemd-firstboot.service
● │ │ ├─systemd-hwdb-update.service
● │ │ ├─systemd-journal-catalog-update.service
● │ │ ├─systemd-journal-flush.service
● │ │ ├─systemd-journald.service
● │ │ ├─systemd-machine-id-commit.service
● │ │ ├─systemd-modules-load.service
● │ │ ├─systemd-random-seed.service
● │ │ ├─systemd-sysctl.service
● │ │ ├─systemd-sysusers.service
● │ │ ├─systemd-tmpfiles-setup-dev.service
● │ │ ├─systemd-tmpfiles-setup.service
● │ │ ├─systemd-udev-trigger.service
● │ │ ├─systemd-udevd.service
● │ │ ├─systemd-update-done.service
● │ │ ├─systemd-update-utmp.service
● │ │ ├─systemd-vconsole-setup.service
● │ │ ├─cryptsetup.target
● │ │ ├─local-fs.target
● │ │ │ ├─-.mount
● │ │ │ ├─fedora-import-state.service
● │ │ │ ├─fedora-readonly.service
● │ │ │ ├─systemd-fsck-root.service
● │ │ │ ├─systemd-remount-fs.service
● │ │ │ └─tmp.mount
● │ │ └─swap.target
● │ └─timers.target
● │ └─systemd-tmpfiles-clean.timer
● ├─getty.target
● │ ├─getty@tty1.service
● │ └─serial-getty@ttyS0.service
● └─remote-fs.target
```

Ahh, i think we are on to something. I notice filebeat runs before network.service.

---

<div class="post-metadata">

**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\
**Post date:** [November 27, 2015, 2:14pm UTC](https://discuss.elastic.co/t/does-filebeat-cache/35369/10 "2015-11-27T14:14:27Z")

</div>

Yep. filebeat and logstash should be among the last processes to start, not the first.

Is a service file available for filebeat/logstash? Can you add network.service as requirement for filebeat/logstash?

---

<div class="post-metadata">

**Author:** ![sijis](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sijis/32/5662_2.png) [@sijis](https://discuss.elastic.co/u/sijis)\
**Post date:** [November 27, 2015, 5:24pm UTC](https://discuss.elastic.co/t/does-filebeat-cache/35369/11 "2015-11-27T17:24:22Z")

</div>

I would agree that those service should start much later in bootup sequence.

> Is a service file available for filebeat/logstash?

There is no filebeat.service or logstash.service file. I installed these from official elastic repositories.

> Can you add network.service as requirement for filebeat/logstash?

Since there's no service file, i can't do it through systemd. There is probably a way to do is via the initd method. i'd have to research it a bit as i don't know it offhand.

My first guess, is that i'd have to modify these lines:

```
# Required-Start: $remote_fs $syslog
# Required-Stop: $remote_fs $syslog
```

This is what the top of the the startup scripts for both logstash and filebeat, so you see what I have.

```
[fedora@ip-xx-xx-xx-xx init.d]$ ls
filebeat functions jexec logstash netconsole network README
[fedora@ip-xx-xx-xx-xx init.d]$ head filebeat logstash -n17
==> filebeat <==
#!/bin/bash
#
# filebeat filebeat shipper
#
# chkconfig: 2345 98 02
#

### BEGIN INIT INFO
# Provides: filebeat
# Required-Start: $local_fs $network $syslog
# Required-Stop: $local_fs $network $syslog
# Default-Start: 2 3 4 5
# Default-Stop: 0 1 6
# Short-Description: Sends log files to Logstash or directly to Elasticsearch.
# Description: filebeat is a shipper part of the Elastic Beats 
# family. Please see: https://www.elastic.co/products/beats
### END INIT INFO

==> logstash <==
#!/bin/sh
# Init script for logstash
# Maintained by Elasticsearch
# Generated by pleaserun.
# Implemented based on LSB Core 3.1:
# * Sections: 20.2, 20.3
#
### BEGIN INIT INFO
# Provides: logstash
# Required-Start: $remote_fs $syslog
# Required-Stop: $remote_fs $syslog
# Default-Start: 2 3 4 5
# Default-Stop: 0 1 6
# Short-Description:
# Description: Starts Logstash as a daemon.
### END INIT INFO
```

---

<div class="post-metadata">

**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\
**Post date:** [November 27, 2015, 5:47pm UTC](https://discuss.elastic.co/t/does-filebeat-cache/35369/12 "2015-11-27T17:47:13Z")

</div>

I'm not really sure about the order of list-dependencies. Can you also try

```
$ systemd list-dependencies filebeat.service

```

I see some more services: cloud-config, cloud-final, cloud-init-local, cloud-init . I don't know amazon ami well, but what's the purpose of these services? Maybe any of these changes the hostname?

---

<div class="post-metadata">

**Author:** ![sijis](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sijis/32/5662_2.png) [@sijis](https://discuss.elastic.co/u/sijis)\
**Post date:** [November 27, 2015, 5:58pm UTC](https://discuss.elastic.co/t/does-filebeat-cache/35369/13 "2015-11-27T17:58:02Z")

</div>

I do believe cloud-init stuff does change it. We don't set or change instances to any specific name or anything and just let ec2 dhcp basically handle it.

I'll provide the output of list-dependencies shortly.

Sijis

---

<div class="post-metadata">

**Author:** ![sijis](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sijis/32/5662_2.png) [@sijis](https://discuss.elastic.co/u/sijis)\
**Post date:** [November 27, 2015, 6:02pm UTC](https://discuss.elastic.co/t/does-filebeat-cache/35369/14 "2015-11-27T18:02:30Z")

</div>

As an added note, cloud-init is needed for user-data stuff in ec2, that's why its there.

This is the output request:

```
[fedora@ip-xx-xx-xx-xx init.d]$ systemctl list-dependencies filebeat.service
filebeat.service
● ├─system.slice
● ├─basic.target
● │ ├─dnf-makecache.timer
● │ ├─fedora-autorelabel-mark.service
● │ ├─fedora-autorelabel.service
● │ ├─fedora-loadmodules.service
● │ ├─paths.target
● │ ├─slices.target
● │ │ ├─-.slice
● │ │ └─system.slice
● │ ├─sockets.target
● │ │ ├─dbus.socket
● │ │ ├─systemd-initctl.socket
● │ │ ├─systemd-journald-audit.socket
● │ │ ├─systemd-journald-dev-log.socket
● │ │ ├─systemd-journald.socket
● │ │ ├─systemd-shutdownd.socket
● │ │ ├─systemd-udevd-control.socket
● │ │ └─systemd-udevd-kernel.socket
● │ ├─sysinit.target
● │ │ ├─dev-hugepages.mount
● │ │ ├─dev-mqueue.mount
● │ │ ├─dracut-shutdown.service
● │ │ ├─kmod-static-nodes.service
● │ │ ├─ldconfig.service
● │ │ ├─plymouth-read-write.service
● │ │ ├─plymouth-start.service
● │ │ ├─proc-sys-fs-binfmt_misc.automount
● │ │ ├─sys-fs-fuse-connections.mount
● │ │ ├─sys-kernel-config.mount
● │ │ ├─sys-kernel-debug.mount
● │ │ ├─systemd-ask-password-console.path
● │ │ ├─systemd-binfmt.service
● │ │ ├─systemd-firstboot.service
● │ │ ├─systemd-hwdb-update.service
● │ │ ├─systemd-journal-catalog-update.service
● │ │ ├─systemd-journal-flush.service
● │ │ ├─systemd-journald.service
● │ │ ├─systemd-machine-id-commit.service
● │ │ ├─systemd-modules-load.service
● │ │ ├─systemd-random-seed.service
● │ │ ├─systemd-sysctl.service
● │ │ ├─systemd-sysusers.service
● │ │ ├─systemd-tmpfiles-setup-dev.service
● │ │ ├─systemd-tmpfiles-setup.service
● │ │ ├─systemd-udev-trigger.service
● │ │ ├─systemd-udevd.service
● │ │ ├─systemd-update-done.service
● │ │ ├─systemd-update-utmp.service
● │ │ ├─systemd-vconsole-setup.service
● │ │ ├─cryptsetup.target
● │ │ ├─local-fs.target
● │ │ │ ├─-.mount
● │ │ │ ├─fedora-import-state.service
● │ │ │ ├─fedora-readonly.service
● │ │ │ ├─systemd-fsck-root.service
● │ │ │ ├─systemd-remount-fs.service
● │ │ │ └─tmp.mount
● │ │ └─swap.target
● │ └─timers.target
● │ └─systemd-tmpfiles-clean.timer
● └─network-online.target
```

---

<div class="post-metadata">

**Author:** ![sijis](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sijis/32/5662_2.png) [@sijis](https://discuss.elastic.co/u/sijis)\
**Post date:** [November 27, 2015, 6:21pm UTC](https://discuss.elastic.co/t/does-filebeat-cache/35369/15 "2015-11-27T18:21:31Z")

</div>

If this helps to reproduce, i'm doing the following.  
i'm using the following ami (ami-81698dea) [us-east-1] + and installing filebeat `sudo dnf install filebeat` from elastic repos

```
[fedora@ip-xx-xx-xx-xx init.d]$ cat /etc/yum.repos.d/filebeat.repo 
[filebeat]
name=filebeat repository
baseurl=http://packages.elasticsearch.org/beats/yum/el/x86_64/
gpgcheck=0
gpgkey=http://packages.elasticsearch.org/GPG-KEY-elasticsearch
enabled=1
```

Then i create an ami from those steps using packer.

Sijis

---

<div class="post-metadata">

**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\
**Post date:** [November 27, 2015, 6:31pm UTC](https://discuss.elastic.co/t/does-filebeat-cache/35369/16 "2015-11-27T18:31:11Z")

</div>

Reading [amzon linux ami basics](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AmazonLinuxAMIBasics.html) cloud-init will set the hostname. with systemd potentially parallelizing service startup, cloud-init must be a dependency for filebeat I would say.

---

<div class="post-metadata">

**Author:** ![sijis](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sijis/32/5662_2.png) [@sijis](https://discuss.elastic.co/u/sijis)\
**Post date:** [November 27, 2015, 6:38pm UTC](https://discuss.elastic.co/t/does-filebeat-cache/35369/17 "2015-11-27T18:38:01Z")

</div>

Yeah, that would seem like what would need to be done.  
The packaging for filebeat doesn't use a systemd script to launch though. i'll see if i can hack something with either the provided script in /init.d or a custom service file.

---

<div class="post-metadata">

**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\
**Post date:** [November 27, 2015, 9:07pm UTC](https://discuss.elastic.co/t/does-filebeat-cache/35369/18 "2015-11-27T21:07:10Z")

</div>

Cool, looking forward to hear about your experience. Unfortunately I don't know systemd or amazon ami this well. Maybe we can integrate your solution into [beats-packer](https://github.com/elastic/beats-packer).

---

<div class="post-metadata">

**Author:** ![sijis](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sijis/32/5662_2.png) [@sijis](https://discuss.elastic.co/u/sijis)\
**Post date:** [November 30, 2015, 9:00pm UTC](https://discuss.elastic.co/t/does-filebeat-cache/35369/19 "2015-11-30T21:00:56Z")

</div>

@steffens I ended up adding the following contents below in `/etc/systemd/system/filebeat.service` and it started working just fine.

`
[Unit]
Description=Filebeat Service
Requires=cloud-final.service
After=syslog.target network.target cloud-final.service
[Service]
Type=simple
ExecStart=/usr/bin/filebeat -c /etc/filebeat/filebeat.yml
[Install]
WantedBy=multi-user.target
`

I did try to incorporate the changes to the beats-packer repo but I wasn't able to fully understand the process thus I did not create a PR. However, I do think, long term, there will have to be a feature implemented in fpm that allows systemd scripts from being used for rpm based systems. In fpm's master branch, that feature appears to be available for debian based systems.

---

<div class="post-metadata">

**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\
**Post date:** [December 1, 2015, 10:42am UTC](https://discuss.elastic.co/t/does-filebeat-cache/35369/20 "2015-12-01T10:42:36Z")

</div>

Did create a new [github issue](https://github.com/elastic/beats-packer/issues/35).

in [platforms](https://github.com/elastic/beats-packer/tree/master/platforms) directory building for different platforms is scripted. Every 'platform' has a build.sh which is responsible to apply the local script templates to current build parameters and run final script (run-&ltid\>.sh).

Maybe for AMI support we will need an extra package or some checks if cloud-init is installed to build service file from template. Don't know much about fpm/rpm myself.

[Next page](https://discuss.elastic.co/t/does-filebeat-cache/35369.md?page=2)