Dynamic configuration of logstash

Elastic Stack Logstash
1

Hi,

Is there anyway to load logstash configuration dynamically using database or some filesystem. ?
The usecase is lets say i have multiple apaches where I am getting data from and I am getting data with their ips (each record has field 'ip' of apache server ip adress). We need to get host names for each ip and stamp on each record and send it to elasticsearch. But I have mapping of ip addresses to host names in my filesystem, So i need to read mapping from there dynamically and stamp host name on each record.

Thanks
Harsha

2

Logstash 2.2 supports dynamic configuration reloads (see https://www.elastic.co/blog/logstash-lines-2016-02-09) but unless you can use the dns filter this sounds like a job for the translate filter.

1 Like
3

sounds to me like the shipper on the sending machine should stamp the events with the hostname if you don't want to DNS-resolve them all. it's not something that should be handled by the collector itself, IMHO, therefore should not require a change in its logstash config.

4

Thanks for prompt response. I think translate can solve my problem giving local file as an input to it. I will explore little bit more and will come back.

5

This is one of requirement like mapping hostname. There are few more where I want to map something with something, like tagging request with some api name eg:based on request I will add a field apiname where I should get this mapping from some database or filesystem. I think translate can solve my problem.

6

If you only have to update the config like hostname you can just make a
template config with keys and then run a sed command or other template
merger prior to starting up Logstash. I did that to inject hostname and
Cloud metadata on ephemeral hosts that used Logstash.

7

Not sure if I misunderstood something, but there's always
https://www.elastic.co/guide/en/logstash/current/plugins-filters-environment.html

8

translate filter plugin solved my case. Thanks for your time :slightly_smiling:

9

Is it true that 2.2 supports this? I don't see anything in the docs or repo to indicate this? It seems like it has only been back ported to 2.3.

10

Is it true that 2.2 supports this? I don't see anything in the docs or repo to indicate this? It seems like it has only been back ported to 2.3.

You're right. It was in the blog post announcing the availability of the 2.2 release and I incorrectly inferred that it was included in 2.2.

11

With translate plugin and I am using yml dictionary from a file. When I change this file translate plugin is not picking latest changes. Is there any way to reload dictionary automatically whenever it changes?

12

By default Logstash will reload the file every five minutes if it has changed (configurable with the refresh_interval option).

1 Like