# Best way to add metadata stored on external server?

**URL:** <https://discuss.elastic.co/t/best-way-to-add-metadata-stored-on-external-server/162679>\
**Category:** Logstash\
**Created:** [January 2, 2019, 3:29pm UTC](https://discuss.elastic.co/t/best-way-to-add-metadata-stored-on-external-server/162679 "2019-01-02T15:29:55Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![asp](https://avatars.discourse-cdn.com/v4/letter/a/9fc348/32.png) [@asp](https://discuss.elastic.co/u/asp)\
**Post date:** [January 2, 2019, 3:29pm UTC](https://discuss.elastic.co/t/best-way-to-add-metadata-stored-on-external-server/162679/1 "2019-01-02T15:29:56Z")

</div>

Hi,

On our nightly build systems we have currently multiple directories prepared for installation. It looks like this:

```
/nightly/test_1
/nightly/test_2
/nightly/test_3

```

After the automatic deployment I have additional metainformation, which is created during the deployment process which I would like to merge:

```
test_1 -> branch_3773
test_2 -> branch_8823
test_3 -> branch_8826

```

Logfiles are written there: /nightly/test\_x/logs/\*.log

No I want to add this meta information of the branch to each log event.  
What is the best, efficient way? The data only changes once a day.

We are using multiple logstash instances. So if the metadata changes, all logstash instances need to know.

For performance reason I would like to have this metadata stored and cached locally inside logstash. I assume easiest way is to place this meta information on a file on the source host.  
What possibilities do I have then?

- using [translate](https://www.elastic.co/guide/en/logstash/current/plugins-filters-translate.html) filter plugin
  - con: I need to access a file on the source server by logstash server. Don't know if this would be allowed in production.
  - what would be the way to access this file from logstash on an external folder?

- shipping the file with filebeat to logstash. Logstash is writing it to elasticsearch. Using [elasticsearch](https://www.elastic.co/guide/en/logstash/current/plugins-filters-elasticsearch.html)-filter-plugin to read the current meta information.
  - con: i tried sth similar in the past, The performance impact was very big when doing elasticsearch-query for each log line. Is there some possibility of caching inside logstash, that the query runs only once a minute or so?

- setting up a database and write metadata information into that database. Use [jdbc\_static](https://www.elastic.co/guide/en/logstash/current/plugins-filters-jdbc_static.html) or [jdbc\_streaming](https://www.elastic.co/guide/en/logstash/current/plugins-filters-jdbc_streaming.html) for getting meta information cachable into all logstash instances
  - con: additional infrastructure needed (database)

- other approaches? What do you suggest?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [January 30, 2019, 3:33pm UTC](https://discuss.elastic.co/t/best-way-to-add-metadata-stored-on-external-server/162679/2 "2019-01-30T15:33:40Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.