# Appending data from MongoDB into log files being processed by Logstash and parsed into Elasticsearch

**URL:** <https://discuss.elastic.co/t/appending-data-from-mongodb-into-log-files-being-processed-by-logstash-and-parsed-into-elasticsearch/92564>\
**Category:** Logstash\
**Created:** [July 11, 2017, 3:08am UTC](https://discuss.elastic.co/t/appending-data-from-mongodb-into-log-files-being-processed-by-logstash-and-parsed-into-elasticsearch/92564 "2017-07-11T03:08:21Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![kmin93](https://avatars.discourse-cdn.com/v4/letter/k/dbc845/32.png) [@kmin93](https://discuss.elastic.co/u/kmin93)\
**Post date:** [July 11, 2017, 3:08am UTC](https://discuss.elastic.co/t/appending-data-from-mongodb-into-log-files-being-processed-by-logstash-and-parsed-into-elasticsearch/92564/1 "2017-07-11T03:08:21Z")

</div>

Sorry about the title, my case really couldn't be explained with a single sentence.

Here is my situation:

1. I have a large set of log files (around 4GB) that I wish to parse with Logstash to use with Elastic stack (Logstash, Elasticsearch, Kibana).
2. In the logs, there is a serial number that I have successfully parsed with Logstash. This number corresponds to an index of a MongoDB collection. As each log is being parsed, I want to be able to query the collection with the parsed number and retrieve data which I want to include in the final output that is passed to Elasticsearch.

To make things clearer, here's a rough example. Suppose I have the raw log:

```
2017-11-20 14:24:14.011 123 log_number_one

```

Before the parsed log gets sent to Elasticsearch, I want to query my MongoDB collection with `123`, and get data `data1` and `data2` to append to the document to be sent to Elasticsearch, so my end result will have fields be similar to something like:

```
{ 
    timestamp: 2017-11-20 14:24:14.011, 
    serial: 123, 
    data1: "foo", 
    data2: "bar", 
    log: log_number_one
}

```

An easier way to accomplish this, I assume, would be to simply preprocess the logs and run the numbers through MongoDB before parsing them through Logstash. However, seeing as though I have 4GBs' worth of logfiles, I was hoping for a way to achieve this in one single swoop. I was wondering whether my edge case would be solvable with the ruby filter plugin, where I could possibly run some arbitrary ruby code to do the above?

Any help / advice would be greatly appreciated!

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [July 11, 2017, 5:24am UTC](https://discuss.elastic.co/t/appending-data-from-mongodb-into-log-files-being-processed-by-logstash-and-parsed-into-elasticsearch/92564/2 "2017-07-11T05:24:16Z")

</div>

Depending on the number of records and total size of the data in MongoDB (assuming it is a reasonable size data set), you may be able to extract the data into a file where each serial number is associated with a string representation of the data in JSON form. You could then use the [translate filter](https://www.elastic.co/guide/en/logstash/current/plugins-filters-translate.html) to populate a field with the serialised JSON based on the serial number and then use a son filter to parse this and add it to the event.

---

<div class="post-metadata">

**Author:** ![kmin93](https://avatars.discourse-cdn.com/v4/letter/k/dbc845/32.png) [@kmin93](https://discuss.elastic.co/u/kmin93)\
**Post date:** [July 11, 2017, 5:33am UTC](https://discuss.elastic.co/t/appending-data-from-mongodb-into-log-files-being-processed-by-logstash-and-parsed-into-elasticsearch/92564/3 "2017-07-11T05:33:19Z")

</div>

Thank you so much for your help, I'll give it a try immediately!

EDIT: @Christian_Dahlqvist I haven't fully implemented it yet, but with the way things are going, I'm pretty sure it's going to work as intended. Thanks again! 🙂

EDIT 2: The MongoDB extract ended up being around 50MB, so I had to increase the JVM heap size for Logstash to be able to run normally. The method worked beautifully!

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [August 8, 2017, 5:33am UTC](https://discuss.elastic.co/t/appending-data-from-mongodb-into-log-files-being-processed-by-logstash-and-parsed-into-elasticsearch/92564/4 "2017-08-08T05:33:19Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.