# Extracting data from multiple events from Elasticsearch using single logstash filter

**URL:** <https://discuss.elastic.co/t/extracting-data-from-multiple-events-from-elasticsearch-using-single-logstash-filter/96195>\
**Category:** Logstash\
**Created:** [August 8, 2017, 6:54am UTC](https://discuss.elastic.co/t/extracting-data-from-multiple-events-from-elasticsearch-using-single-logstash-filter/96195 "2017-08-08T06:54:01Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![sushil1016](https://avatars.discourse-cdn.com/v4/letter/s/67e7ee/32.png) [@sushil1016](https://discuss.elastic.co/u/sushil1016)\
**Post date:** [August 8, 2017, 6:54am UTC](https://discuss.elastic.co/t/extracting-data-from-multiple-events-from-elasticsearch-using-single-logstash-filter/96195/1 "2017-08-08T06:54:01Z")

</div>

I have log lines loaded in ElasticSearch which has the data scattered in multiple events, say event\_id is in event (line) number 5 and event\_action is available in event number 88, further event\_port information is available in event number 455. How can i extract this data so that my output looks like following. For this case multiline codec will not work.

```
{
event_id: 1223
event_action: "socket_open"
event_port: 76654
}

```

Currently I have the log files persisted so i can get the file path from ES. I tried to have a shell script executed from ruby filter, this shell script can perform grep commands and put the stdout data in a new event, like following.

```
input {
  elasticsearch {
	hosts => "localhost:9200"
	index => "my-logs"
  }
}

filter
{

     ruby {
    code => 'require "open3"
             file_path = event.get("file_path")
             cmd = "my_filter.sh -f #{file_path}"
             stdin, stdout, stderr = Open3.popen3(cmd)
             event.set("process_result", stdout.read)
             err = stderr.read
             if err.to_s.empty?
               filter_matched(event)
             else
               event.set("ext_script_err_msg", err)
             end'
      remove_field => ["file_path"]
   }
   }

```

With above approach I am facing problems.

1. Doing grep in huge files can be time consuming. Is there any alternative, whithout having to grep on files?

2. My input plugin (attached above) is taking events from Elastic Search which has file\_path set for "ALL" events in an index, this makes my\_filter.sh to be executed multiple times which is something I want to avoid. How can I extract unique file\_path from ES?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [September 5, 2017, 6:54am UTC](https://discuss.elastic.co/t/extracting-data-from-multiple-events-from-elasticsearch-using-single-logstash-filter/96195/2 "2017-09-05T06:54:20Z")

</div>

Closed on September 5, 2017, 6:54am UTC

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.