# Parsing and search data in logstash and elasticsearch

**URL:** <https://discuss.elastic.co/t/parsing-and-search-data-in-logstash-and-elasticsearch/127637>\
**Category:** Logstash\
**Created:** [April 11, 2018, 1:17pm UTC](https://discuss.elastic.co/t/parsing-and-search-data-in-logstash-and-elasticsearch/127637 "2018-04-11T13:17:44Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![Mohamed\_Amine\_Bogate](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mohamed_amine_bogate/32/29868_2.png) [@Mohamed\_Amine\_Bogate](https://discuss.elastic.co/u/Mohamed_Amine_Bogate)\
**Post date:** [April 11, 2018, 1:17pm UTC](https://discuss.elastic.co/t/parsing-and-search-data-in-logstash-and-elasticsearch/127637/1 "2018-04-11T13:17:44Z")

</div>

This is an extract of my log file

 ![30531388_1972358626168912_8191993682908413952_o](https://us1.discourse-cdn.com/elastic/original/3X/d/5/d5b8389415d41de7763111b4ab711c5d718557e5.png)  
I have parsed it and this is the result json  
 ![30516672_1972358362835605_1713377130001203200_o](https://us1.discourse-cdn.com/elastic/original/3X/2/1/21108725800fe1a400dad88b9941c4ce13f27fca.png)  
 ![30594431_1972358459502262_7381339779977183232_o](https://us1.discourse-cdn.com/elastic/original/3X/2/1/214bdc9b660084a602bc87b6e57fe92e76d15c0d.png)  
and the result as shown in the dashboard  
 ![30652806_1972359552835486_503403023591014400_n](https://us1.discourse-cdn.com/elastic/original/3X/f/4/f4e19c5a6a5049d83c7dcb96e9e3e431ee9666de.png)

Now , I want to determine the execution time of each request in each thread, like this:

QueryA  
thread262 2 milliseconds (the first)

thread263 4 milliseconds

QueryB  
thread262 1 milliseconds

Thank you!

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [April 11, 2018, 1:32pm UTC](https://discuss.elastic.co/t/parsing-and-search-data-in-logstash-and-elasticsearch/127637/2 "2018-04-11T13:32:29Z")

</div>

How do you know that query A is 2 milliseconds and query B is 1 millisecond and not the other way around?

---

<div class="post-metadata">

**Author:** ![Mohamed\_Amine\_Bogate](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mohamed_amine_bogate/32/29868_2.png) [@Mohamed\_Amine\_Bogate](https://discuss.elastic.co/u/Mohamed_Amine_Bogate)\
**Post date:** [April 11, 2018, 4:53pm UTC](https://discuss.elastic.co/t/parsing-and-search-data-in-logstash-and-elasticsearch/127637/3 "2018-04-11T16:53:31Z")

</div>

it's just a hypothesis : the first timestamp must be granted to the first query  
Thank you

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [April 11, 2018, 5:59pm UTC](https://discuss.elastic.co/t/parsing-and-search-data-in-logstash-and-elasticsearch/127637/4 "2018-04-11T17:59:46Z")

</div>

This does it the other way around, with the first timestamp going to the last query, but it should get you started

```auto
filter {
  dissect { mapping => ["message", '%{ts} %{+ts} %{+ts} %{+ts} |%{} : %{} |%{}-[%{thread}]: %{} %{text}' ] }
  if [text] =~ /^Find/ {
    mutate { gsub => ["text", "Find query : : ", ""] }
    aggregate {
      task_id => "%{thread}"
      code => "(map['queries'] ||= []).push(event.get('text'))"
    }
    drop {}
  }
  if [text] =~ /^Elapsed/ {
    mutate { gsub => ["text", "ElapsedTime for Query Execution ", "", "text", " milliseconds", ""] }
    mutate { convert => { "text" => "integer" } }
    aggregate {
      task_id => "%{thread}"
      code => "event.set('query', map['queries'].pop)"
      map_action => "update"
    }
  }
  date { match => ["ts" , "MMM dd',' YYYY HH:mm:ss:SSS"] }
}

```

You would need to think about how to do timeouts, otherwise this will leak memory. Also, make sure you set --pipeline.workers 1

---

<div class="post-metadata">

**Author:** ![Mohamed\_Amine\_Bogate](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mohamed_amine_bogate/32/29868_2.png) [@Mohamed\_Amine\_Bogate](https://discuss.elastic.co/u/Mohamed_Amine_Bogate)\
**Post date:** [April 12, 2018, 1:26pm UTC](https://discuss.elastic.co/t/parsing-and-search-data-in-logstash-and-elasticsearch/127637/5 "2018-04-12T13:26:44Z")

</div>

thankyou so much, it works

---

<div class="post-metadata">

**Author:** ![Mohamed\_Amine\_Bogate](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mohamed_amine_bogate/32/29868_2.png) [@Mohamed\_Amine\_Bogate](https://discuss.elastic.co/u/Mohamed_Amine_Bogate)\
**Post date:** [April 18, 2018, 2:06pm UTC](https://discuss.elastic.co/t/parsing-and-search-data-in-logstash-and-elasticsearch/127637/6 "2018-04-18T14:06:36Z")

</div>

Can i assign a unique id to every query while parsing my log and show the really query with a pou-up in kibana ?  
Thank you

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [April 18, 2018, 4:49pm UTC](https://discuss.elastic.co/t/parsing-and-search-data-in-logstash-and-elasticsearch/127637/7 "2018-04-18T16:49:34Z")

</div>

You might want to ask a new question for that.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 16, 2018, 4:49pm UTC](https://discuss.elastic.co/t/parsing-and-search-data-in-logstash-and-elasticsearch/127637/8 "2018-05-16T16:49:40Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.