# How can I filter certain information from the logs?

**URL:** https://discuss.elastic.co/t/how-can-i-filter-certain-information-from-the-logs/344293
**Category:** Kibana
**Created:** [October 3, 2023, 8:30am UTC](https://discuss.elastic.co/t/how-can-i-filter-certain-information-from-the-logs/344293 "2023-10-03T08:30:29Z")
**Posts on this page:** 9
**Page:** 1

<div class="post-metadata">

### Author: ![hta](https://avatars.discourse-cdn.com/v4/letter/h/a8b319/32.png) [@hta](https://discuss.elastic.co/u/hta)
#### Post date: [October 3, 2023, 8:30am UTC](https://discuss.elastic.co/t/how-can-i-filter-certain-information-from-the-logs/344293/1 "2023-10-03T08:30:29Z")

</div>

We work with ELK Stack and I have the task of creating meaningful visualizations from the log entries. I have logs in the following format:

```auto
{
  "@timestamp": [
    "2023-08-08T00:00:11.2123"
  ],
  "xxxxx": [
    "yyyyy"
  ],
  "message": [
    "some text some informations; name{name='marc', school='dsds', moreinformation='more'} more information"
  ],
  "abcdfg": [
    "some text some informations"
  ]
}

```

How could I, for example, filter out and count certain information from the message field?

I need the field for the school and it would be good if I count how many students a school has (based on the students' names).

I have it with the grok processors at the pipeline. Unfortunately I didn't come to a solution. Is there a better way? what else can I do?

---

<div class="post-metadata">

### Author: ![cperzrt10](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cperzrt10/32/116152_2.png) [@cperzrt10](https://discuss.elastic.co/u/cperzrt10)
#### Post date: [October 3, 2023, 10:04am UTC](https://discuss.elastic.co/t/how-can-i-filter-certain-information-from-the-logs/344293/2 "2023-10-03T10:04:18Z")

</div>

Hi @hta , try the following grok pattern

```auto
filter {
  grok {
    id => "name school grok filter"
    match => { 'message' => '^.*name=\'%{WORD:name}\'.*school=\'%{WORD:school}\''}
  }
}

```

it gets the name and the school from the message.

---

<div class="post-metadata">

### Author: ![hta](https://avatars.discourse-cdn.com/v4/letter/h/a8b319/32.png) [@hta](https://discuss.elastic.co/u/hta)
#### Post date: [October 3, 2023, 9:27pm UTC](https://discuss.elastic.co/t/how-can-i-filter-certain-information-from-the-logs/344293/4 "2023-10-03T21:27:21Z")

</div>

thank you very much for your help. That worked. I'm hoping the newly extracted fields will show up tomorrow when new logs come in.  
Where is the best place to enter the pattern? I added this under Kibana in the pipeline as a processor without "filter" and "grok" only `'^.*name=\'%{WORD:name}\'.*school=\'%{WORD:school}\'`  
It worked in the debugger. Is that ok?

---

<div class="post-metadata">

### Author: ![cperzrt10](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cperzrt10/32/116152_2.png) [@cperzrt10](https://discuss.elastic.co/u/cperzrt10)
#### Post date: [October 5, 2023, 7:31am UTC](https://discuss.elastic.co/t/how-can-i-filter-certain-information-from-the-logs/344293/5 "2023-10-05T07:31:22Z")

</div>

Finally it worked?

---

<div class="post-metadata">

### Author: ![hta](https://avatars.discourse-cdn.com/v4/letter/h/a8b319/32.png) [@hta](https://discuss.elastic.co/u/hta)
#### Post date: [October 12, 2023, 9:38pm UTC](https://discuss.elastic.co/t/how-can-i-filter-certain-information-from-the-logs/344293/6 "2023-10-12T21:38:17Z")

</div>

I can not say it. we are having technical difficulties. There are currently no new logs

---

<div class="post-metadata">

### Author: ![hta](https://avatars.discourse-cdn.com/v4/letter/h/a8b319/32.png) [@hta](https://discuss.elastic.co/u/hta)
#### Post date: [October 27, 2023, 7:55pm UTC](https://discuss.elastic.co/t/how-can-i-filter-certain-information-from-the-logs/344293/7 "2023-10-27T19:55:01Z")

</div>

yes, it worked. Thx!!! 🙂

---

<div class="post-metadata">

### Author: ![hta](https://avatars.discourse-cdn.com/v4/letter/h/a8b319/32.png) [@hta](https://discuss.elastic.co/u/hta)
#### Post date: [October 27, 2023, 8:02pm UTC](https://discuss.elastic.co/t/how-can-i-filter-certain-information-from-the-logs/344293/8 "2023-10-27T20:02:35Z")

</div>

Can you maybe tell me what I can do? I changed the above expression slightly. For example, with WORD:student.name I would like to create a field "student" and this field contains the information such as name. In Kibana "discover" I can see the logs, but it shows me that "student.name" is not mapped (Unmapped fields). How do I get that? I described the field under Index Management -\> Index Templates -\> Settings (from template). Still it doesn't work... ☹

---

<div class="post-metadata">

### Author: ![cperzrt10](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cperzrt10/32/116152_2.png) [@cperzrt10](https://discuss.elastic.co/u/cperzrt10)
#### Post date: [November 8, 2023, 7:41am UTC](https://discuss.elastic.co/t/how-can-i-filter-certain-information-from-the-logs/344293/9 "2023-11-08T07:41:48Z")

</div>

Hi. maibe somethin like this

```auto
filter {
  grok {
    id => "name school grok filter"
    match => { 'message' => '^.*name=\'%{WORD:name}\'.*school=\'%{WORD:school}\''}
  }
 mutate {
   # Renames the 'HOSTORIP' field to 'client_ip'
   rename => { "student" => "student.name" }
  }
}

```

[https://www.elastic.co/guide/en/logstash/current/plugins-filters-mutate.html#plugins-filters-mutate-rename](https://www.elastic.co/guide/en/logstash/current/plugins-filters-mutate.html#plugins-filters-mutate-rename)

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [December 6, 2023, 7:41am UTC](https://discuss.elastic.co/t/how-can-i-filter-certain-information-from-the-logs/344293/10 "2023-12-06T07:41:49Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
