# Logstash doesn't parse message into separate parts

**URL:** <https://discuss.elastic.co/t/logstash-doesnt-parse-message-into-separate-parts/369958>\
**Category:** Logstash\
**Created:** [November 3, 2024, 8:50am UTC](https://discuss.elastic.co/t/logstash-doesnt-parse-message-into-separate-parts/369958 "2024-11-03T08:50:23Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![Xavier\_24314](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/xavier_24314/32/138933_2.png) [@Xavier\_24314](https://discuss.elastic.co/u/Xavier_24314)\
**Post date:** [November 3, 2024, 8:50am UTC](https://discuss.elastic.co/t/logstash-doesnt-parse-message-into-separate-parts/369958/1 "2024-11-03T08:50:23Z")

</div>

I'm trying to parse my error.log file through logstash into elasticsearch but the message shows up in kibana as a whole without being split. I am not sure if my logstash conf file is wrong or if the error is elsewhere.  
logstash .conf file

```auto
input {
  beats {
    port => 5044
  }
}
filter {
  grok {
    match => {
    "message" => "\[%{DAY:day} %{MONTH:month} %{MONTHDAY:monthday} %{TIME:time} %{YEAR:year}\] \[%{WORD:log_type}:%{LOGLEVEL:log_level}\] \[pid %{NUMBER:pid}\] (?:\[client %{IP:client_ip}:%{NUMBER:client_port}\] ){0,1}%{GREEDYDATA:message}"
    }
  }
}
output {
  elasticsearch {
    hosts => ["localhost:9200"]
    user => "elastic"
    password => "password"
  }
}

```

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [November 3, 2024, 10:25am UTC](https://discuss.elastic.co/t/logstash-doesnt-parse-message-into-separate-parts/369958/2 "2024-11-03T10:25:38Z")

</div>

Please provide examples of the messages you are trying to parse. Do the documents in elasticsearch have a \_grokparsefailure tag?

---

<div class="post-metadata">

**Author:** ![Xavier\_24314](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/xavier_24314/32/138933_2.png) [@Xavier\_24314](https://discuss.elastic.co/u/Xavier_24314)\
**Post date:** [November 3, 2024, 11:56am UTC](https://discuss.elastic.co/t/logstash-doesnt-parse-message-into-separate-parts/369958/3 "2024-11-03T11:56:33Z")

</div>

The messages are currently going through the filebeat-\* index instead of logstash. I previously connected filebeat directly to elasticsearch.  
Message example

```auto
[Sun Nov 03 18:31:28.085085 2024] [security2:error] [pid 543501] [client 127.0.0.1:43240] ModSecurity: Warning. Invalid URL Encoding: Non-hexadecimal digits used at TX:0. [file "/usr/share/modsecurity-crs/rules/REQUEST-920-PROTOCOL-ENFORCEMENT.conf"] [line "427"] [id "920221"] [msg "URL Encoding Abuse Attack Attempt"] [data ""] [severity "CRITICAL"] [ver "OWASP_CRS/4.9.0-dev"] [tag "application-multi"] [tag "language-multi"] [tag "platform-multi"] [tag "attack-protocol"] [tag "paranoia-level/1"] [tag "OWASP_CRS"] [tag "capec/1000/255/153/267/72"] [hostname "localhost"] [uri "/DVWA/vulnerabilities/sqli/"] [unique_id "ZydRALr8WrdfkM5o6WVj4AAAAAI"], referer: http://localhost/DVWA/vulnerabilities/sqli/

```

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [November 3, 2024, 1:11pm UTC](https://discuss.elastic.co/t/logstash-doesnt-parse-message-into-separate-parts/369958/4 "2024-11-03T13:11:09Z")

</div>

That sounds like your events are still going directly from filebeat to elasticsearch.

The grok works if the message goes through that pipeline. I would suggest adding `overwrite => ["message"]`. If you do not then [message] will be an array, with one entry being the original log line, and the second entry being whatever matches the trailing GREEDYDATA in the grok pattern. Having an array like that is unlikely to be useful.

If you want to parse out more of [message] then I would suggest using ruby

```
    ruby {
        code => '
            matches = event.get("message")&.scan(/ \[(\w+) "([^"]+)"\]/)
            matches.each { |k, v|
                newK = "[stuff][#{k}]"
                if event.include?(newK)
                    a = Array(event.get(newK))
                    a << v
                    event.set(newK, a)
                else
                    event.set(newK, v)
                end
            }
        '
    }

```

which will produce

```
      "stuff" => {
           "id" => "920221",
     "severity" => "CRITICAL",
          "msg" => "URL Encoding Abuse Attack Attempt",
          "tag" => [
        [0] "application-multi",
        [1] "language-multi",
        [2] "platform-multi",

```

etc.

---

<div class="post-metadata">

**Author:** ![Xavier\_24314](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/xavier_24314/32/138933_2.png) [@Xavier\_24314](https://discuss.elastic.co/u/Xavier_24314)\
**Post date:** [November 3, 2024, 4:20pm UTC](https://discuss.elastic.co/t/logstash-doesnt-parse-message-into-separate-parts/369958/5 "2024-11-03T16:20:58Z")

</div>

Does this mean that there is an error in my logstash config that's why the logs are bypassing logstash?

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [November 3, 2024, 4:56pm UTC](https://discuss.elastic.co/t/logstash-doesnt-parse-message-into-separate-parts/369958/6 "2024-11-03T16:56:12Z")

</div>

No, filebeat will send logs to either logstash or elasticsearch. If it is sending them to elasticsearch it is because the configuration is telling it to do so. You may not be running the filebeat configuration you think you are.