# Issue with logstash then elasticsearch : message (from grok) not creating fields?

**URL:** <https://discuss.elastic.co/t/issue-with-logstash-then-elasticsearch-message-from-grok-not-creating-fields/297555>\
**Category:** Logstash\
**Created:** [February 17, 2022, 11:38pm UTC](https://discuss.elastic.co/t/issue-with-logstash-then-elasticsearch-message-from-grok-not-creating-fields/297555 "2022-02-17T23:38:44Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![ledufakademy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ledufakademy/32/101973_2.png) [@ledufakademy](https://discuss.elastic.co/u/ledufakademy)\
**Post date:** [February 17, 2022, 11:38pm UTC](https://discuss.elastic.co/t/issue-with-logstash-then-elasticsearch-message-from-grok-not-creating-fields/297555/1 "2022-02-17T23:38:44Z")

</div>

First i 'm trying to add my HAPROXY ALOHA 13.5 LTS (we cannot install nothing on this appliance, so FileBeat ... not for us , but perhaps i'm wrong ?) syslog to Elasticsearch (Kiban gui).  
So i decide to send syslog data , over UDP port 22514.

I have just installing ELK with 8.0 release, Debian 11 (full updated)

Then configure my logstach like that :

```auto

cat /etc/logstash/conf.d/haproxy.conf 

input {
  tcp {
    port => 22514
# type => "haproxy"
  }
  udp {
    port => 22514
# type => "haproxy"
  }
}

filter {
# if [type] == "haproxy" {
    grok {
      patterns_dir => "/etc/logstash/patterns"
      match => { "message" => "%{DATE_HAPROXY:haproxy_date}%{SPACE}*%{TIME_HAPROXY:haproxy_time}%{SPACE}*%{LOGLEVEL:log-level}%{SPACE}*%{IPORHOST:haproxy_server}%{SPACE}*%{SYSLOGPROG}%{SPACE}*%{PROG:syslog_service}%{SPACE}*%{IP:client_ip}:%{INT:client_port}%{SPACE}*\[%{HAPROXYDATE:accept_date}\] %{NOTSPACE:frontend_name} %{NOTSPACE:backend_name}/%{NOTSPACE:server_name} %{INT:time_request}/%{INT:time_queue}/%{INT:time_backend_connect}/%{INT:time_backend_response}/%{NOTSPACE:time_duration} %{INT:http_status_code} %{NOTSPACE:bytes_read} %{DATA:captured_request_cookie} %{DATA:captured_response_cookie} %{NOTSPACE:termination_state} %{INT:actconn}/%{INT:feconn}/%{INT:beconn}/%{INT:srvconn}/%{NOTSPACE:retries} %{INT:srv_queue}/%{INT:backend_queue} (\{%{HAPROXYCAPTUREDREQUESTHEADERS}\})?( )?(\{%{HAPROXYCAPTUREDRESPONSEHEADERS}\})?( )?\"(<BADREQ>|(%{WORD:http_verb} (%{URIPROTO:http_proto}://)?(?:%{USER:http_user}(?::[^@]*)?@)?(?:%{URIHOST:http_host})?(?:%{URIPATHPARAM:http_request})?( HTTP/%{NUMBER:http_version})?))?\""}
    }
  }
#}

output {
  elasticsearch {
    hosts => "127.0.0.1:9200"
    index => "haproxy-trafic-%{+YYYY.MM.dd}"
    user => "elastic"
    password => "xxxxxxxxxxxxxxxx"
    ssl => true
    ssl_certificate_verification => false
  }
}

Note : i'm using %{SPACE}* , because i don't know how many space i can found in the log :-( 

```

**Note** : the grok work perfectly under Grok Debugger (kibana) ... all fields properly displayed

Here is my patterns

```auto
cat /etc/logstash/patterns/haproxy 

DATE_HAPROXY %{YEAR}-%{MONTHNUM}-%{MONTHDAY}
TIME_HAPROXY %{HOUR}:%{MINUTE}:%{SECOND}

```

**the problem is that when i want to Discover my index, then choose display "JSON", i only see a fields called "message" with all the data ... this not the good at all !!! lol.**

(i notice :this thing in Elasticsearch.log : " GrokProcessor [hostanme] regular expression has redundant nested repeat operator \* ")

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [February 17, 2022, 11:50pm UTC](https://discuss.elastic.co/t/issue-with-logstash-then-elasticsearch-message-from-grok-not-creating-fields/297555/2 "2022-02-17T23:50:27Z")

</div>

What does a message look like if you use

```
output { stdout { codec => rubydebug } }

```

BTW, the SPACE pattern in grok is `\s*`, so you do not need the \* after %{SPACE}.

Also, do you really want to do this in logstash, or using a grok processor in an ingest pipeline in Elasticsearch?

---

<div class="post-metadata">

**Author:** ![ledufakademy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ledufakademy/32/101973_2.png) [@ledufakademy](https://discuss.elastic.co/u/ledufakademy)\
**Post date:** [February 18, 2022, 1:48pm UTC](https://discuss.elastic.co/t/issue-with-logstash-then-elasticsearch-message-from-grok-not-creating-fields/297555/3 "2022-02-18T13:48:15Z")

</div>

Hello badger,  
thank you for those answer.  
the same as i can see in Elasticsearch / Kibana.  
"message" part is the same fields ....  
note : i put it in /tmp/my\_output\_file.txt , because console not showing something (i don't know where i must see result ..)

---

<div class="post-metadata">

**Author:** ![ledufakademy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ledufakademy/32/101973_2.png) [@ledufakademy](https://discuss.elastic.co/u/ledufakademy)\
**Post date:** [February 18, 2022, 3:49pm UTC](https://discuss.elastic.co/t/issue-with-logstash-then-elasticsearch-message-from-grok-not-creating-fields/297555/4 "2022-02-18T15:49:00Z")

</div>

same issue while viewing output via local file 🙂

```auto
<134>haproxy[14880]: 192.134.152.190:25145 [18/Feb/2022:14:59:29.400] web-service-https~ Internet-prod/back1.local.intra 0/0/0/17/48 200 89837 - - ---- 272/272/1/1/0 0/0 {www.toto.fr} \"GET /core/assets/vendor/jquery/jquery.min.js?v=3.5.1 HTTP/1.1\" 

```

Version with no custom patterns ...

```auto
%{IP:client_ip}:%{INT:client_port} \[%{HAPROXYDATE:accept_date}\] %{NOTSPACE:frontend_name} %{NOTSPACE:backend_name}/%{NOTSPACE:server_name} %{INT:time_request}/%{INT:time_queue}/%{INT:time_backend_connect}/%{INT:time_backend_response}/%{NOTSPACE:time_duration} %{INT:http_status_code} %{NOTSPACE:bytes_read} %{DATA:captured_request_cookie} %{DATA:captured_response_cookie} %{NOTSPACE:termination_state} %{INT:actconn}/%{INT:feconn}/%{INT:beconn}/%{INT:srvconn}/%{NOTSPACE:retries} %{INT:srv_queue}/%{INT:backend_queue} \{%{HAPROXYCAPTUREDREQUESTHEADERS}\} \\"%{WORD:Method} %{URIPATHPARAM:request} HTTP/%{NUMBER:http_version}\\" 

```

This working under ALL grok debugger (elk, online etc) , but not for logstatsh.

i do not understand logstach grok ...

---

<div class="post-metadata">

**Author:** ![ledufakademy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ledufakademy/32/101973_2.png) [@ledufakademy](https://discuss.elastic.co/u/ledufakademy)\
**Post date:** [February 18, 2022, 4:16pm UTC](https://discuss.elastic.co/t/issue-with-logstash-then-elasticsearch-message-from-grok-not-creating-fields/297555/5 "2022-02-18T16:16:55Z")

</div>

Always got : "\_grokparsefailure" in ELK kibana (Discover)  
while grok is ok.

 ![Capture d’écran du 2022-02-18 17-19-52](https://us1.discourse-cdn.com/elastic/original/3X/c/8/c81030ee26dd63bc20d9d6f2a179a476af8fbffa.png)

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [February 18, 2022, 5:54pm UTC](https://discuss.elastic.co/t/issue-with-logstash-then-elasticsearch-message-from-grok-not-creating-fields/297555/6 "2022-02-18T17:54:49Z")

</div>

This

```
input { generator { count => 1 lines => ['<134>haproxy[14880]: 192.134.152.190:25145 [18/Feb/2022:14:59:29.400] web-service-https~ Internet-prod/back1.local.intra 0/0/0/17/48 200 89837 - - ---- 272/272/1/1/0 0/0 {www.toto.fr} "GET /core/assets/vendor/jquery/jquery.min.js?v=3.5.1 HTTP/1.1"' ] } }
filter {
    grok { match => { "message" => '%{IP:client_ip}:%{INT:client_port} \[%{HAPROXYDATE:accept_date}\] %{NOTSPACE:frontend_name} %{NOTSPACE:backend_name}/%{NOTSPACE:server_name} %{INT:time_request}/%{INT:time_queue}/%{INT:time_backend_connect}/%{INT:time_backend_response}/%{NOTSPACE:time_duration} %{INT:http_status_code} %{NOTSPACE:bytes_read} %{DATA:captured_request_cookie} %{DATA:captured_response_cookie} %{NOTSPACE:termination_state} %{INT:actconn}/%{INT:feconn}/%{INT:beconn}/%{INT:srvconn}/%{NOTSPACE:retries} %{INT:srv_queue}/%{INT:backend_queue} {%{HAPROXYCAPTUREDREQUESTHEADERS}} "%{WORD:Method} %{URIPATHPARAM:request} HTTP/%{NUMBER:http_version}"' } }
}

```

works just fine in logstash, which suggests the problem is with the escapes.

---

<div class="post-metadata">

**Author:** ![ledufakademy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ledufakademy/32/101973_2.png) [@ledufakademy](https://discuss.elastic.co/u/ledufakademy)\
**Post date:** [February 18, 2022, 6:02pm UTC](https://discuss.elastic.co/t/issue-with-logstash-then-elasticsearch-message-from-grok-not-creating-fields/297555/7 "2022-02-18T18:02:30Z")

</div>

yes, sure.  
But when output to file , i have discovered that in /tmp/my\_output\_file.txt (tail -f -n 100 the file)  
there was

```auto
\"GET /

```

Not

```auto
"GET /

```

so need to protect it :

```auto
\\"%{WORD:Method}

```

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [February 18, 2022, 6:14pm UTC](https://discuss.elastic.co/t/issue-with-logstash-then-elasticsearch-message-from-grok-not-creating-fields/297555/8 "2022-02-18T18:14:53Z")

</div>

How is the file output configured? It may be escaping the quotes itself.

---

<div class="post-metadata">

**Author:** ![ledufakademy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ledufakademy/32/101973_2.png) [@ledufakademy](https://discuss.elastic.co/u/ledufakademy)\
**Post date:** [February 18, 2022, 6:24pm UTC](https://discuss.elastic.co/t/issue-with-logstash-then-elasticsearch-message-from-grok-not-creating-fields/297555/9 "2022-02-18T18:24:56Z")

</div>

it's work , that was just because i put " (double quote) just replace with ' (simple quote)

```auto
match => { "message" => "%{ xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx "}

```

Then by 🙂

```auto
match => { "message" => '%{xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx '}

```

that 's it. (we need to escape , because syslog from HAPROXY ALOHA , have the

```auto
\"GET)

```

thank u badger.

But : do you think we must work with FileBeat ? (install it on ELK server), on ALOHA HAPROXY we can not install fileBeat.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [March 18, 2022, 6:25pm UTC](https://discuss.elastic.co/t/issue-with-logstash-then-elasticsearch-message-from-grok-not-creating-fields/297555/10 "2022-03-18T18:25:32Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.