Elasticsearch Query DSL Filter Including Middle Occurrences

[Dokh_Ahmed (Add123454321)]

Hello
I am facing an issue with Elastisearch Query DSL while using a prefix filter for the "log_message" field.
The goal is to display logs where the "log_message" field has a prefix of "Started". However, the filter is also including lines where "Started" appears in the middle of the message, not just at the beginning.

• Here's my query:

{
  "prefix": {
    "log_message": "started"
  }
}

• Here's my log message:

2023-11-20 10:49:34.445 | INFO | [restartedMain] --- 14348 | o.a.c.impl.engine.AbstractCamelContext | | Started SEND_AGENT_REF_TO_S3<2> (direct://agentRefFromSiCas2-3)

PS: I have an espace between '| |' and 'Started'.

• Her's my logstash Pattern:

%{TIMESTAMP_ISO8601:timestamp} |%{SPACE}%{LOGLEVEL:log_level}%{SPACE}|%{SPACE}[%{DATA:thread}]%{SPACE}---%{SPACE}%{NUMBER:process_id}%{SPACE}|%{SPACE}%{DATA:class}%{SPACE}| |%{SPACE}%{GREEDYDATA:log_message}(\r|\n)?

[yago82]

Hi,

You can adjust your Logstash grok pattern to split the "log_message" field into two separate fields: one for the "| |" part and one for the actual log message. Here's an example of how you can adjust your Logstash pattern:

%{TIMESTAMP_ISO8601:timestamp} \| %{LOGLEVEL:log_level} \| \[%{WORD:thread}] --- %{NUMBER:process_id} \| %{NOTSPACE:class} \| \| %{WORD:log_prefix} %{GREEDYDATA:log_message}

2023-11-20 10:49:34.445	INFO	[restartedMain] --- 14348	o.a.c.impl.engine.AbstractCamelContext		Started SEND_AGENT_REF_TO_S3<2> (direct://agentRefFromSiCas2-3)
MATCHED
timestamp	2023-11-20·10:49:34.445
log_message	SEND_AGENT_REF_TO_S3<2>·(direct://agentRefFromSiCas2-3)
process_id	14348
log_level	INFO
thread	restartedMain
class	o.a.c.impl.engine.AbstractCamelContext
log_prefix	Started

This will return documents where the "log_prefix" field starts with "Started"