Filebeat is not sending any logs to logstash please help urgent!

Hello, i have installed elk in an aws ec2 instance ubuntu and filebeat in another ubuntu instance.
Below is my filebeat.yml file

cat /etc/filebeat/filebeat.yml 
###################### Filebeat Configuration Example #########################

# This file is an example configuration file highlighting only the most common
# options. The filebeat.reference.yml file from the same directory contains all the
# supported options with more comments. You can use it as a reference.
#
# You can find the full configuration reference here:
# https://www.elastic.co/guide/en/beats/filebeat/index.html

# For more available modules and options, please see the filebeat.reference.yml sample
# configuration file.

# ============================== Filebeat inputs ===============================

filebeat.inputs:

# Each - is an input. Most options can be set at the input level, so
# you can use different inputs for various configurations.
# Below are the input-specific configurations.

# filestream is an input for collecting log messages from files.
- type: log 

  # Unique ID among all inputs, an ID is required.
  id: orocrm

  # Change to true to enable this input configuration.
  enabled: true

  # Paths that should be crawled and fetched. Glob based paths.
  paths:
    - /orocrmprod.log
    #- c:\programdata\elasticsearch\logs\*

  # Exclude lines. A list of regular expressions to match. It drops the lines that are
  # matching any regular expression from the list.
  # Line filtering happens after the parsers pipeline. If you would like to filter lines
  # before parsers, use include_message parser.
  #exclude_lines: ['^DBG']

  # Include lines. A list of regular expressions to match. It exports the lines that are
  # matching any regular expression from the list.
  # Line filtering happens after the parsers pipeline. If you would like to filter lines
  # before parsers, use include_message parser.
  #include_lines: ['^ERR', '^WARN']

  # Exclude files. A list of regular expressions to match. Filebeat drops the files that
  # are matching any regular expression from the list. By default, no files are dropped.
  #prospector.scanner.exclude_files: ['.gz$']

  # Optional additional fields. These fields can be freely picked
  # to add additional information to the crawled log files for filtering
  #fields:
  #  level: debug
  #  review: 1

# ============================== Filebeat modules ==============================

filebeat.config.modules:
  # Glob pattern for configuration loading
  path: ${path.config}/modules.d/*.yml

  # Set to true to enable config reloading
  reload.enabled: false

  # Period on which files under path should be checked for changes
  #reload.period: 10s

# ======================= Elasticsearch template setting =======================

setup.template.settings:
  index.number_of_shards: 1
  #index.codec: best_compression
  #_source.enabled: false


# ================================== General ===================================

# The name of the shipper that publishes the network data. It can be used to group
# all the transactions sent by a single shipper in the web interface.
#name:

# The tags of the shipper are included in their field with each
# transaction published.
#tags: ["service-X", "web-tier"]

# Optional fields that you can specify to add additional information to the
# output.
#fields:
#  env: staging

# ================================= Dashboards =================================
# These settings control loading the sample dashboards to the Kibana index. Loading
# the dashboards is disabled by default and can be enabled either by setting the
# options here or by using the `setup` command.
#setup.dashboards.enabled: false

# The URL from where to download the dashboard archive. By default, this URL
# has a value that is computed based on the Beat name and version. For released
# versions, this URL points to the dashboard archive on the artifacts.elastic.co
# website.
#setup.dashboards.url:

# =================================== Kibana ===================================

# Starting with Beats version 6.0.0, the dashboards are loaded via the Kibana API.
# This requires a Kibana endpoint configuration.
setup.kibana:

  # Kibana Host
  # Scheme and port can be left out and will be set to the default (http and 5601)
  # In case you specify and additional path, the scheme is required: http://localhost:5601/path
  # IPv6 addresses should always be defined as: https://[2001:db8::1]:5601
  #host: "localhost:5601"

  # Kibana Space ID
  # ID of the Kibana Space into which the dashboards should be loaded. By default,
  # the Default Space will be used.
  #space.id:

# =============================== Elastic Cloud ================================

# These settings simplify using Filebeat with the Elastic Cloud (https://cloud.elastic.co/).

# The cloud.id setting overwrites the `output.elasticsearch.hosts` and
# `setup.kibana.host` options.
# You can find the `cloud.id` in the Elastic Cloud web UI.
#cloud.id:

# The cloud.auth setting overwrites the `output.elasticsearch.username` and
# `output.elasticsearch.password` settings. The format is `<user>:<pass>`.
#cloud.auth:

# ================================== Outputs ===================================

# Configure what output to use when sending the data collected by the beat.

# ---------------------------- Elasticsearch Output ----------------------------
#output.elasticsearch:
  # Array of hosts to connect to.
  #hosts: ["localhost:9200"]

  # Performance preset - one of "balanced", "throughput", "scale",
  # "latency", or "custom".
  # preset: balanced

  # Protocol - either `http` (default) or `https`.
  #protocol: "https"

  # Authentication credentials - either API key or username/password.
  #api_key: "id:api_key"
  #username: "elastic"
  #password: "changeme"

# ------------------------------ Logstash Output -------------------------------
output.logstash:
  # The Logstash hosts
  hosts: ["172.31.34.24:5044"]

  # Optional SSL. By default is off.
  # List of root certificates for HTTPS server verifications
  #ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]

  # Certificate for SSL client authentication
  #ssl.certificate: "/etc/pki/client/cert.pem"

  # Client Certificate Key
  #ssl.key: "/etc/pki/client/cert.key"

# ================================= Processors =================================
processors:
  - add_host_metadata:
      when.not.contains.tags: forwarded
  - add_cloud_metadata: ~
  - add_docker_metadata: ~
  - add_kubernetes_metadata: ~

# ================================== Logging ===================================

# Sets log level. The default log level is info.
# Available log levels are: error, warning, info, debug
#logging.level: debug

# At debug level, you can selectively enable logging only for some components.
# To enable all selectors, use ["*"]. Examples of other selectors are "beat",
# "publisher", "service".
#logging.selectors: ["*"]

# ============================= X-Pack Monitoring ==============================
# Filebeat can export internal metrics to a central Elasticsearch monitoring
# cluster.  This requires xpack monitoring to be enabled in Elasticsearch.  The
# reporting is disabled by default.

# Set to true to enable the monitoring reporter.
#monitoring.enabled: false

# Sets the UUID of the Elasticsearch cluster under which monitoring data for this
# Filebeat instance will appear in the Stack Monitoring UI. If output.elasticsearch
# is enabled, the UUID is derived from the Elasticsearch cluster referenced by output.elasticsearch.
#monitoring.cluster_uuid:

# Uncomment to send the metrics to Elasticsearch. Most settings from the
# Elasticsearch outputs are accepted here as well.
# Note that the settings should point to your Elasticsearch *monitoring* cluster.
# Any setting that is not set is automatically inherited from the Elasticsearch
# output configuration, so if you have the Elasticsearch output configured such
# that it is pointing to your Elasticsearch monitoring cluster, you can simply
# uncomment the following line.
#monitoring.elasticsearch:

# ============================== Instrumentation ===============================

# Instrumentation support for the filebeat.
#instrumentation:
    # Set to true to enable instrumentation of filebeat.
    #enabled: false

    # Environment in which filebeat is running on (eg: staging, production, etc.)
    #environment: ""

    # APM Server hosts to report instrumentation results to.
    #hosts:
    #  - http://localhost:8200

    # API Key for the APM Server(s).
    # If api_key is set then secret_token will be ignored.
    #api_key:

    # Secret token for the APM Server(s).
    #secret_token:


# ================================= Migration ==================================

# This allows to enable 6.7 migration aliases
#migration.6_to_7.enabled: true

here is my logstash conf file. I have tried with the output to elasticsearch it was not showing anything in kibana so i have used
stdout {
codec => rubydebug
}
here where i got that it was not receiving any logs .

cat /etc/logstash/conf.d/oroprod.conf 
input {
  beats {
    port => 5044
  }
}
filter {
  grok {
    match => {
      "message" => "\[%{TIMESTAMP_ISO8601:timestamp}\]\[%{DATA:loglevel}\] %{GREEDYDATA:message}"
    }
  }
  grok {
    match => {
      "message" => "HOST: %{HOSTNAME:host} REQUEST URI: %{URIPATH:request_uri}"
    }
  }
}

output {
stdout {
    codec => rubydebug
  }
}

here is the output it got stuck there it was not printing anything error also

bin/logstash -f /etc/logstash/conf.d/oroprod.conf 
Using bundled JDK: /usr/share/logstash/jdk
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[WARN ] 2024-06-27 12:54:03.835 [main] runner - NOTICE: Running Logstash as superuser is not recommended and won't be allowed in the future. Set 'allow_superuser' to 'false' to avoid startup errors in future releases.
[INFO ] 2024-06-27 12:54:03.846 [main] runner - Starting Logstash {"logstash.version"=>"8.14.1", "jruby.version"=>"jruby 9.4.7.0 (3.1.4) 2024-04-29 597ff08ac1 OpenJDK 64-Bit Server VM 17.0.11+9 on 17.0.11+9 +indy +jit [x86_64-linux]"}
[INFO ] 2024-06-27 12:54:03.849 [main] runner - JVM bootstrap flags: [-Xms1g, -Xmx1g, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djruby.compile.invokedynamic=true, -XX:+HeapDumpOnOutOfMemoryError, -Djava.security.egd=file:/dev/urandom, -Dlog4j2.isThreadContextMapInheritable=true, -Dlogstash.jackson.stream-read-constraints.max-string-length=200000000, -Dlogstash.jackson.stream-read-constraints.max-number-length=10000, -Djruby.regexp.interruptible=true, -Djdk.io.File.enableADS=true, --add-exports=jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.parser=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED, --add-opens=java.base/java.security=ALL-UNNAMED, --add-opens=java.base/java.io=ALL-UNNAMED, --add-opens=java.base/java.nio.channels=ALL-UNNAMED, --add-opens=java.base/sun.nio.ch=ALL-UNNAMED, --add-opens=java.management/sun.management=ALL-UNNAMED, -Dio.netty.allocator.maxOrder=11]
[INFO ] 2024-06-27 12:54:03.851 [main] runner - Jackson default value override `logstash.jackson.stream-read-constraints.max-string-length` configured to `200000000`
[INFO ] 2024-06-27 12:54:03.851 [main] runner - Jackson default value override `logstash.jackson.stream-read-constraints.max-number-length` configured to `10000`
[WARN ] 2024-06-27 12:54:04.025 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified
[INFO ] 2024-06-27 12:54:04.604 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600, :ssl_enabled=>false}
[INFO ] 2024-06-27 12:54:04.922 [Converge PipelineAction::Create<main>] Reflections - Reflections took 109 ms to scan 1 urls, producing 132 keys and 468 values
[INFO ] 2024-06-27 12:54:05.345 [Converge PipelineAction::Create<main>] javapipeline - Pipeline `main` is configured with `pipeline.ecs_compatibility: v8` setting. All plugins in this pipeline will default to `ecs_compatibility => v8` unless explicitly configured otherwise.
[WARN ] 2024-06-27 12:54:05.368 [[main]-pipeline-manager] grok - ECS v8 support is a preview of the unreleased ECS v8, and uses the v1 patterns. When Version 8 of the Elastic Common Schema becomes available, this plugin will need to be updated
[WARN ] 2024-06-27 12:54:05.475 [[main]-pipeline-manager] grok - ECS v8 support is a preview of the unreleased ECS v8, and uses the v1 patterns. When Version 8 of the Elastic Common Schema becomes available, this plugin will need to be updated
[INFO ] 2024-06-27 12:54:05.513 [[main]-pipeline-manager] javapipeline - Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>250, "pipeline.sources"=>["/etc/logstash/conf.d/oroprod.conf"], :thread=>"#<Thread:0x73af94cb /usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:134 run>"}
[INFO ] 2024-06-27 12:54:05.973 [[main]-pipeline-manager] javapipeline - Pipeline Java execution initialization time {"seconds"=>0.46}
[INFO ] 2024-06-27 12:54:05.978 [[main]-pipeline-manager] beats - Starting input listener {:address=>"0.0.0.0:5044"}
[INFO ] 2024-06-27 12:54:05.984 [[main]-pipeline-manager] javapipeline - Pipeline started {"pipeline.id"=>"main"}
[INFO ] 2024-06-27 12:54:06.001 [Agent thread] agent - Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[INFO ] 2024-06-27 12:54:06.055 [[main]<beats] Server - Starting server on port: 5044

below is my log entry

[2024-06-26 06:56:53][][ HOST: distqa.izdemosite.com REQUEST URI: /user/login] app.CRITICAL: Error Occured while getting visit report images:Doctrine\ORM\Query\QueryException: SELECT mf.mediaFileId as id, CONCAT(mf.rootPath, mf.mediaFileName) as url FROM IZMOMediaLibraryBundle:MediaFile mf  WHERE mf.mediaFileId =  in /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/QueryException.php:41 Stack trace: #0 /websites/diost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(448): Doctrine\ORM\Query\QueryException::dqlError('SELECT mf.media...') #1 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(2649): Doctrine\ORM\Query\Parser->syntaxError('Literal') #2 /websites/disst/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(2837):pplication/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(2739): Doctrine\ORM\Query\Parser->ArithmeticFactor() #5 /websites/distrigoboost/crm-apt/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/AbstractQuery.php(924): Doctrine\ORM\AbstractQuery->exectrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/AbstractQuery.php(804): Doctrine\ORM\AbstractQuery->execute(NULL, NULL) #23 /websites/distrigoboost/crm-application/src/IZMO/MediaLibraryBundle/Entity/Repository/MediaFileRepository.php(128): Doctrine\ORM\AbstractQuery->getSingleResult() #latform/src/Oro/Bundle/UserBundle/Controller/SecurityController.php(99): IZMO\MediaLibraryBundle\Provider\MediaLibrary->getUrlOnLoginPage(NULL) #26 /websites/dist/crm-application/vendor/oro/platform/src/Oro/Bundle/UserBundle/Controller/SecurityController.php(32): Oro\Bundle\UserBundle\Controller\SecurityController->getBadgesInfo() #27 [internal function]: Oro\Bundle\UserBundle\Controller\SecurityController->loginAction() #28 /websites/dist/crm-application/app/bootstrap.php.cache(3242): call_user_func_array(Array, Array) #29 /websites/diost/crm-application/app/bootstrap.php.cache(3201): Symfony\Component\HttpKernel\HttpKernel->handleRaw(Object(Symfony\Component\HttpFoundation\Request), 1) #30 /websites/dist/crm-application/app/bootstrap.php.cache(3355): Symfony\Component\HttpKernel\HttpKernel->handle(Object(Symfony\Component\HttpFoundation\Request), 1, true) #31 /websites/distrigoboost/crm-application/app/bootstrap.php.cache(2540): Symfony\Component\HttpKernel\DependencyInjection\ContainerAwareHttpKernel->handle(Object(Symfony\Component\HttpFoundation\Request), 1, true) #32 /websites/distrigoboost/crm-application/web/app.php(22): Symfony\Component\HttpKernel\Kernel->handle(Object(Symfony\Component\HttpFoundation\Request)) #33 {main} [] []

here in this log entry i no need all things to get parsed i just need the request uri and timstamp log.level that's it so that why i used grok pattern

\[%{TIMESTAMP_ISO8601:timestamp}\]\[\]\[ HOST: %{DATA:host} REQUEST URI: %{DATA:request_uri}\] %{WORD:source}\.%{LOGLEVEL:loglevel}: %{DATA:error_message} %{JAVACLASS:exception_class}: %{DATA:exception_message} in %{DATA:exception_location} Stack trace: %{GREEDYDATA:stack_trace} \[\] \[\]

its giving the correct output in grok debuger why its not displaying anything when i run logstash idk
and there us no filebeat logs in syslog it showing the metric information

Run tcpdump on the logstash host for that port, are you getting input?

thanks for the reply, here is the ans for tcpdump on the logstash host

sudo tcpdump -i any port 5044 -vvv
tcpdump: data link type LINUX_SLL2
tcpdump: listening on any, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 262144 bytes
^C
0 packets captured
1 packet received by filter
0 packets dropped by kernel
root@oroprod-elk:/# 

But im able to connect to logstash host from filebeat server

filebeat test output
logstash: 172.31.34.24:5044...
  connection...
    parse host... OK
    dns lookup... OK
    addresses: 172.31.34.24
    dial up... OK
  TLS... WARN secure connection disabled
  talk to server... OK

How did you install logstash?

If you installed with a package manager this is not the proper command. (It is incomplete)

bin/logstash -f /etc/logstash/conf.d/oroprod.conf 

Perhaps take a look at this post

thanks for your reply i have installed the elk stack using the debain commands from the document i have tried moving logstash.yml file to /usr/share/logsatsh/config and gave permission to /var/lib/logstash and all hierarchy But still idk im not able to see anything

parsing logs can be done through the other way also right using custom integration by that i have sent logs in elasticsearch But it storing whole log entry in message field
IDK how to use the custom log integration according to our need i mean how to parse the logs using custom log integration
please explain!!!!

That will not help, put it back...

Did you try the command I shared in the post

/usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/oroprod.conf -r --path.settings /etc/logstash

the output for the command `/usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/oroprod.conf -r --path.settings /etc/logstash

/usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/orocrmprod.conf -r --path.settings /etc/logstash
Using bundled JDK: /usr/share/logstash/jdk
/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/concurrent-ruby-1.1.9/lib/concurrent-ruby/concurrent/executor/java_thread_pool_executor.rb:13: warning: method redefined; discarding old to_int
/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/concurrent-ruby-1.1.9/lib/concurrent-ruby/concurrent/executor/java_thread_pool_executor.rb:13: warning: method redefined; discarding old to_f
Sending Logstash logs to /var/log/logstash which is now configured via log4j2.properties
[2024-07-03T05:50:27,637][INFO ][logstash.runner          ] Log4j configuration path used is: /etc/logstash/log4j2.properties
[2024-07-03T05:50:27,640][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"8.13.4", "jruby.version"=>"jruby 9.4.5.0 (3.1.4) 2023-11-02 1abae2700f OpenJDK 64-Bit Server VM 17.0.11+9 on 17.0.11+9 +indy +jit [x86_64-linux]"}
[2024-07-03T05:50:27,643][INFO ][logstash.runner          ] JVM bootstrap flags: [-Xms1g, -Xmx1g, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djruby.compile.invokedynamic=true, -XX:+HeapDumpOnOutOfMemoryError, -Djava.security.egd=file:/dev/urandom, -Dlog4j2.isThreadContextMapInheritable=true, -Dlogstash.jackson.stream-read-constraints.max-string-length=200000000, -Dlogstash.jackson.stream-read-constraints.max-number-length=10000, -Djruby.regexp.interruptible=true, -Djdk.io.File.enableADS=true, --add-exports=jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.parser=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED, --add-opens=java.base/java.security=ALL-UNNAMED, --add-opens=java.base/java.io=ALL-UNNAMED, --add-opens=java.base/java.nio.channels=ALL-UNNAMED, --add-opens=java.base/sun.nio.ch=ALL-UNNAMED, --add-opens=java.management/sun.management=ALL-UNNAMED, -Dio.netty.allocator.maxOrder=11]
[2024-07-03T05:50:27,644][INFO ][logstash.runner          ] Jackson default value override `logstash.jackson.stream-read-constraints.max-string-length` configured to `200000000`
[2024-07-03T05:50:27,644][INFO ][logstash.runner          ] Jackson default value override `logstash.jackson.stream-read-constraints.max-number-length` configured to `10000`
[2024-07-03T05:50:27,798][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2024-07-03T05:50:28,234][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600, :ssl_enabled=>false}
[2024-07-03T05:50:28,506][INFO ][org.reflections.Reflections] Reflections took 91 ms to scan 1 urls, producing 132 keys and 468 values
/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/amazing_print-1.6.0/lib/amazing_print/formatter.rb:37: warning: previous definition of cast was here
[2024-07-03T05:50:29,250][INFO ][logstash.javapipeline    ] Pipeline `main` is configured with `pipeline.ecs_compatibility: v8` setting. All plugins in this pipeline will default to `ecs_compatibility => v8` unless explicitly configured otherwise.
[2024-07-03T05:50:29,263][WARN ][logstash.filters.grok    ][main] ECS v8 support is a preview of the unreleased ECS v8, and uses the v1 patterns. When Version 8 of the Elastic Common Schema becomes available, this plugin will need to be updated
[2024-07-03T05:50:29,338][INFO ][logstash.javapipeline    ][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>500, "pipeline.sources"=>["/etc/logstash/conf.d/orocrmprod.conf"], :thread=>"#<Thread:0x3cbb510f /usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:134 run>"}
[2024-07-03T05:50:29,861][INFO ][logstash.javapipeline    ][main] Pipeline Java execution initialization time {"seconds"=>0.52}
[2024-07-03T05:50:29,864][INFO ][logstash.inputs.beats    ][main] Starting input listener {:address=>"0.0.0.0:5044"}
[2024-07-03T05:50:29,871][INFO ][logstash.javapipeline    ][main] Pipeline started {"pipeline.id"=>"main"}
[2024-07-03T05:50:29,882][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2024-07-03T05:50:29,927][INFO ][org.logstash.beats.Server][main][7f0f85fa7b42843f3bfef4b709e2ade01002d5187b424b486a1f2a8e93c37901] Starting server on port: 5044

it was not displaying anything after this

Sir, side by side i'm trying to parse the logs using custom log integration . In that whole log entry was storing inside the message field idk how to parse it, i tried mentioning the below configuration in processor in custom log integration

[
  {
    "dissect": {
      "field": "message",
      "tokenizer": "[%{timestamp}] [%{loglevel}] [HOST: %{host} REQUEST URI: %{request_uri}] %{message}"
    }
  }
]

And i have tried mention the pipeline in the custom configuration,
But still it storing the whole log entry in message field can you help me how to parse the logs using custom log integration.

That means logstash is up an running and waiting for data...

Sorry I am confused you are also trying an ingest pipeline... please do not mix topics...

Can you back up

Are you trying to do

a) Filebeat -> Elasticsearch with Ingest Pipeline

or

b) Filebeat -> Logstash -> Elasticsearch

Also What version are you on?

You do not need to use logstash to parse logs

Working example
Filebeat -> Elasticsearch with Ingest Pipeline

file

[2024-06-26 06:56:53][][ HOST: distqa.izdemosite.com REQUEST URI: /user/login] app.CRITICAL: Error Occured while getting visit report images:Doctrine\ORM\Query\QueryException: SELECT mf.mediaFileId as id, CONCAT(mf.rootPath, mf.mediaFileName) as url FROM IZMOMediaLibraryBundle:MediaFile mf  WHERE mf.mediaFileId =  in /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/QueryException.php:41 Stack trace: #0 /websites/diost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(448): Doctrine\ORM\Query\QueryException::dqlError('SELECT mf.media...') #1 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(2649): Doctrine\ORM\Query\Parser->syntaxError('Literal') #2 /websites/disst/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(2837):pplication/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(2739): Doctrine\ORM\Query\Parser->ArithmeticFactor() #5 /websites/distrigoboost/crm-apt/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/AbstractQuery.php(924): Doctrine\ORM\AbstractQuery->exectrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/AbstractQuery.php(804): Doctrine\ORM\AbstractQuery->execute(NULL, NULL) #23 /websites/distrigoboost/crm-application/src/IZMO/MediaLibraryBundle/Entity/Repository/MediaFileRepository.php(128): Doctrine\ORM\AbstractQuery->getSingleResult() #latform/src/Oro/Bundle/UserBundle/Controller/SecurityController.php(99): IZMO\MediaLibraryBundle\Provider\MediaLibrary->getUrlOnLoginPage(NULL) #26 /websites/dist/crm-application/vendor/oro/platform/src/Oro/Bundle/UserBundle/Controller/SecurityController.php(32): Oro\Bundle\UserBundle\Controller\SecurityController->getBadgesInfo() #27 [internal function]: Oro\Bundle\UserBundle\Controller\SecurityController->loginAction() #28 /websites/dist/crm-application/app/bootstrap.php.cache(3242): call_user_func_array(Array, Array) #29 /websites/diost/crm-application/app/bootstrap.php.cache(3201): Symfony\Component\HttpKernel\HttpKernel->handleRaw(Object(Symfony\Component\HttpFoundation\Request), 1) #30 /websites/dist/crm-application/app/bootstrap.php.cache(3355): Symfony\Component\HttpKernel\HttpKernel->handle(Object(Symfony\Component\HttpFoundation\Request), 1, true) #31 /websites/distrigoboost/crm-application/app/bootstrap.php.cache(2540): Symfony\Component\HttpKernel\DependencyInjection\ContainerAwareHttpKernel->handle(Object(Symfony\Component\HttpFoundation\Request), 1, true) #32 /websites/distrigoboost/crm-application/web/app.php(22): Symfony\Component\HttpKernel\Kernel->handle(Object(Symfony\Component\HttpFoundation\Request)) #33 {main} [] []
[2024-06-26 06:56:53][][ HOST: distqa.izdemosite.com REQUEST URI: /user/login] app.CRITICAL: Error Occured while getting visit report images:Doctrine\ORM\Query\QueryException: SELECT mf.mediaFileId as id, CONCAT(mf.rootPath, mf.mediaFileName) as url FROM IZMOMediaLibraryBundle:MediaFile mf  WHERE mf.mediaFileId =  in /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/QueryException.php:41 Stack trace: #0 /websites/diost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(448): Doctrine\ORM\Query\QueryException::dqlError('SELECT mf.media...') #1 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(2649): Doctrine\ORM\Query\Parser->syntaxError('Literal') #2 /websites/disst/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(2837):pplication/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(2739): Doctrine\ORM\Query\Parser->ArithmeticFactor() #5 /websites/distrigoboost/crm-apt/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/AbstractQuery.php(924): Doctrine\ORM\AbstractQuery->exectrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/AbstractQuery.php(804): Doctrine\ORM\AbstractQuery->execute(NULL, NULL) #23 /websites/distrigoboost/crm-application/src/IZMO/MediaLibraryBundle/Entity/Repository/MediaFileRepository.php(128): Doctrine\ORM\AbstractQuery->getSingleResult() #latform/src/Oro/Bundle/UserBundle/Controller/SecurityController.php(99): IZMO\MediaLibraryBundle\Provider\MediaLibrary->getUrlOnLoginPage(NULL) #26 /websites/dist/crm-application/vendor/oro/platform/src/Oro/Bundle/UserBundle/Controller/SecurityController.php(32): Oro\Bundle\UserBundle\Controller\SecurityController->getBadgesInfo() #27 [internal function]: Oro\Bundle\UserBundle\Controller\SecurityController->loginAction() #28 /websites/dist/crm-application/app/bootstrap.php.cache(3242): call_user_func_array(Array, Array) #29 /websites/diost/crm-application/app/bootstrap.php.cache(3201): Symfony\Component\HttpKernel\HttpKernel->handleRaw(Object(Symfony\Component\HttpFoundation\Request), 1) #30 /websites/dist/crm-application/app/bootstrap.php.cache(3355): Symfony\Component\HttpKernel\HttpKernel->handle(Object(Symfony\Component\HttpFoundation\Request), 1, true) #31 /websites/distrigoboost/crm-application/app/bootstrap.php.cache(2540): Symfony\Component\HttpKernel\DependencyInjection\ContainerAwareHttpKernel->handle(Object(Symfony\Component\HttpFoundation\Request), 1, true) #32 /websites/distrigoboost/crm-application/web/app.php(22): Symfony\Component\HttpKernel\Kernel->handle(Object(Symfony\Component\HttpFoundation\Request)) #33 {main} [] []
[2024-06-26 06:56:53][][ HOST: distqa.izdemosite.com REQUEST URI: /user/login] app.CRITICAL: Error Occured while getting visit report images:Doctrine\ORM\Query\QueryException: SELECT mf.mediaFileId as id, CONCAT(mf.rootPath, mf.mediaFileName) as url FROM IZMOMediaLibraryBundle:MediaFile mf  WHERE mf.mediaFileId =  in /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/QueryException.php:41 Stack trace: #0 /websites/diost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(448): Doctrine\ORM\Query\QueryException::dqlError('SELECT mf.media...') #1 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(2649): Doctrine\ORM\Query\Parser->syntaxError('Literal') #2 /websites/disst/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(2837):pplication/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(2739): Doctrine\ORM\Query\Parser->ArithmeticFactor() #5 /websites/distrigoboost/crm-apt/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/AbstractQuery.php(924): Doctrine\ORM\AbstractQuery->exectrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/AbstractQuery.php(804): Doctrine\ORM\AbstractQuery->execute(NULL, NULL) #23 /websites/distrigoboost/crm-application/src/IZMO/MediaLibraryBundle/Entity/Repository/MediaFileRepository.php(128): Doctrine\ORM\AbstractQuery->getSingleResult() #latform/src/Oro/Bundle/UserBundle/Controller/SecurityController.php(99): IZMO\MediaLibraryBundle\Provider\MediaLibrary->getUrlOnLoginPage(NULL) #26 /websites/dist/crm-application/vendor/oro/platform/src/Oro/Bundle/UserBundle/Controller/SecurityController.php(32): Oro\Bundle\UserBundle\Controller\SecurityController->getBadgesInfo() #27 [internal function]: Oro\Bundle\UserBundle\Controller\SecurityController->loginAction() #28 /websites/dist/crm-application/app/bootstrap.php.cache(3242): call_user_func_array(Array, Array) #29 /websites/diost/crm-application/app/bootstrap.php.cache(3201): Symfony\Component\HttpKernel\HttpKernel->handleRaw(Object(Symfony\Component\HttpFoundation\Request), 1) #30 /websites/dist/crm-application/app/bootstrap.php.cache(3355): Symfony\Component\HttpKernel\HttpKernel->handle(Object(Symfony\Component\HttpFoundation\Request), 1, true) #31 /websites/distrigoboost/crm-application/app/bootstrap.php.cache(2540): Symfony\Component\HttpKernel\DependencyInjection\ContainerAwareHttpKernel->handle(Object(Symfony\Component\HttpFoundation\Request), 1, true) #32 /websites/distrigoboost/crm-application/web/app.php(22): Symfony\Component\HttpKernel\Kernel->handle(Object(Symfony\Component\HttpFoundation\Request)) #33 {main} [] []

Ingest Pipeline
Note very carefully you had extra spaces and you use host which is a reserved object so it needs to be host.name etc This Follows ECS

Ingest Pipeline

PUT _ingest/pipeline/discuss-test
{
  "processors": [
    {
      "dissect": {
        "field": "message",
        "pattern": "[%{timestamp}][%{loglevel}][HOST: %{host.name} REQUEST URI: %{request_uri}] %{message_detail}"
      }
    }
  ]
}

filbeat.yml



filebeat.inputs:

# filestream is an input for collecting log messages from files.
- type: filestream

  # Unique ID among all inputs, an ID is required.
  id: my-filestream-id

  # Change to true to enable this input configuration.
  enabled: true

  # Paths that should be crawled and fetched. Glob based paths.
  paths:
    - /Users/sbrown/workspace/sample-data/discuss/discuss-362161.log
 

filebeat.config.modules:
  # Glob pattern for configuration loading
  path: ${path.config}/modules.d/*.yml

  # Set to true to enable config reloading
  reload.enabled: false

setup.template.settings:
  index.number_of_shards: 1
  #index.codec: best_compression
  #_source.enabled: false


setup.kibana:

output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["localhost:9200"]
  pipeline: discuss-test
# RESULT
  },
  "hits": {
    "total": {
      "value": 3,
      "relation": "eq"
    },
    "max_score": 1,
    "hits": [
      {
        "_index": ".ds-filebeat-8.14.1-2024.07.03-000001",
        "_id": "dasueZABjwac8wPZA_wr",
        "_score": 1,
        "_ignored": [
          "message_detail"
        ],
        "_source": {
          "input": {
            "type": "filestream"
          },
          "agent": {
            "name": "hyperion",
            "id": "b9fc4918-2cd8-45d3-a288-68378ca4627b",
            "type": "filebeat",
            "ephemeral_id": "327b4def-6306-42c5-a82d-490fd88c11a8",
            "version": "8.14.1"
          },
          "@timestamp": "2024-07-03T15:19:26.745Z",
          "ecs": {
            "version": "8.0.0"
          },
          "log": {
            "file": {
              "inode": "114854596",
              "path": "/Users/sbrown/workspace/sample-data/discuss/discuss-362161.log",
              "device_id": "16777221"
            },
            "offset": 0
          },
          "loglevel": "",
          "host": {
            "name": " HOST: distqa.izdemosite.com"
          },
          "message": """[2024-06-26 06:56:53][][ HOST: distqa.izdemosite.com REQUEST URI: /user/login] app.CRITICAL: Error Occured while getting visit report images:Doctrine\ORM\Query\QueryException: SELECT mf.mediaFileId as id, CONCAT(mf.rootPath, mf.mediaFileName) as url FROM IZMOMediaLibraryBundle:MediaFile mf  WHERE mf.mediaFileId =  in /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/QueryException.php:41 Stack trace: #0 /websites/diost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(448): Doctrine\ORM\Query\QueryException::dqlError('SELECT mf.media...') #1 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(2649): Doctrine\ORM\Query\Parser->syntaxError('Literal') #2 /websites/disst/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(2837):pplication/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(2739): Doctrine\ORM\Query\Parser->ArithmeticFactor() #5 /websites/distrigoboost/crm-apt/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/AbstractQuery.php(924): Doctrine\ORM\AbstractQuery->exectrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/AbstractQuery.php(804): Doctrine\ORM\AbstractQuery->execute(NULL, NULL) #23 /websites/distrigoboost/crm-application/src/IZMO/MediaLibraryBundle/Entity/Repository/MediaFileRepository.php(128): Doctrine\ORM\AbstractQuery->getSingleResult() #latform/src/Oro/Bundle/UserBundle/Controller/SecurityController.php(99): IZMO\MediaLibraryBundle\Provider\MediaLibrary->getUrlOnLoginPage(NULL) #26 /websites/dist/crm-application/vendor/oro/platform/src/Oro/Bundle/UserBundle/Controller/SecurityController.php(32): Oro\Bundle\UserBundle\Controller\SecurityController->getBadgesInfo() #27 [internal function]: Oro\Bundle\UserBundle\Controller\SecurityController->loginAction() #28 /websites/dist/crm-application/app/bootstrap.php.cache(3242): call_user_func_array(Array, Array) #29 /websites/diost/crm-application/app/bootstrap.php.cache(3201): Symfony\Component\HttpKernel\HttpKernel->handleRaw(Object(Symfony\Component\HttpFoundation\Request), 1) #30 /websites/dist/crm-application/app/bootstrap.php.cache(3355): Symfony\Component\HttpKernel\HttpKernel->handle(Object(Symfony\Component\HttpFoundation\Request), 1, true) #31 /websites/distrigoboost/crm-application/app/bootstrap.php.cache(2540): Symfony\Component\HttpKernel\DependencyInjection\ContainerAwareHttpKernel->handle(Object(Symfony\Component\HttpFoundation\Request), 1, true) #32 /websites/distrigoboost/crm-application/web/app.php(22): Symfony\Component\HttpKernel\Kernel->handle(Object(Symfony\Component\HttpFoundation\Request)) #33 {main} [] []""",
          "request_uri": "/user/login",
          "message_detail": """app.CRITICAL: Error Occured while getting visit report images:Doctrine\ORM\Query\QueryException: SELECT mf.mediaFileId as id, CONCAT(mf.rootPath, mf.mediaFileName) as url FROM IZMOMediaLibraryBundle:MediaFile mf  WHERE mf.mediaFileId =  in /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/QueryException.php:41 Stack trace: #0 /websites/diost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(448): Doctrine\ORM\Query\QueryException::dqlError('SELECT mf.media...') #1 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(2649): Doctrine\ORM\Query\Parser->syntaxError('Literal') #2 /websites/disst/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(2837):pplication/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(2739): Doctrine\ORM\Query\Parser->ArithmeticFactor() #5 /websites/distrigoboost/crm-apt/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/AbstractQuery.php(924): Doctrine\ORM\AbstractQuery->exectrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/AbstractQuery.php(804): Doctrine\ORM\AbstractQuery->execute(NULL, NULL) #23 /websites/distrigoboost/crm-application/src/IZMO/MediaLibraryBundle/Entity/Repository/MediaFileRepository.php(128): Doctrine\ORM\AbstractQuery->getSingleResult() #latform/src/Oro/Bundle/UserBundle/Controller/SecurityController.php(99): IZMO\MediaLibraryBundle\Provider\MediaLibrary->getUrlOnLoginPage(NULL) #26 /websites/dist/crm-application/vendor/oro/platform/src/Oro/Bundle/UserBundle/Controller/SecurityController.php(32): Oro\Bundle\UserBundle\Controller\SecurityController->getBadgesInfo() #27 [internal function]: Oro\Bundle\UserBundle\Controller\SecurityController->loginAction() #28 /websites/dist/crm-application/app/bootstrap.php.cache(3242): call_user_func_array(Array, Array) #29 /websites/diost/crm-application/app/bootstrap.php.cache(3201): Symfony\Component\HttpKernel\HttpKernel->handleRaw(Object(Symfony\Component\HttpFoundation\Request), 1) #30 /websites/dist/crm-application/app/bootstrap.php.cache(3355): Symfony\Component\HttpKernel\HttpKernel->handle(Object(Symfony\Component\HttpFoundation\Request), 1, true) #31 /websites/distrigoboost/crm-application/app/bootstrap.php.cache(2540): Symfony\Component\HttpKernel\DependencyInjection\ContainerAwareHttpKernel->handle(Object(Symfony\Component\HttpFoundation\Request), 1, true) #32 /websites/distrigoboost/crm-application/web/app.php(22): Symfony\Component\HttpKernel\Kernel->handle(Object(Symfony\Component\HttpFoundation\Request)) #33 {main} [] []""",
          "timestamp": "2024-06-26 06:56:53"
        }
      }

version is 8.14.1
i have installed elk in another system and installed elastic agent in endpoint where i'm having the log i have attaches an agent policy to that elastic agent with system integration custom log in that custom log integration i have mentioned the path of the log file for this itself it started to collecting the log from the endpoint but it is storing the whole log entry in messaged field i want the below fields to view

"[%{timestamp}] [%{loglevel}] [HOST: %{host} REQUEST URI: %{request_uri}] %{message}"

For that i have tried adding custom configuration like pipeline: custom-pipeline in custom log integration, and in processor session like:

[
  {
    "dissect": {
      "field": "message",
      "tokenizer": "[%{timestamp}] [%{loglevel}] [HOST: %{host} REQUEST URI: %{request_uri}] %{message}"
    }
  }
]  

still the index which the custom log integration is creating is not adding those fields which i want. I want it to display in the below format

          "host": "distqa.izdemosite.com",
          "request_uri": "/user/login",
          "loglevel": "app.CRITICAL",
          "timestamp": "2024-07-02 06:56:57"
          "message": """app.CRITICAL: Error Occurred while getting visit report images:Doctrine\ORM\Query\QueryException: SELECT mf.mediaFileId as id, CONCAT(mf.rootPath, mf.mediaFileName) as url FROM IZdiaLibraryBundle:MediaFile mf WHERE mf.mediaFileId = '123' in /websites/distr/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/QueryException.php:41 Stack trace: #0 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(448): Doctrine\ORM\Query\QueryException::dqlError('SELECT mf.media...') #1 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(2649): Doctrine\ORM\Query\Parser->syntaxError('Literal') #2 /websites/distst/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/UserBundle/Controller/SecurityController.php(99): \HttpFoundation\Request), 1) #30 /websites/distt/crm-application/app/bootstrap.php.cache(3355): Symfony\Component\HttpKernel\HttpKernel->handle(Object(Symfony\Component\HttpFoundation\Request), 1, true) #31 /websites/dist/crm-application/app/bootstrap.php.cache(2540): Symfony\Component\HttpKernel\DependencyInjection\ContainerAwareHttpKernel->handle(Object(Symfony\Component\HttpFoundation\Request), 1, true) #32 /websites/dit/crm-application/web/app.php(22): Symfony\Component\HttpKernel\Kernel->handle(Object(Symfony\Component\HttpFoundation\Request)) #33 {main} [] []""",

the above format i got by the pipeline and index mapping. IDK how to use that pipeline and index in custom log integration to achieve the above format. i'm trying this from past 1 month Please suggest! to view the logs in the above format :disappointed:

This i have tried this sir the logs are getting displayed but only the first 3 are getting displayed in discover after that the log file is storing many logs but those logs are not getting displayed in discover

why it's not showing the remaining logs in discover tab :disappointed:

hi @evangelin

Well, in my post but I gave you the corrected ingest pipeline.

The pipeline you have in the post above will not work because it does not match your logs pattern There are extra spaces and also because of the other reasons such as the wrong field name host

Also, the dissect processor has no field called tokenizer The field is called pattern

I think you're confusing and I can understand why the dissect processor that's used with an ingest pipeline versus a dissect processor that's inside filebeat

I gave you the code for an ingest pipeline which would work with filebeat or the agent

Either way your pattern's not correct and the host field is not correct.

Why don't you try what I provided and see if that helps.

Also I'm confused because one post you're talking about filebeat and now you're talking about agent very difficult to help... We'd love to help, but it's hard when the subject is changing.

Sorry sir for the confusion. since it is not working i'm trying in many ways to make it work and main things is log entries in that log file are not same as each other
one log entry is

[2024-06-21 06:56:53][][ HOST: diboostqa.izdemosite.com REQUEST URI: /user/login] app.CRITICAL: Error Occured while getting visit report images:Doctrine\ORM\Query\QueryException: SELECT mf.mediaFileId as id, CONCAT(mf.rootPath, mf.mediaFileName) as url FROM IZMOMediaLibraryBundle:MediaFile mf  WHERE mf.mediaFileId =  in /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/QueryException.php:41 Stack trace: #0 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(448): Doctrine\ORM\Query\QueryException::dqlError('SELECT mf.media...') #1 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(2649): Doctrine\ORM\Query\Parser->syntaxError('Literal') #2 /websites/di=oost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(2837): Doctrine\ORM\Query\Parser->Literal() #3 /websites/dioboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(2771): Doctrine\ORM\Query\Parser->ArithmeticPrimary() #4 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(2739): Doctrine\ORM\Query\Parser->ArithmeticFactor() #5 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(2713): Doctrine\ORM\Query\Parser->ArithmeticTerm() #6 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(2700): Doctrine\ORM\Query\Parser->SimpleArithmeticExpression() #7 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(3043): Doctrine\ORM\Query\Parser->ArithmeticExpression() #8 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(2564): Doctrine\ORM\Query\Parser->ComparisonExpression() #9 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(2447): Doctrine\ORM\Query\Parser->SimpleConditionalExpression() #10 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(2423): Doctrine\ORM\Query\Parser->ConditionalPrimary() #11 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(2391): Doctrine\ORM\Query\Parser->ConditionalFactor() #12 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(2366): Doctrine\ORM\Query\Parser->ConditionalTerm() #13 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(1333): Doctrine\ORM\Query\Parser->ConditionalExpression() #14 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(876): Doctrine\ORM\Query\Parser->WhereClause() #15 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(843): Doctrine\ORM\Query\Parser->SelectStatement() #16 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(268): Doctrine\ORM\Query\Parser->QueryLanguage() #17 /websites/dboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(351): Doctrine\ORM\Query\Parser->getAST() #18 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query.php(281): Doctrine\ORM\Query\Parser->parse() #19 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query.php(293): Doctrine\ORM\Query->_parse() #20 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/AbstractQuery.php(969): Doctrine\ORM\Query->_doExecute() #21 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/AbstractQuery.php(924): Doctrine\ORM\AbstractQuery->executeIgnoreQueryCache(NULL, NULL) #22 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/AbstractQuery.php(804): Doctrine\ORM\AbstractQuery->execute(NULL, NULL) #23 /websites/distrigoboost/crm-application/src/IZMO/MediaLibraryBundle/Entity/Repository/MediaFileRepository.php(128): Doctrine\ORM\AbstractQuery->getSingleResult() #24 /websites/distrigoboost/crm-application/src/IZMO/MediaLibraryBundle/Provider/MediaLibrary.php(237): IZMO\MediaLibraryBundle\Entity\Repository\MediaFileRepository->getMediaFile(NULL) #25 /websites/distrigoboost/crm-application/vendor/oro/platform/src/Oro/Bundle/UserBundle/Controller/SecurityController.php(99): IZMO\MediaLibraryBundle\Provider\MediaLibrary->getUrlOnLoginPage(NULL) #26 /websites/distrigoboost/crm-application/vendor/oro/platform/src/Oro/Bundle/UserBundle/Controller/SecurityController.php(32): Oro\Bundle\UserBundle\Controller\SecurityController->getBadgesInfo() #27 [internal function]: Oro\Bundle\UserBundle\Controller\SecurityController->loginAction() #28 /websites/distrigoboost/crm-application/app/bootstrap.php.cache(3242): call_user_func_array(Array, Array) #29 /websites/distrigoboost/crm-application/app/bootstrap.php.cache(3201): Symfony\Component\HttpKernel\HttpKernel->handleRaw(Object(Symfony\Component\HttpFoundation\Request), 1) #30 /websites/distrigoboost/crm-application/app/bootstrap.php.cache(3355): Symfony\Component\HttpKernel\HttpKernel->handle(Object(Symfony\Component\HttpFoundation\Request), 1, true) #31 /websites/distrigoboost/crm-application/app/bootstrap.php.cache(2540): Symfony\Component\HttpKernel\DependencyInjection\ContainerAwareHttpKernel->handle(Object(Symfony\Component\HttpFoundation\Request), 1, true) #32 /websites/distrigoboost/crm-application/web/app.php(22): Symfony\Component\HttpKernel\Kernel->handle(Object(Symfony\Component\HttpFoundation\Request)) #33 {main}  Next Doctrine\ORM\Query\QueryException: [Syntax Error] line 0, col -1: Error: Expected Literal, got end of string. in /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/QueryException.php:52 Stack trace: #0 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(448): Doctrine\ORM\Query\QueryException::syntaxError('line 0, col -1:...', Object(Doctrine\ORM\Query\QueryException)) #1 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(2649): Doctrine\ORM\Query\Parser->syntaxError('Literal') #2 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(2837): Doctrine\ORM\Query\Parser->Literal() #3 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(2771): Doctrine\ORM\Query\Parser->ArithmeticPrimary() #4 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(2739): Doctrine\ORM\Query\Parser->ArithmeticFactor() #5 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(2713): Doctrine\ORM\Query\Parser->ArithmeticTerm() #6 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(2700): Doctrine\ORM\Query\Parser->SimpleArithmeticExpression() #7 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(3043): Doctrine\ORM\Query\Parser->ArithmeticExpression() #8 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(2564): Doctrine\ORM\Query\Parser->ComparisonExpression() #9 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(2447): Doctrine\ORM\Query\Parser->SimpleConditionalExpression() #10 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(2423): Doctrine\ORM\Query\Parser->ConditionalPrimary() #11 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(2391): Doctrine\ORM\Query\Parser->ConditionalFactor() #12 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(2366): Doctrine\ORM\Query\Parser->ConditionalTerm() #13 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(1333): Doctrine\ORM\Query\Parser->ConditionalExpression() #14 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(876): Doctrine\ORM\Query\Parser->WhereClause() #15 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(843): Doctrine\ORM\Query\Parser->SelectStatement() #16 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(268): Doctrine\ORM\Query\Parser->QueryLanguage() #17 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(351): Doctrine\ORM\Query\Parser->getAST() #18 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query.php(281): Doctrine\ORM\Query\Parser->parse() #19 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query.php(293): Doctrine\ORM\Query->_parse() #20 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/AbstractQuery.php(969): Doctrine\ORM\Query->_doExecute() #21 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/AbstractQuery.php(924): Doctrine\ORM\AbstractQuery->executeIgnoreQueryCache(NULL, NULL) #22 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/AbstractQuery.php(804): Doctrine\ORM\AbstractQuery->execute(NULL, NULL) #23 /websites/distrigoboost/crm-application/src/IZMO/MediaLibraryBundle/Entity/Repository/MediaFileRepository.php(128): Doctrine\ORM\AbstractQuery->getSingleResult() #24 /websites/distrigoboost/crm-application/src/IZMO/MediaLibraryBundle/Provider/MediaLibrary.php(237): IZMO\MediaLibraryBundle\Entity\Repository\MediaFileRepository->getMediaFile(NULL) #25 /websites/distrigoboost/crm-application/vendor/oro/platform/src/Oro/Bundle/UserBundle/Controller/SecurityController.php(99): IZMO\MediaLibraryBundle\Provider\MediaLibrary->getUrlOnLoginPage(NULL) #26 /websites/distrigoboost/crm-application/vendor/oro/platform/src/Oro/Bundle/UserBundle/Controller/SecurityController.php(32): Oro\Bundle\UserBundle\Controller\SecurityController->getBadgesInfo() #27 [internal function]: Oro\Bundle\UserBundle\Controller\SecurityController->loginAction() #28 /websites/distrigoboost/crm-application/app/bootstrap.php.cache(3242): call_user_func_array(Array, Array) #29 /websites/distrigoboost/crm-application/app/bootstrap.php.cache(3201): Symfony\Component\HttpKernel\HttpKernel->handleRaw(Object(Symfony\Component\HttpFoundation\Request), 1) #30 /websites/distrigoboost/crm-application/app/bootstrap.php.cache(3355): Symfony\Component\HttpKernel\HttpKernel->handle(Object(Symfony\Component\HttpFoundation\Request), 1, true) #31 /websites/distrigoboost/crm-application/app/bootstrap.php.cache(2540): Symfony\Component\HttpKernel\DependencyInjection\ContainerAwareHttpKernel->handle(Object(Symfony\Component\HttpFoundation\Request), 1, true) #32 /websites/distrigoboost/crm-application/web/app.php(22): Symfony\Component\HttpKernel\Kernel->handle(Object(Symfony\Component\HttpFoundation\Request)) #33 {main} [] []

and another log entry is

[2024-06-21 06:56:53] app.CRITICAL: Error Occured while getting visit report images:Doctrine\ORM\Query\QueryException: SELECT mf.mediaFileId as id, CONCAT(mf.rootPath, mf.mediaFileName) as url FROM IZMOMediaLibraryBundle:MediaFile mf  WHERE mf.mediaFileId =  in /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/QueryException.php:41 Stack trace: #0 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(448): Doctrine\ORM\Query\QueryException::dqlError('SELECT mf.media...') #1 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(2649): Doctrine\ORM\Query\Parser->syntaxError('Literal') #2 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(2837): Doctrine\ORM\Query\Parser->Literal() #3 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(2771): Doctrine\ORM\Query\Parser->ArithmeticPrimary() #4 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(2739): Doctrine\ORM\Query\Parser->ArithmeticFactor() #5 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(2713): Doctrine\ORM\Query\Parser->ArithmeticTerm() #6 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(2700): Doctrine\ORM\Query\Parser->SimpleArithmeticExpression() #7 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(3043): Doctrine\ORM\Query\Parser->ArithmeticExpression() #8 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(2564): Doctrine\ORM\Query\Parser->ComparisonExpression() #9 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(2447): Doctrine\ORM\Query\Parser->SimpleConditionalExpression() #10 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(2423): Doctrine\ORM\Query\Parser->ConditionalPrimary() #11 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(2391): Doctrine\ORM\Query\Parser->ConditionalFactor() #12 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(2366): Doctrine\ORM\Query\Parser->ConditionalTerm() #13 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(1333): Doctrine\ORM\Query\Parser->ConditionalExpression() #14 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(876): Doctrine\ORM\Query\Parser->WhereClause() #15 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(843): Doctrine\ORM\Query\Parser->SelectStatement() #16 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(268): Doctrine\ORM\Query\Parser->QueryLanguage() #17 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(351): Doctrine\ORM\Query\Parser->getAST() #18 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query.php(281): Doctrine\ORM\Query\Parser->parse() #19 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query.php(293): Doctrine\ORM\Query->_parse() #20 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/AbstractQuery.php(969): Doctrine\ORM\Query->_doExecute() #21 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/AbstractQuery.php(924): Doctrine\ORM\AbstractQuery->executeIgnoreQueryCache(NULL, NULL) #22 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/AbstractQuery.php(804): Doctrine\ORM\AbstractQuery->execute(NULL, NULL) #23 /websites/distrigoboost/crm-application/src/IZMO/MediaLibraryBundle/Entity/Repository/MediaFileRepository.php(128): Doctrine\ORM\AbstractQuery->getSingleResult() #24 /websites/distrigoboost/crm-application/src/IZMO/MediaLibraryBundle/Provider/MediaLibrary.php(237): IZMO\MediaLibraryBundle\Entity\Repository\MediaFileRepository->getMediaFile(NULL) #25 /websites/distrigoboost/crm-application/vendor/oro/platform/src/Oro/Bundle/UserBundle/Controller/SecurityController.php(99): IZMO\MediaLibraryBundle\Provider\MediaLibrary->getUrlOnLoginPage(NULL) #26 /websites/distrigoboost/crm-application/vendor/oro/platform/src/Oro/Bundle/UserBundle/Controller/SecurityController.php(32): Oro\Bundle\UserBundle\Controller\SecurityController->getBadgesInfo() #27 [internal function]: Oro\Bundle\UserBundle\Controller\SecurityController->loginAction() #28 /websites/distrigoboost/crm-application/app/bootstrap.php.cache(3242): call_user_func_array(Array, Array) #29 /websites/distrigoboost/crm-application/app/bootstrap.php.cache(3201): Symfony\Component\HttpKernel\HttpKernel->handleRaw(Object(Symfony\Component\HttpFoundation\Request), 1) #30 /websites/distrigoboost/crm-application/app/bootstrap.php.cache(3355): Symfony\Component\HttpKernel\HttpKernel->handle(Object(Symfony\Component\HttpFoundation\Request), 1, true) #31 /websites/distrigoboost/crm-application/app/bootstrap.php.cache(2540): Symfony\Component\HttpKernel\DependencyInjection\ContainerAwareHttpKernel->handle(Object(Symfony\Component\HttpFoundation\Request), 1, true) #32 /websites/distrigoboost/crm-application/web/app.php(22): Symfony\Component\HttpKernel\Kernel->handle(Object(Symfony\Component\HttpFoundation\Request)) #33 {main}  Next Doctrine\ORM\Query\QueryException: [Syntax Error] line 0, col -1: Error: Expected Literal, got end of string. in /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/QueryException.php:52 Stack trace: #0 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(448): Doctrine\ORM\Query\QueryException::syntaxError('line 0, col -1:...', Object(Doctrine\ORM\Query\QueryException)) #1 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(2649): Doctrine\ORM\Query\Parser->syntaxError('Literal') #2 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(2837): Doctrine\ORM\Query\Parser->Literal() #3 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(2771): Doctrine\ORM\Query\Parser->ArithmeticPrimary() #4 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(2739): Doctrine\ORM\Query\Parser->ArithmeticFactor() #5 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(2713): Doctrine\ORM\Query\Parser->ArithmeticTerm() #6 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(2700): Doctrine\ORM\Query\Parser->SimpleArithmeticExpression() #7 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(3043): Doctrine\ORM\Query\Parser->ArithmeticExpression() #8 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(2564): Doctrine\ORM\Query\Parser->ComparisonExpression() #9 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(2447): Doctrine\ORM\Query\Parser->SimpleConditionalExpression() #10 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(2423): Doctrine\ORM\Query\Parser->ConditionalPrimary() #11 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(2391): Doctrine\ORM\Query\Parser->ConditionalFactor() #12 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(2366): Doctrine\ORM\Query\Parser->ConditionalTerm() #13 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(1333): Doctrine\ORM\Query\Parser->ConditionalExpression() #14 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(876): Doctrine\ORM\Query\Parser->WhereClause() #15 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(843): Doctrine\ORM\Query\Parser->SelectStatement() #16 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(268): Doctrine\ORM\Query\Parser->QueryLanguage() #17 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(351): Doctrine\ORM\Query\Parser->getAST() #18 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query.php(281): Doctrine\ORM\Query\Parser->parse() #19 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query.php(293): Doctrine\ORM\Query->_parse() #20 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/AbstractQuery.php(969): Doctrine\ORM\Query->_doExecute() #21 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/AbstractQuery.php(924): Doctrine\ORM\AbstractQuery->executeIgnoreQueryCache(NULL, NULL) #22 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/AbstractQuery.php(804): Doctrine\ORM\AbstractQuery->execute(NULL, NULL) #23 /websites/distrigoboost/crm-application/src/IZMO/MediaLibraryBundle/Entity/Repository/MediaFileRepository.php(128): Doctrine\ORM\AbstractQuery->getSingleResult() #24 /websites/distrigoboost/crm-application/src/IZMO/MediaLibraryBundle/Provider/MediaLibrary.php(237): IZMO\MediaLibraryBundle\Entity\Repository\MediaFileRepository->getMediaFile(NULL) #25 /websites/distrigoboost/crm-application/vendor/oro/platform/src/Oro/Bundle/UserBundle/Controller/SecurityController.php(99): IZMO\MediaLibraryBundle\Provider\MediaLibrary->getUrlOnLoginPage(NULL) #26 /websites/distrigoboost/crm-application/vendor/oro/platform/src/Oro/Bundle/UserBundle/Controller/SecurityController.php(32): Oro\Bundle\UserBundle\Controller\SecurityController->getBadgesInfo() #27 [internal function]: Oro\Bundle\UserBundle\Controller\SecurityController->loginAction() #28 /websites/distrigoboost/crm-application/app/bootstrap.php.cache(3242): call_user_func_array(Array, Array) #29 /websites/distrigoboost/crm-application/app/bootstrap.php.cache(3201): Symfony\Component\HttpKernel\HttpKernel->handleRaw(Object(Symfony\Component\HttpFoundation\Request), 1) #30 /websites/distrigoboost/crm-application/app/bootstrap.php.cache(3355): Symfony\Component\HttpKernel\HttpKernel->handle(Object(Symfony\Component\HttpFoundation\Request), 1, true) #31 /websites/distrigoboost/crm-application/app/bootstrap.php.cache(2540): Symfony\Component\HttpKernel\DependencyInjection\ContainerAwareHttpKernel->handle(Object(Symfony\Component\HttpFoundation\Request), 1, true) #32 /websites/distrigoboost/crm-application/web/app.php(22): Symfony\Component\HttpKernel\Kernel->handle(Object(Symfony\Component\HttpFoundation\Request)) #33 {main} [] {"user_info":"","url_info":" HOST: distrigoboostqa.izdemosite.com REQUEST URI: /user/login"}

this are the frequent log entries apart from this there are other different log entries also but those are not frequent
from all those log entries i want to view ( timestamp, loglevel, host, request uri, and message field ) message field with whole log entry, only those fields!

PUT _ingest/pipeline/discuss-test
{
  "processors": [
    {
      "dissect": {
        "field": "message",
        "pattern": "[%{timestamp}][%{loglevel}][HOST: %{host.name} REQUEST URI: %{request_uri}] %{message_detail}"
      }
    }
  ]
}

for this pipeline for first log entry im not able see to loglevel it dispalying empty
image
for second log entry it giving an error like

{
  "error": {
    "root_cause": [
      {
        "type": "find_match",
        "reason": """Unable to find match for dissect pattern: [%{timestamp}][%{loglevel}][HOST: %{host} REQUEST URI: %{request_uri}] %{message_detail} 

i want a pipeline which matches for log kind of log entries in that file and from that it have to extract these fields => ( timestamp, loglevel, host, request uri, and message field )

Hi @evangelin

No super simple solution.

In general You are going to need to create a grok pattern match for each log pattern.

grok takes an array of patterns dissect only takes one pattern.

Create the most common pattern first, then add the next most common pattern and keep going.

How do you check these logs, I'm encountries the same same issue!