# Filebeat is not sending any logs to logstash please help urgent! **URL:** https://discuss.elastic.co/t/filebeat-is-not-sending-any-logs-to-logstash-please-help-urgent/362161 **Category:** Beats **Tags:** filebeat **Created:** [June 27, 2024, 1:26pm UTC](https://discuss.elastic.co/t/filebeat-is-not-sending-any-logs-to-logstash-please-help-urgent/362161 "2024-06-27T13:26:29Z") **Posts on this page:** 15 **Page:** 1
### Author: ![evangelin](https://avatars.discourse-cdn.com/v4/letter/e/838e76/32.png) [@evangelin](https://discuss.elastic.co/u/evangelin) #### Post date: [June 27, 2024, 1:26pm UTC](https://discuss.elastic.co/t/filebeat-is-not-sending-any-logs-to-logstash-please-help-urgent/362161/1 "2024-06-27T13:26:29Z")
Hello, i have installed elk in an aws ec2 instance ubuntu and filebeat in another ubuntu instance. Below is my filebeat.yml file ```auto cat /etc/filebeat/filebeat.yml ###################### Filebeat Configuration Example ######################### # This file is an example configuration file highlighting only the most common # options. The filebeat.reference.yml file from the same directory contains all the # supported options with more comments. You can use it as a reference. # # You can find the full configuration reference here: # https://www.elastic.co/guide/en/beats/filebeat/index.html # For more available modules and options, please see the filebeat.reference.yml sample # configuration file. # ============================== Filebeat inputs =============================== filebeat.inputs: # Each - is an input. Most options can be set at the input level, so # you can use different inputs for various configurations. # Below are the input-specific configurations. # filestream is an input for collecting log messages from files. - type: log # Unique ID among all inputs, an ID is required. id: orocrm # Change to true to enable this input configuration. enabled: true # Paths that should be crawled and fetched. Glob based paths. paths: - /orocrmprod.log #- c:\programdata\elasticsearch\logs\* # Exclude lines. A list of regular expressions to match. It drops the lines that are # matching any regular expression from the list. # Line filtering happens after the parsers pipeline. If you would like to filter lines # before parsers, use include_message parser. #exclude_lines: ['^DBG'] # Include lines. A list of regular expressions to match. It exports the lines that are # matching any regular expression from the list. # Line filtering happens after the parsers pipeline. If you would like to filter lines # before parsers, use include_message parser. #include_lines: ['^ERR', '^WARN'] # Exclude files. A list of regular expressions to match. Filebeat drops the files that # are matching any regular expression from the list. By default, no files are dropped. #prospector.scanner.exclude_files: ['.gz$'] # Optional additional fields. These fields can be freely picked # to add additional information to the crawled log files for filtering #fields: # level: debug # review: 1 # ============================== Filebeat modules ============================== filebeat.config.modules: # Glob pattern for configuration loading path: ${path.config}/modules.d/*.yml # Set to true to enable config reloading reload.enabled: false # Period on which files under path should be checked for changes #reload.period: 10s # ======================= Elasticsearch template setting ======================= setup.template.settings: index.number_of_shards: 1 #index.codec: best_compression #_source.enabled: false # ================================== General =================================== # The name of the shipper that publishes the network data. It can be used to group # all the transactions sent by a single shipper in the web interface. #name: # The tags of the shipper are included in their field with each # transaction published. #tags: ["service-X", "web-tier"] # Optional fields that you can specify to add additional information to the # output. #fields: # env: staging # ================================= Dashboards ================================= # These settings control loading the sample dashboards to the Kibana index. Loading # the dashboards is disabled by default and can be enabled either by setting the # options here or by using the `setup` command. #setup.dashboards.enabled: false # The URL from where to download the dashboard archive. By default, this URL # has a value that is computed based on the Beat name and version. For released # versions, this URL points to the dashboard archive on the artifacts.elastic.co # website. #setup.dashboards.url: # =================================== Kibana =================================== # Starting with Beats version 6.0.0, the dashboards are loaded via the Kibana API. # This requires a Kibana endpoint configuration. setup.kibana: # Kibana Host # Scheme and port can be left out and will be set to the default (http and 5601) # In case you specify and additional path, the scheme is required: http://localhost:5601/path # IPv6 addresses should always be defined as: https://[2001:db8::1]:5601 #host: "localhost:5601" # Kibana Space ID # ID of the Kibana Space into which the dashboards should be loaded. By default, # the Default Space will be used. #space.id: # =============================== Elastic Cloud ================================ # These settings simplify using Filebeat with the Elastic Cloud (https://cloud.elastic.co/). # The cloud.id setting overwrites the `output.elasticsearch.hosts` and # `setup.kibana.host` options. # You can find the `cloud.id` in the Elastic Cloud web UI. #cloud.id: # The cloud.auth setting overwrites the `output.elasticsearch.username` and # `output.elasticsearch.password` settings. The format is `:`. #cloud.auth: # ================================== Outputs =================================== # Configure what output to use when sending the data collected by the beat. # ---------------------------- Elasticsearch Output ---------------------------- #output.elasticsearch: # Array of hosts to connect to. #hosts: ["localhost:9200"] # Performance preset - one of "balanced", "throughput", "scale", # "latency", or "custom". # preset: balanced # Protocol - either `http` (default) or `https`. #protocol: "https" # Authentication credentials - either API key or username/password. #api_key: "id:api_key" #username: "elastic" #password: "changeme" # ------------------------------ Logstash Output ------------------------------- output.logstash: # The Logstash hosts hosts: ["172.31.34.24:5044"] # Optional SSL. By default is off. # List of root certificates for HTTPS server verifications #ssl.certificate_authorities: ["/etc/pki/root/ca.pem"] # Certificate for SSL client authentication #ssl.certificate: "/etc/pki/client/cert.pem" # Client Certificate Key #ssl.key: "/etc/pki/client/cert.key" # ================================= Processors ================================= processors: - add_host_metadata: when.not.contains.tags: forwarded - add_cloud_metadata: ~ - add_docker_metadata: ~ - add_kubernetes_metadata: ~ # ================================== Logging =================================== # Sets log level. The default log level is info. # Available log levels are: error, warning, info, debug #logging.level: debug # At debug level, you can selectively enable logging only for some components. # To enable all selectors, use ["*"]. Examples of other selectors are "beat", # "publisher", "service". #logging.selectors: ["*"] # ============================= X-Pack Monitoring ============================== # Filebeat can export internal metrics to a central Elasticsearch monitoring # cluster. This requires xpack monitoring to be enabled in Elasticsearch. The # reporting is disabled by default. # Set to true to enable the monitoring reporter. #monitoring.enabled: false # Sets the UUID of the Elasticsearch cluster under which monitoring data for this # Filebeat instance will appear in the Stack Monitoring UI. If output.elasticsearch # is enabled, the UUID is derived from the Elasticsearch cluster referenced by output.elasticsearch. #monitoring.cluster_uuid: # Uncomment to send the metrics to Elasticsearch. Most settings from the # Elasticsearch outputs are accepted here as well. # Note that the settings should point to your Elasticsearch *monitoring* cluster. # Any setting that is not set is automatically inherited from the Elasticsearch # output configuration, so if you have the Elasticsearch output configured such # that it is pointing to your Elasticsearch monitoring cluster, you can simply # uncomment the following line. #monitoring.elasticsearch: # ============================== Instrumentation =============================== # Instrumentation support for the filebeat. #instrumentation: # Set to true to enable instrumentation of filebeat. #enabled: false # Environment in which filebeat is running on (eg: staging, production, etc.) #environment: "" # APM Server hosts to report instrumentation results to. #hosts: # - http://localhost:8200 # API Key for the APM Server(s). # If api_key is set then secret_token will be ignored. #api_key: # Secret token for the APM Server(s). #secret_token: # ================================= Migration ================================== # This allows to enable 6.7 migration aliases #migration.6_to_7.enabled: true ``` here is my logstash conf file. I have tried with the output to elasticsearch it was not showing anything in kibana so i have used stdout { codec =\> rubydebug } here where i got that it was not receiving any logs . ```auto cat /etc/logstash/conf.d/oroprod.conf input { beats { port => 5044 } } filter { grok { match => { "message" => "\[%{TIMESTAMP_ISO8601:timestamp}\]\[%{DATA:loglevel}\] %{GREEDYDATA:message}" } } grok { match => { "message" => "HOST: %{HOSTNAME:host} REQUEST URI: %{URIPATH:request_uri}" } } } output { stdout { codec => rubydebug } } ``` here is the output it got stuck there it was not printing anything error also ```auto bin/logstash -f /etc/logstash/conf.d/oroprod.conf Using bundled JDK: /usr/share/logstash/jdk WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console [WARN] 2024-06-27 12:54:03.835 [main] runner - NOTICE: Running Logstash as superuser is not recommended and won't be allowed in the future. Set 'allow_superuser' to 'false' to avoid startup errors in future releases. [INFO] 2024-06-27 12:54:03.846 [main] runner - Starting Logstash {"logstash.version"=>"8.14.1", "jruby.version"=>"jruby 9.4.7.0 (3.1.4) 2024-04-29 597ff08ac1 OpenJDK 64-Bit Server VM 17.0.11+9 on 17.0.11+9 +indy +jit [x86_64-linux]"} [INFO] 2024-06-27 12:54:03.849 [main] runner - JVM bootstrap flags: [-Xms1g, -Xmx1g, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djruby.compile.invokedynamic=true, -XX:+HeapDumpOnOutOfMemoryError, -Djava.security.egd=file:/dev/urandom, -Dlog4j2.isThreadContextMapInheritable=true, -Dlogstash.jackson.stream-read-constraints.max-string-length=200000000, -Dlogstash.jackson.stream-read-constraints.max-number-length=10000, -Djruby.regexp.interruptible=true, -Djdk.io.File.enableADS=true, --add-exports=jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.parser=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED, --add-opens=java.base/java.security=ALL-UNNAMED, --add-opens=java.base/java.io=ALL-UNNAMED, --add-opens=java.base/java.nio.channels=ALL-UNNAMED, --add-opens=java.base/sun.nio.ch=ALL-UNNAMED, --add-opens=java.management/sun.management=ALL-UNNAMED, -Dio.netty.allocator.maxOrder=11] [INFO] 2024-06-27 12:54:03.851 [main] runner - Jackson default value override `logstash.jackson.stream-read-constraints.max-string-length` configured to `200000000` [INFO] 2024-06-27 12:54:03.851 [main] runner - Jackson default value override `logstash.jackson.stream-read-constraints.max-number-length` configured to `10000` [WARN] 2024-06-27 12:54:04.025 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified [INFO] 2024-06-27 12:54:04.604 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600, :ssl_enabled=>false} [INFO] 2024-06-27 12:54:04.922 [Converge PipelineAction::Create
] Reflections - Reflections took 109 ms to scan 1 urls, producing 132 keys and 468 values [INFO] 2024-06-27 12:54:05.345 [Converge PipelineAction::Create
] javapipeline - Pipeline `main` is configured with `pipeline.ecs_compatibility: v8` setting. All plugins in this pipeline will default to `ecs_compatibility => v8` unless explicitly configured otherwise. [WARN] 2024-06-27 12:54:05.368 [[main]-pipeline-manager] grok - ECS v8 support is a preview of the unreleased ECS v8, and uses the v1 patterns. When Version 8 of the Elastic Common Schema becomes available, this plugin will need to be updated [WARN] 2024-06-27 12:54:05.475 [[main]-pipeline-manager] grok - ECS v8 support is a preview of the unreleased ECS v8, and uses the v1 patterns. When Version 8 of the Elastic Common Schema becomes available, this plugin will need to be updated [INFO] 2024-06-27 12:54:05.513 [[main]-pipeline-manager] javapipeline - Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>250, "pipeline.sources"=>["/etc/logstash/conf.d/oroprod.conf"], :thread=>"#"} [INFO] 2024-06-27 12:54:05.973 [[main]-pipeline-manager] javapipeline - Pipeline Java execution initialization time {"seconds"=>0.46} [INFO] 2024-06-27 12:54:05.978 [[main]-pipeline-manager] beats - Starting input listener {:address=>"0.0.0.0:5044"} [INFO] 2024-06-27 12:54:05.984 [[main]-pipeline-manager] javapipeline - Pipeline started {"pipeline.id"=>"main"} [INFO] 2024-06-27 12:54:06.001 [Agent thread] agent - Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]} [INFO] 2024-06-27 12:54:06.055 [[main]syntaxError('Literal') #2 /websites/disst/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(2837):pplication/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(2739): Doctrine\ORM\Query\Parser->ArithmeticFactor() #5 /websites/distrigoboost/crm-apt/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/AbstractQuery.php(924): Doctrine\ORM\AbstractQuery->exectrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/AbstractQuery.php(804): Doctrine\ORM\AbstractQuery->execute(NULL, NULL) #23 /websites/distrigoboost/crm-application/src/IZMO/MediaLibraryBundle/Entity/Repository/MediaFileRepository.php(128): Doctrine\ORM\AbstractQuery->getSingleResult() #latform/src/Oro/Bundle/UserBundle/Controller/SecurityController.php(99): IZMO\MediaLibraryBundle\Provider\MediaLibrary->getUrlOnLoginPage(NULL) #26 /websites/dist/crm-application/vendor/oro/platform/src/Oro/Bundle/UserBundle/Controller/SecurityController.php(32): Oro\Bundle\UserBundle\Controller\SecurityController->getBadgesInfo() #27 [internal function]: Oro\Bundle\UserBundle\Controller\SecurityController->loginAction() #28 /websites/dist/crm-application/app/bootstrap.php.cache(3242): call_user_func_array(Array, Array) #29 /websites/diost/crm-application/app/bootstrap.php.cache(3201): Symfony\Component\HttpKernel\HttpKernel->handleRaw(Object(Symfony\Component\HttpFoundation\Request), 1) #30 /websites/dist/crm-application/app/bootstrap.php.cache(3355): Symfony\Component\HttpKernel\HttpKernel->handle(Object(Symfony\Component\HttpFoundation\Request), 1, true) #31 /websites/distrigoboost/crm-application/app/bootstrap.php.cache(2540): Symfony\Component\HttpKernel\DependencyInjection\ContainerAwareHttpKernel->handle(Object(Symfony\Component\HttpFoundation\Request), 1, true) #32 /websites/distrigoboost/crm-application/web/app.php(22): Symfony\Component\HttpKernel\Kernel->handle(Object(Symfony\Component\HttpFoundation\Request)) #33 {main} [] [] ``` here in this log entry i no need all things to get parsed i just need the request uri and timstamp log.level that's it so that why i used grok pattern ```auto \[%{TIMESTAMP_ISO8601:timestamp}\]\[\]\[HOST: %{DATA:host} REQUEST URI: %{DATA:request_uri}\] %{WORD:source}\.%{LOGLEVEL:loglevel}: %{DATA:error_message} %{JAVACLASS:exception_class}: %{DATA:exception_message} in %{DATA:exception_location} Stack trace: %{GREEDYDATA:stack_trace} \[\] \[\] ``` its giving the correct output in grok debuger why its not displaying anything when i run logstash idk and there us no filebeat logs in syslog it showing the metric information ---
### Author: ![rugenl](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rugenl/32/12887_2.png) [@rugenl](https://discuss.elastic.co/u/rugenl) #### Post date: [June 27, 2024, 2:17pm UTC](https://discuss.elastic.co/t/filebeat-is-not-sending-any-logs-to-logstash-please-help-urgent/362161/2 "2024-06-27T14:17:10Z")
Run tcpdump on the logstash host for that port, are you getting input? ---
### Author: ![evangelin](https://avatars.discourse-cdn.com/v4/letter/e/838e76/32.png) [@evangelin](https://discuss.elastic.co/u/evangelin) #### Post date: [June 28, 2024, 11:59am UTC](https://discuss.elastic.co/t/filebeat-is-not-sending-any-logs-to-logstash-please-help-urgent/362161/3 "2024-06-28T11:59:53Z")
thanks for the reply, here is the ans for tcpdump on the logstash host ```auto sudo tcpdump -i any port 5044 -vvv tcpdump: data link type LINUX_SLL2 tcpdump: listening on any, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 262144 bytes ^C 0 packets captured 1 packet received by filter 0 packets dropped by kernel root@oroprod-elk:/# ``` But im able to connect to logstash host from filebeat server ```auto filebeat test output logstash: 172.31.34.24:5044... connection... parse host... OK dns lookup... OK addresses: 172.31.34.24 dial up... OK TLS... WARN secure connection disabled talk to server... OK ``` ---
### Author: ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb) #### Post date: [June 30, 2024, 3:11pm UTC](https://discuss.elastic.co/t/filebeat-is-not-sending-any-logs-to-logstash-please-help-urgent/362161/4 "2024-06-30T15:11:47Z")
How did you install logstash? If you installed with a package manager this is not the proper command. (It is incomplete) ```auto bin/logstash -f /etc/logstash/conf.d/oroprod.conf ``` Perhaps take a look at this post > [@Pipeline is running but index is not created at elasticsearch](https://discuss.elastic.co/t/pipeline-is-running-but-index-is-not-created-at-elasticsearch/333940/28): > > @Yasser_Alsawy Good job trudging through this! And now like I said I think it was something simple that we all did not see. Did you install as a package? And then you were running from the command line. And like I said I should have seen this... If you install as a package then run as a command line you have to add additional parameters to tell it where the config is at. --path.settings /etc/logstash This tells logstash where logstash.yml and pipeline.yml are at. So the command should b… ---
### Author: ![evangelin](https://avatars.discourse-cdn.com/v4/letter/e/838e76/32.png) [@evangelin](https://discuss.elastic.co/u/evangelin) #### Post date: [July 2, 2024, 4:32am UTC](https://discuss.elastic.co/t/filebeat-is-not-sending-any-logs-to-logstash-please-help-urgent/362161/5 "2024-07-02T04:32:41Z")
thanks for your reply i have installed the elk stack using the debain commands from the document i have tried moving logstash.yml file to /usr/share/logsatsh/config and gave permission to /var/lib/logstash and all hierarchy But still idk im not able to see anything parsing logs can be done through the other way also right using custom integration by that i have sent logs in elasticsearch But it storing whole log entry in message field IDK how to use the custom log integration according to our need i mean how to parse the logs using custom log integration please explain!!!! ---
### Author: ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb) #### Post date: [July 2, 2024, 2:51pm UTC](https://discuss.elastic.co/t/filebeat-is-not-sending-any-logs-to-logstash-please-help-urgent/362161/6 "2024-07-02T14:51:08Z")
> [@evangelin](#): > > i have tried moving logstash.yml That will not help, put it back... Did you try the command I shared in the post `/usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/oroprod.conf -r --path.settings /etc/logstash` ---
### Author: ![evangelin](https://avatars.discourse-cdn.com/v4/letter/e/838e76/32.png) [@evangelin](https://discuss.elastic.co/u/evangelin) #### Post date: [July 3, 2024, 1:21pm UTC](https://discuss.elastic.co/t/filebeat-is-not-sending-any-logs-to-logstash-please-help-urgent/362161/7 "2024-07-03T13:21:18Z")
the output for the command `/usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/oroprod.conf -r --path.settings /etc/logstash ```auto /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/orocrmprod.conf -r --path.settings /etc/logstash Using bundled JDK: /usr/share/logstash/jdk /usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/concurrent-ruby-1.1.9/lib/concurrent-ruby/concurrent/executor/java_thread_pool_executor.rb:13: warning: method redefined; discarding old to_int /usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/concurrent-ruby-1.1.9/lib/concurrent-ruby/concurrent/executor/java_thread_pool_executor.rb:13: warning: method redefined; discarding old to_f Sending Logstash logs to /var/log/logstash which is now configured via log4j2.properties [2024-07-03T05:50:27,637][INFO][logstash.runner] Log4j configuration path used is: /etc/logstash/log4j2.properties [2024-07-03T05:50:27,640][INFO][logstash.runner] Starting Logstash {"logstash.version"=>"8.13.4", "jruby.version"=>"jruby 9.4.5.0 (3.1.4) 2023-11-02 1abae2700f OpenJDK 64-Bit Server VM 17.0.11+9 on 17.0.11+9 +indy +jit [x86_64-linux]"} [2024-07-03T05:50:27,643][INFO][logstash.runner] JVM bootstrap flags: [-Xms1g, -Xmx1g, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djruby.compile.invokedynamic=true, -XX:+HeapDumpOnOutOfMemoryError, -Djava.security.egd=file:/dev/urandom, -Dlog4j2.isThreadContextMapInheritable=true, -Dlogstash.jackson.stream-read-constraints.max-string-length=200000000, -Dlogstash.jackson.stream-read-constraints.max-number-length=10000, -Djruby.regexp.interruptible=true, -Djdk.io.File.enableADS=true, --add-exports=jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.parser=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED, --add-opens=java.base/java.security=ALL-UNNAMED, --add-opens=java.base/java.io=ALL-UNNAMED, --add-opens=java.base/java.nio.channels=ALL-UNNAMED, --add-opens=java.base/sun.nio.ch=ALL-UNNAMED, --add-opens=java.management/sun.management=ALL-UNNAMED, -Dio.netty.allocator.maxOrder=11] [2024-07-03T05:50:27,644][INFO][logstash.runner] Jackson default value override `logstash.jackson.stream-read-constraints.max-string-length` configured to `200000000` [2024-07-03T05:50:27,644][INFO][logstash.runner] Jackson default value override `logstash.jackson.stream-read-constraints.max-number-length` configured to `10000` [2024-07-03T05:50:27,798][WARN][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified [2024-07-03T05:50:28,234][INFO][logstash.agent] Successfully started Logstash API endpoint {:port=>9600, :ssl_enabled=>false} [2024-07-03T05:50:28,506][INFO][org.reflections.Reflections] Reflections took 91 ms to scan 1 urls, producing 132 keys and 468 values /usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/amazing_print-1.6.0/lib/amazing_print/formatter.rb:37: warning: previous definition of cast was here [2024-07-03T05:50:29,250][INFO][logstash.javapipeline] Pipeline `main` is configured with `pipeline.ecs_compatibility: v8` setting. All plugins in this pipeline will default to `ecs_compatibility => v8` unless explicitly configured otherwise. [2024-07-03T05:50:29,263][WARN][logstash.filters.grok][main] ECS v8 support is a preview of the unreleased ECS v8, and uses the v1 patterns. When Version 8 of the Elastic Common Schema becomes available, this plugin will need to be updated [2024-07-03T05:50:29,338][INFO][logstash.javapipeline][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>500, "pipeline.sources"=>["/etc/logstash/conf.d/orocrmprod.conf"], :thread=>"#"} [2024-07-03T05:50:29,861][INFO][logstash.javapipeline][main] Pipeline Java execution initialization time {"seconds"=>0.52} [2024-07-03T05:50:29,864][INFO][logstash.inputs.beats][main] Starting input listener {:address=>"0.0.0.0:5044"} [2024-07-03T05:50:29,871][INFO][logstash.javapipeline][main] Pipeline started {"pipeline.id"=>"main"} [2024-07-03T05:50:29,882][INFO][logstash.agent] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]} [2024-07-03T05:50:29,927][INFO][org.logstash.beats.Server][main][7f0f85fa7b42843f3bfef4b709e2ade01002d5187b424b486a1f2a8e93c37901] Starting server on port: 5044 ``` it was not displaying anything after this Sir, side by side i'm trying to parse the logs using custom log integration . In that whole log entry was storing inside the message field idk how to parse it, i tried mentioning the below configuration in processor in custom log integration ```auto [ { "dissect": { "field": "message", "tokenizer": "[%{timestamp}] [%{loglevel}] [HOST: %{host} REQUEST URI: %{request_uri}] %{message}" } } ] ``` And i have tried mention the pipeline in the custom configuration, But still it storing the whole log entry in message field can you help me how to parse the logs using custom log integration. ---
### Author: ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb) #### Post date: [July 3, 2024, 2:26pm UTC](https://discuss.elastic.co/t/filebeat-is-not-sending-any-logs-to-logstash-please-help-urgent/362161/8 "2024-07-03T14:26:22Z")
> [@evangelin](#): > > it was not displaying anything after this That means logstash is up an running and waiting for data... > [@evangelin](#): > > Sir, side by side i'm trying to parse the logs using custom log integration . In that whole log entry was storing inside the message field idk how to parse it, i tried mentioning the below configuration in processor in custom log integration Sorry I am confused you are also trying an ingest pipeline... please do not mix topics... Can you back up Are you trying to do a) Filebeat -\> Elasticsearch with Ingest Pipeline or b) Filebeat -\> Logstash -\> Elasticsearch Also What version are you on? You do not need to use logstash to parse logs ---
### Author: ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb) #### Post date: [July 3, 2024, 3:20pm UTC](https://discuss.elastic.co/t/filebeat-is-not-sending-any-logs-to-logstash-please-help-urgent/362161/9 "2024-07-03T15:20:03Z")
Working example Filebeat -\> Elasticsearch with Ingest Pipeline file ```auto [2024-06-26 06:56:53][][HOST: distqa.izdemosite.com REQUEST URI: /user/login] app.CRITICAL: Error Occured while getting visit report images:Doctrine\ORM\Query\QueryException: SELECT mf.mediaFileId as id, CONCAT(mf.rootPath, mf.mediaFileName) as url FROM IZMOMediaLibraryBundle:MediaFile mf WHERE mf.mediaFileId = in /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/QueryException.php:41 Stack trace: #0 /websites/diost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(448): Doctrine\ORM\Query\QueryException::dqlError('SELECT mf.media...') #1 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(2649): Doctrine\ORM\Query\Parser->syntaxError('Literal') #2 /websites/disst/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(2837):pplication/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(2739): Doctrine\ORM\Query\Parser->ArithmeticFactor() #5 /websites/distrigoboost/crm-apt/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/AbstractQuery.php(924): Doctrine\ORM\AbstractQuery->exectrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/AbstractQuery.php(804): Doctrine\ORM\AbstractQuery->execute(NULL, NULL) #23 /websites/distrigoboost/crm-application/src/IZMO/MediaLibraryBundle/Entity/Repository/MediaFileRepository.php(128): Doctrine\ORM\AbstractQuery->getSingleResult() #latform/src/Oro/Bundle/UserBundle/Controller/SecurityController.php(99): IZMO\MediaLibraryBundle\Provider\MediaLibrary->getUrlOnLoginPage(NULL) #26 /websites/dist/crm-application/vendor/oro/platform/src/Oro/Bundle/UserBundle/Controller/SecurityController.php(32): Oro\Bundle\UserBundle\Controller\SecurityController->getBadgesInfo() #27 [internal function]: Oro\Bundle\UserBundle\Controller\SecurityController->loginAction() #28 /websites/dist/crm-application/app/bootstrap.php.cache(3242): call_user_func_array(Array, Array) #29 /websites/diost/crm-application/app/bootstrap.php.cache(3201): Symfony\Component\HttpKernel\HttpKernel->handleRaw(Object(Symfony\Component\HttpFoundation\Request), 1) #30 /websites/dist/crm-application/app/bootstrap.php.cache(3355): Symfony\Component\HttpKernel\HttpKernel->handle(Object(Symfony\Component\HttpFoundation\Request), 1, true) #31 /websites/distrigoboost/crm-application/app/bootstrap.php.cache(2540): Symfony\Component\HttpKernel\DependencyInjection\ContainerAwareHttpKernel->handle(Object(Symfony\Component\HttpFoundation\Request), 1, true) #32 /websites/distrigoboost/crm-application/web/app.php(22): Symfony\Component\HttpKernel\Kernel->handle(Object(Symfony\Component\HttpFoundation\Request)) #33 {main} [] [] [2024-06-26 06:56:53][][HOST: distqa.izdemosite.com REQUEST URI: /user/login] app.CRITICAL: Error Occured while getting visit report images:Doctrine\ORM\Query\QueryException: SELECT mf.mediaFileId as id, CONCAT(mf.rootPath, mf.mediaFileName) as url FROM IZMOMediaLibraryBundle:MediaFile mf WHERE mf.mediaFileId = in /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/QueryException.php:41 Stack trace: #0 /websites/diost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(448): Doctrine\ORM\Query\QueryException::dqlError('SELECT mf.media...') #1 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(2649): Doctrine\ORM\Query\Parser->syntaxError('Literal') #2 /websites/disst/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(2837):pplication/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(2739): Doctrine\ORM\Query\Parser->ArithmeticFactor() #5 /websites/distrigoboost/crm-apt/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/AbstractQuery.php(924): Doctrine\ORM\AbstractQuery->exectrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/AbstractQuery.php(804): Doctrine\ORM\AbstractQuery->execute(NULL, NULL) #23 /websites/distrigoboost/crm-application/src/IZMO/MediaLibraryBundle/Entity/Repository/MediaFileRepository.php(128): Doctrine\ORM\AbstractQuery->getSingleResult() #latform/src/Oro/Bundle/UserBundle/Controller/SecurityController.php(99): IZMO\MediaLibraryBundle\Provider\MediaLibrary->getUrlOnLoginPage(NULL) #26 /websites/dist/crm-application/vendor/oro/platform/src/Oro/Bundle/UserBundle/Controller/SecurityController.php(32): Oro\Bundle\UserBundle\Controller\SecurityController->getBadgesInfo() #27 [internal function]: Oro\Bundle\UserBundle\Controller\SecurityController->loginAction() #28 /websites/dist/crm-application/app/bootstrap.php.cache(3242): call_user_func_array(Array, Array) #29 /websites/diost/crm-application/app/bootstrap.php.cache(3201): Symfony\Component\HttpKernel\HttpKernel->handleRaw(Object(Symfony\Component\HttpFoundation\Request), 1) #30 /websites/dist/crm-application/app/bootstrap.php.cache(3355): Symfony\Component\HttpKernel\HttpKernel->handle(Object(Symfony\Component\HttpFoundation\Request), 1, true) #31 /websites/distrigoboost/crm-application/app/bootstrap.php.cache(2540): Symfony\Component\HttpKernel\DependencyInjection\ContainerAwareHttpKernel->handle(Object(Symfony\Component\HttpFoundation\Request), 1, true) #32 /websites/distrigoboost/crm-application/web/app.php(22): Symfony\Component\HttpKernel\Kernel->handle(Object(Symfony\Component\HttpFoundation\Request)) #33 {main} [] [] [2024-06-26 06:56:53][][HOST: distqa.izdemosite.com REQUEST URI: /user/login] app.CRITICAL: Error Occured while getting visit report images:Doctrine\ORM\Query\QueryException: SELECT mf.mediaFileId as id, CONCAT(mf.rootPath, mf.mediaFileName) as url FROM IZMOMediaLibraryBundle:MediaFile mf WHERE mf.mediaFileId = in /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/QueryException.php:41 Stack trace: #0 /websites/diost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(448): Doctrine\ORM\Query\QueryException::dqlError('SELECT mf.media...') #1 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(2649): Doctrine\ORM\Query\Parser->syntaxError('Literal') #2 /websites/disst/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(2837):pplication/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(2739): Doctrine\ORM\Query\Parser->ArithmeticFactor() #5 /websites/distrigoboost/crm-apt/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/AbstractQuery.php(924): Doctrine\ORM\AbstractQuery->exectrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/AbstractQuery.php(804): Doctrine\ORM\AbstractQuery->execute(NULL, NULL) #23 /websites/distrigoboost/crm-application/src/IZMO/MediaLibraryBundle/Entity/Repository/MediaFileRepository.php(128): Doctrine\ORM\AbstractQuery->getSingleResult() #latform/src/Oro/Bundle/UserBundle/Controller/SecurityController.php(99): IZMO\MediaLibraryBundle\Provider\MediaLibrary->getUrlOnLoginPage(NULL) #26 /websites/dist/crm-application/vendor/oro/platform/src/Oro/Bundle/UserBundle/Controller/SecurityController.php(32): Oro\Bundle\UserBundle\Controller\SecurityController->getBadgesInfo() #27 [internal function]: Oro\Bundle\UserBundle\Controller\SecurityController->loginAction() #28 /websites/dist/crm-application/app/bootstrap.php.cache(3242): call_user_func_array(Array, Array) #29 /websites/diost/crm-application/app/bootstrap.php.cache(3201): Symfony\Component\HttpKernel\HttpKernel->handleRaw(Object(Symfony\Component\HttpFoundation\Request), 1) #30 /websites/dist/crm-application/app/bootstrap.php.cache(3355): Symfony\Component\HttpKernel\HttpKernel->handle(Object(Symfony\Component\HttpFoundation\Request), 1, true) #31 /websites/distrigoboost/crm-application/app/bootstrap.php.cache(2540): Symfony\Component\HttpKernel\DependencyInjection\ContainerAwareHttpKernel->handle(Object(Symfony\Component\HttpFoundation\Request), 1, true) #32 /websites/distrigoboost/crm-application/web/app.php(22): Symfony\Component\HttpKernel\Kernel->handle(Object(Symfony\Component\HttpFoundation\Request)) #33 {main} [] [] ``` Ingest Pipeline Note very carefully you had extra spaces and you use `host` which is a reserved object so it needs to be `host.name` etc This Follows [ECS](https://www.elastic.co/guide/en/ecs/current/ecs-host.html) Ingest Pipeline ```auto PUT _ingest/pipeline/discuss-test { "processors": [ { "dissect": { "field": "message", "pattern": "[%{timestamp}][%{loglevel}][HOST: %{host.name} REQUEST URI: %{request_uri}] %{message_detail}" } } ] } ``` filbeat.yml ```auto filebeat.inputs: # filestream is an input for collecting log messages from files. - type: filestream # Unique ID among all inputs, an ID is required. id: my-filestream-id # Change to true to enable this input configuration. enabled: true # Paths that should be crawled and fetched. Glob based paths. paths: - /Users/sbrown/workspace/sample-data/discuss/discuss-362161.log filebeat.config.modules: # Glob pattern for configuration loading path: ${path.config}/modules.d/*.yml # Set to true to enable config reloading reload.enabled: false setup.template.settings: index.number_of_shards: 1 #index.codec: best_compression #_source.enabled: false setup.kibana: output.elasticsearch: # Array of hosts to connect to. hosts: ["localhost:9200"] pipeline: discuss-test ``` ```auto # RESULT }, "hits": { "total": { "value": 3, "relation": "eq" }, "max_score": 1, "hits": [ { "_index": ".ds-filebeat-8.14.1-2024.07.03-000001", "_id": "dasueZABjwac8wPZA_wr", "_score": 1, "_ignored": [ "message_detail" ], "_source": { "input": { "type": "filestream" }, "agent": { "name": "hyperion", "id": "b9fc4918-2cd8-45d3-a288-68378ca4627b", "type": "filebeat", "ephemeral_id": "327b4def-6306-42c5-a82d-490fd88c11a8", "version": "8.14.1" }, "@timestamp": "2024-07-03T15:19:26.745Z", "ecs": { "version": "8.0.0" }, "log": { "file": { "inode": "114854596", "path": "/Users/sbrown/workspace/sample-data/discuss/discuss-362161.log", "device_id": "16777221" }, "offset": 0 }, "loglevel": "", "host": { "name": " HOST: distqa.izdemosite.com" }, "message": """[2024-06-26 06:56:53][][HOST: distqa.izdemosite.com REQUEST URI: /user/login] app.CRITICAL: Error Occured while getting visit report images:Doctrine\ORM\Query\QueryException: SELECT mf.mediaFileId as id, CONCAT(mf.rootPath, mf.mediaFileName) as url FROM IZMOMediaLibraryBundle:MediaFile mf WHERE mf.mediaFileId = in /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/QueryException.php:41 Stack trace: #0 /websites/diost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(448): Doctrine\ORM\Query\QueryException::dqlError('SELECT mf.media...') #1 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(2649): Doctrine\ORM\Query\Parser->syntaxError('Literal') #2 /websites/disst/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(2837):pplication/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(2739): Doctrine\ORM\Query\Parser->ArithmeticFactor() #5 /websites/distrigoboost/crm-apt/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/AbstractQuery.php(924): Doctrine\ORM\AbstractQuery->exectrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/AbstractQuery.php(804): Doctrine\ORM\AbstractQuery->execute(NULL, NULL) #23 /websites/distrigoboost/crm-application/src/IZMO/MediaLibraryBundle/Entity/Repository/MediaFileRepository.php(128): Doctrine\ORM\AbstractQuery->getSingleResult() #latform/src/Oro/Bundle/UserBundle/Controller/SecurityController.php(99): IZMO\MediaLibraryBundle\Provider\MediaLibrary->getUrlOnLoginPage(NULL) #26 /websites/dist/crm-application/vendor/oro/platform/src/Oro/Bundle/UserBundle/Controller/SecurityController.php(32): Oro\Bundle\UserBundle\Controller\SecurityController->getBadgesInfo() #27 [internal function]: Oro\Bundle\UserBundle\Controller\SecurityController->loginAction() #28 /websites/dist/crm-application/app/bootstrap.php.cache(3242): call_user_func_array(Array, Array) #29 /websites/diost/crm-application/app/bootstrap.php.cache(3201): Symfony\Component\HttpKernel\HttpKernel->handleRaw(Object(Symfony\Component\HttpFoundation\Request), 1) #30 /websites/dist/crm-application/app/bootstrap.php.cache(3355): Symfony\Component\HttpKernel\HttpKernel->handle(Object(Symfony\Component\HttpFoundation\Request), 1, true) #31 /websites/distrigoboost/crm-application/app/bootstrap.php.cache(2540): Symfony\Component\HttpKernel\DependencyInjection\ContainerAwareHttpKernel->handle(Object(Symfony\Component\HttpFoundation\Request), 1, true) #32 /websites/distrigoboost/crm-application/web/app.php(22): Symfony\Component\HttpKernel\Kernel->handle(Object(Symfony\Component\HttpFoundation\Request)) #33 {main} [] []""", "request_uri": "/user/login", "message_detail": """app.CRITICAL: Error Occured while getting visit report images:Doctrine\ORM\Query\QueryException: SELECT mf.mediaFileId as id, CONCAT(mf.rootPath, mf.mediaFileName) as url FROM IZMOMediaLibraryBundle:MediaFile mf WHERE mf.mediaFileId = in /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/QueryException.php:41 Stack trace: #0 /websites/diost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(448): Doctrine\ORM\Query\QueryException::dqlError('SELECT mf.media...') #1 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(2649): Doctrine\ORM\Query\Parser->syntaxError('Literal') #2 /websites/disst/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(2837):pplication/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(2739): Doctrine\ORM\Query\Parser->ArithmeticFactor() #5 /websites/distrigoboost/crm-apt/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/AbstractQuery.php(924): Doctrine\ORM\AbstractQuery->exectrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/AbstractQuery.php(804): Doctrine\ORM\AbstractQuery->execute(NULL, NULL) #23 /websites/distrigoboost/crm-application/src/IZMO/MediaLibraryBundle/Entity/Repository/MediaFileRepository.php(128): Doctrine\ORM\AbstractQuery->getSingleResult() #latform/src/Oro/Bundle/UserBundle/Controller/SecurityController.php(99): IZMO\MediaLibraryBundle\Provider\MediaLibrary->getUrlOnLoginPage(NULL) #26 /websites/dist/crm-application/vendor/oro/platform/src/Oro/Bundle/UserBundle/Controller/SecurityController.php(32): Oro\Bundle\UserBundle\Controller\SecurityController->getBadgesInfo() #27 [internal function]: Oro\Bundle\UserBundle\Controller\SecurityController->loginAction() #28 /websites/dist/crm-application/app/bootstrap.php.cache(3242): call_user_func_array(Array, Array) #29 /websites/diost/crm-application/app/bootstrap.php.cache(3201): Symfony\Component\HttpKernel\HttpKernel->handleRaw(Object(Symfony\Component\HttpFoundation\Request), 1) #30 /websites/dist/crm-application/app/bootstrap.php.cache(3355): Symfony\Component\HttpKernel\HttpKernel->handle(Object(Symfony\Component\HttpFoundation\Request), 1, true) #31 /websites/distrigoboost/crm-application/app/bootstrap.php.cache(2540): Symfony\Component\HttpKernel\DependencyInjection\ContainerAwareHttpKernel->handle(Object(Symfony\Component\HttpFoundation\Request), 1, true) #32 /websites/distrigoboost/crm-application/web/app.php(22): Symfony\Component\HttpKernel\Kernel->handle(Object(Symfony\Component\HttpFoundation\Request)) #33 {main} [] []""", "timestamp": "2024-06-26 06:56:53" } } ``` ---
### Author: ![evangelin](https://avatars.discourse-cdn.com/v4/letter/e/838e76/32.png) [@evangelin](https://discuss.elastic.co/u/evangelin) #### Post date: [July 4, 2024, 7:14am UTC](https://discuss.elastic.co/t/filebeat-is-not-sending-any-logs-to-logstash-please-help-urgent/362161/10 "2024-07-04T07:14:44Z")
version is 8.14.1 i have installed elk in another system and installed elastic agent in endpoint where i'm having the log i have attaches an agent policy to that elastic agent with system integration custom log in that custom log integration i have mentioned the path of the log file for this itself it started to collecting the log from the endpoint but it is storing the whole log entry in messaged field i want the below fields to view ```auto "[%{timestamp}] [%{loglevel}] [HOST: %{host} REQUEST URI: %{request_uri}] %{message}" ``` For that i have tried adding custom configuration like pipeline: custom-pipeline in custom log integration, and in processor session like: ```auto [ { "dissect": { "field": "message", "tokenizer": "[%{timestamp}] [%{loglevel}] [HOST: %{host} REQUEST URI: %{request_uri}] %{message}" } } ] ``` still the index which the custom log integration is creating is not adding those fields which i want. I want it to display in the below format ```auto "host": "distqa.izdemosite.com", "request_uri": "/user/login", "loglevel": "app.CRITICAL", "timestamp": "2024-07-02 06:56:57" "message": """app.CRITICAL: Error Occurred while getting visit report images:Doctrine\ORM\Query\QueryException: SELECT mf.mediaFileId as id, CONCAT(mf.rootPath, mf.mediaFileName) as url FROM IZdiaLibraryBundle:MediaFile mf WHERE mf.mediaFileId = '123' in /websites/distr/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/QueryException.php:41 Stack trace: #0 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(448): Doctrine\ORM\Query\QueryException::dqlError('SELECT mf.media...') #1 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(2649): Doctrine\ORM\Query\Parser->syntaxError('Literal') #2 /websites/distst/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/UserBundle/Controller/SecurityController.php(99): \HttpFoundation\Request), 1) #30 /websites/distt/crm-application/app/bootstrap.php.cache(3355): Symfony\Component\HttpKernel\HttpKernel->handle(Object(Symfony\Component\HttpFoundation\Request), 1, true) #31 /websites/dist/crm-application/app/bootstrap.php.cache(2540): Symfony\Component\HttpKernel\DependencyInjection\ContainerAwareHttpKernel->handle(Object(Symfony\Component\HttpFoundation\Request), 1, true) #32 /websites/dit/crm-application/web/app.php(22): Symfony\Component\HttpKernel\Kernel->handle(Object(Symfony\Component\HttpFoundation\Request)) #33 {main} [] []""", ``` the above format i got by the pipeline and index mapping. IDK how to use that pipeline and index in custom log integration to achieve the above format. i'm trying this from past 1 month Please suggest! to view the logs in the above format 😞 ---
### Author: ![evangelin](https://avatars.discourse-cdn.com/v4/letter/e/838e76/32.png) [@evangelin](https://discuss.elastic.co/u/evangelin) #### Post date: [July 4, 2024, 1:26pm UTC](https://discuss.elastic.co/t/filebeat-is-not-sending-any-logs-to-logstash-please-help-urgent/362161/11 "2024-07-04T13:26:50Z")
This i have tried this sir the logs are getting displayed but only the first 3 are getting displayed in discover after that the log file is storing many logs but those logs are not getting displayed in discover ![image](https://us1.discourse-cdn.com/elastic/original/3X/c/a/caf92c2d7703df5918ff1ef1b7064cbec47bed6a.png) ![image](https://us1.discourse-cdn.com/elastic/original/3X/e/9/e960f79298bc5fbc16c678d52f943945b342f09b.png) why it's not showing the remaining logs in discover tab 😞 ---
### Author: ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb) #### Post date: [July 4, 2024, 2:43pm UTC](https://discuss.elastic.co/t/filebeat-is-not-sending-any-logs-to-logstash-please-help-urgent/362161/12 "2024-07-04T14:43:51Z")
hi @evangelin Well, in my post but I gave you the corrected ingest pipeline. The pipeline you have in the post above **will not work because it does not match your logs pattern** There are extra spaces and also because of the other reasons such as the wrong field name `host` Also, the [dissect processor](https://www.elastic.co/guide/en/elasticsearch/reference/current/dissect-processor.html) has no field called `tokenizer` The field is called `pattern` I think you're confusing and I can understand why the dissect processor that's used with an ingest pipeline versus a [dissect processor that's inside filebeat](https://www.elastic.co/guide/en/beats/filebeat/current/dissect.html) I gave you the code for an ingest pipeline which would work with filebeat or the agent Either way your pattern's not correct and the `host` field is not correct. Why don't you try what I provided and see if that helps. Also I'm confused because one post you're talking about filebeat and now you're talking about agent very difficult to help... We'd love to help, but it's hard when the subject is changing. ---
### Author: ![evangelin](https://avatars.discourse-cdn.com/v4/letter/e/838e76/32.png) [@evangelin](https://discuss.elastic.co/u/evangelin) #### Post date: [July 5, 2024, 12:10pm UTC](https://discuss.elastic.co/t/filebeat-is-not-sending-any-logs-to-logstash-please-help-urgent/362161/13 "2024-07-05T12:10:11Z")
Sorry sir for the confusion. since it is not working i'm trying in many ways to make it work and main things is log entries in that log file are not same as each other one log entry is ```auto [2024-06-21 06:56:53][][HOST: diboostqa.izdemosite.com REQUEST URI: /user/login] app.CRITICAL: Error Occured while getting visit report images:Doctrine\ORM\Query\QueryException: SELECT mf.mediaFileId as id, CONCAT(mf.rootPath, mf.mediaFileName) as url FROM IZMOMediaLibraryBundle:MediaFile mf WHERE mf.mediaFileId = in /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/QueryException.php:41 Stack trace: #0 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(448): Doctrine\ORM\Query\QueryException::dqlError('SELECT mf.media...') #1 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(2649): Doctrine\ORM\Query\Parser->syntaxError('Literal') #2 /websites/di=oost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(2837): Doctrine\ORM\Query\Parser->Literal() #3 /websites/dioboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(2771): Doctrine\ORM\Query\Parser->ArithmeticPrimary() #4 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(2739): Doctrine\ORM\Query\Parser->ArithmeticFactor() #5 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(2713): Doctrine\ORM\Query\Parser->ArithmeticTerm() #6 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(2700): Doctrine\ORM\Query\Parser->SimpleArithmeticExpression() #7 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(3043): Doctrine\ORM\Query\Parser->ArithmeticExpression() #8 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(2564): Doctrine\ORM\Query\Parser->ComparisonExpression() #9 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(2447): Doctrine\ORM\Query\Parser->SimpleConditionalExpression() #10 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(2423): Doctrine\ORM\Query\Parser->ConditionalPrimary() #11 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(2391): Doctrine\ORM\Query\Parser->ConditionalFactor() #12 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(2366): Doctrine\ORM\Query\Parser->ConditionalTerm() #13 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(1333): Doctrine\ORM\Query\Parser->ConditionalExpression() #14 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(876): Doctrine\ORM\Query\Parser->WhereClause() #15 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(843): Doctrine\ORM\Query\Parser->SelectStatement() #16 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(268): Doctrine\ORM\Query\Parser->QueryLanguage() #17 /websites/dboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(351): Doctrine\ORM\Query\Parser->getAST() #18 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query.php(281): Doctrine\ORM\Query\Parser->parse() #19 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query.php(293): Doctrine\ORM\Query->_parse() #20 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/AbstractQuery.php(969): Doctrine\ORM\Query->_doExecute() #21 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/AbstractQuery.php(924): Doctrine\ORM\AbstractQuery->executeIgnoreQueryCache(NULL, NULL) #22 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/AbstractQuery.php(804): Doctrine\ORM\AbstractQuery->execute(NULL, NULL) #23 /websites/distrigoboost/crm-application/src/IZMO/MediaLibraryBundle/Entity/Repository/MediaFileRepository.php(128): Doctrine\ORM\AbstractQuery->getSingleResult() #24 /websites/distrigoboost/crm-application/src/IZMO/MediaLibraryBundle/Provider/MediaLibrary.php(237): IZMO\MediaLibraryBundle\Entity\Repository\MediaFileRepository->getMediaFile(NULL) #25 /websites/distrigoboost/crm-application/vendor/oro/platform/src/Oro/Bundle/UserBundle/Controller/SecurityController.php(99): IZMO\MediaLibraryBundle\Provider\MediaLibrary->getUrlOnLoginPage(NULL) #26 /websites/distrigoboost/crm-application/vendor/oro/platform/src/Oro/Bundle/UserBundle/Controller/SecurityController.php(32): Oro\Bundle\UserBundle\Controller\SecurityController->getBadgesInfo() #27 [internal function]: Oro\Bundle\UserBundle\Controller\SecurityController->loginAction() #28 /websites/distrigoboost/crm-application/app/bootstrap.php.cache(3242): call_user_func_array(Array, Array) #29 /websites/distrigoboost/crm-application/app/bootstrap.php.cache(3201): Symfony\Component\HttpKernel\HttpKernel->handleRaw(Object(Symfony\Component\HttpFoundation\Request), 1) #30 /websites/distrigoboost/crm-application/app/bootstrap.php.cache(3355): Symfony\Component\HttpKernel\HttpKernel->handle(Object(Symfony\Component\HttpFoundation\Request), 1, true) #31 /websites/distrigoboost/crm-application/app/bootstrap.php.cache(2540): Symfony\Component\HttpKernel\DependencyInjection\ContainerAwareHttpKernel->handle(Object(Symfony\Component\HttpFoundation\Request), 1, true) #32 /websites/distrigoboost/crm-application/web/app.php(22): Symfony\Component\HttpKernel\Kernel->handle(Object(Symfony\Component\HttpFoundation\Request)) #33 {main} Next Doctrine\ORM\Query\QueryException: [Syntax Error] line 0, col -1: Error: Expected Literal, got end of string. in /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/QueryException.php:52 Stack trace: #0 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(448): Doctrine\ORM\Query\QueryException::syntaxError('line 0, col -1:...', Object(Doctrine\ORM\Query\QueryException)) #1 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(2649): Doctrine\ORM\Query\Parser->syntaxError('Literal') #2 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(2837): Doctrine\ORM\Query\Parser->Literal() #3 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(2771): Doctrine\ORM\Query\Parser->ArithmeticPrimary() #4 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(2739): Doctrine\ORM\Query\Parser->ArithmeticFactor() #5 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(2713): Doctrine\ORM\Query\Parser->ArithmeticTerm() #6 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(2700): Doctrine\ORM\Query\Parser->SimpleArithmeticExpression() #7 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(3043): Doctrine\ORM\Query\Parser->ArithmeticExpression() #8 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(2564): Doctrine\ORM\Query\Parser->ComparisonExpression() #9 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(2447): Doctrine\ORM\Query\Parser->SimpleConditionalExpression() #10 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(2423): Doctrine\ORM\Query\Parser->ConditionalPrimary() #11 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(2391): Doctrine\ORM\Query\Parser->ConditionalFactor() #12 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(2366): Doctrine\ORM\Query\Parser->ConditionalTerm() #13 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(1333): Doctrine\ORM\Query\Parser->ConditionalExpression() #14 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(876): Doctrine\ORM\Query\Parser->WhereClause() #15 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(843): Doctrine\ORM\Query\Parser->SelectStatement() #16 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(268): Doctrine\ORM\Query\Parser->QueryLanguage() #17 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(351): Doctrine\ORM\Query\Parser->getAST() #18 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query.php(281): Doctrine\ORM\Query\Parser->parse() #19 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query.php(293): Doctrine\ORM\Query->_parse() #20 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/AbstractQuery.php(969): Doctrine\ORM\Query->_doExecute() #21 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/AbstractQuery.php(924): Doctrine\ORM\AbstractQuery->executeIgnoreQueryCache(NULL, NULL) #22 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/AbstractQuery.php(804): Doctrine\ORM\AbstractQuery->execute(NULL, NULL) #23 /websites/distrigoboost/crm-application/src/IZMO/MediaLibraryBundle/Entity/Repository/MediaFileRepository.php(128): Doctrine\ORM\AbstractQuery->getSingleResult() #24 /websites/distrigoboost/crm-application/src/IZMO/MediaLibraryBundle/Provider/MediaLibrary.php(237): IZMO\MediaLibraryBundle\Entity\Repository\MediaFileRepository->getMediaFile(NULL) #25 /websites/distrigoboost/crm-application/vendor/oro/platform/src/Oro/Bundle/UserBundle/Controller/SecurityController.php(99): IZMO\MediaLibraryBundle\Provider\MediaLibrary->getUrlOnLoginPage(NULL) #26 /websites/distrigoboost/crm-application/vendor/oro/platform/src/Oro/Bundle/UserBundle/Controller/SecurityController.php(32): Oro\Bundle\UserBundle\Controller\SecurityController->getBadgesInfo() #27 [internal function]: Oro\Bundle\UserBundle\Controller\SecurityController->loginAction() #28 /websites/distrigoboost/crm-application/app/bootstrap.php.cache(3242): call_user_func_array(Array, Array) #29 /websites/distrigoboost/crm-application/app/bootstrap.php.cache(3201): Symfony\Component\HttpKernel\HttpKernel->handleRaw(Object(Symfony\Component\HttpFoundation\Request), 1) #30 /websites/distrigoboost/crm-application/app/bootstrap.php.cache(3355): Symfony\Component\HttpKernel\HttpKernel->handle(Object(Symfony\Component\HttpFoundation\Request), 1, true) #31 /websites/distrigoboost/crm-application/app/bootstrap.php.cache(2540): Symfony\Component\HttpKernel\DependencyInjection\ContainerAwareHttpKernel->handle(Object(Symfony\Component\HttpFoundation\Request), 1, true) #32 /websites/distrigoboost/crm-application/web/app.php(22): Symfony\Component\HttpKernel\Kernel->handle(Object(Symfony\Component\HttpFoundation\Request)) #33 {main} [] [] ``` and another log entry is ```auto [2024-06-21 06:56:53] app.CRITICAL: Error Occured while getting visit report images:Doctrine\ORM\Query\QueryException: SELECT mf.mediaFileId as id, CONCAT(mf.rootPath, mf.mediaFileName) as url FROM IZMOMediaLibraryBundle:MediaFile mf WHERE mf.mediaFileId = in /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/QueryException.php:41 Stack trace: #0 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(448): Doctrine\ORM\Query\QueryException::dqlError('SELECT mf.media...') #1 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(2649): Doctrine\ORM\Query\Parser->syntaxError('Literal') #2 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(2837): Doctrine\ORM\Query\Parser->Literal() #3 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(2771): Doctrine\ORM\Query\Parser->ArithmeticPrimary() #4 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(2739): Doctrine\ORM\Query\Parser->ArithmeticFactor() #5 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(2713): Doctrine\ORM\Query\Parser->ArithmeticTerm() #6 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(2700): Doctrine\ORM\Query\Parser->SimpleArithmeticExpression() #7 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(3043): Doctrine\ORM\Query\Parser->ArithmeticExpression() #8 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(2564): Doctrine\ORM\Query\Parser->ComparisonExpression() #9 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(2447): Doctrine\ORM\Query\Parser->SimpleConditionalExpression() #10 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(2423): Doctrine\ORM\Query\Parser->ConditionalPrimary() #11 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(2391): Doctrine\ORM\Query\Parser->ConditionalFactor() #12 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(2366): Doctrine\ORM\Query\Parser->ConditionalTerm() #13 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(1333): Doctrine\ORM\Query\Parser->ConditionalExpression() #14 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(876): Doctrine\ORM\Query\Parser->WhereClause() #15 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(843): Doctrine\ORM\Query\Parser->SelectStatement() #16 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(268): Doctrine\ORM\Query\Parser->QueryLanguage() #17 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(351): Doctrine\ORM\Query\Parser->getAST() #18 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query.php(281): Doctrine\ORM\Query\Parser->parse() #19 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query.php(293): Doctrine\ORM\Query->_parse() #20 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/AbstractQuery.php(969): Doctrine\ORM\Query->_doExecute() #21 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/AbstractQuery.php(924): Doctrine\ORM\AbstractQuery->executeIgnoreQueryCache(NULL, NULL) #22 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/AbstractQuery.php(804): Doctrine\ORM\AbstractQuery->execute(NULL, NULL) #23 /websites/distrigoboost/crm-application/src/IZMO/MediaLibraryBundle/Entity/Repository/MediaFileRepository.php(128): Doctrine\ORM\AbstractQuery->getSingleResult() #24 /websites/distrigoboost/crm-application/src/IZMO/MediaLibraryBundle/Provider/MediaLibrary.php(237): IZMO\MediaLibraryBundle\Entity\Repository\MediaFileRepository->getMediaFile(NULL) #25 /websites/distrigoboost/crm-application/vendor/oro/platform/src/Oro/Bundle/UserBundle/Controller/SecurityController.php(99): IZMO\MediaLibraryBundle\Provider\MediaLibrary->getUrlOnLoginPage(NULL) #26 /websites/distrigoboost/crm-application/vendor/oro/platform/src/Oro/Bundle/UserBundle/Controller/SecurityController.php(32): Oro\Bundle\UserBundle\Controller\SecurityController->getBadgesInfo() #27 [internal function]: Oro\Bundle\UserBundle\Controller\SecurityController->loginAction() #28 /websites/distrigoboost/crm-application/app/bootstrap.php.cache(3242): call_user_func_array(Array, Array) #29 /websites/distrigoboost/crm-application/app/bootstrap.php.cache(3201): Symfony\Component\HttpKernel\HttpKernel->handleRaw(Object(Symfony\Component\HttpFoundation\Request), 1) #30 /websites/distrigoboost/crm-application/app/bootstrap.php.cache(3355): Symfony\Component\HttpKernel\HttpKernel->handle(Object(Symfony\Component\HttpFoundation\Request), 1, true) #31 /websites/distrigoboost/crm-application/app/bootstrap.php.cache(2540): Symfony\Component\HttpKernel\DependencyInjection\ContainerAwareHttpKernel->handle(Object(Symfony\Component\HttpFoundation\Request), 1, true) #32 /websites/distrigoboost/crm-application/web/app.php(22): Symfony\Component\HttpKernel\Kernel->handle(Object(Symfony\Component\HttpFoundation\Request)) #33 {main} Next Doctrine\ORM\Query\QueryException: [Syntax Error] line 0, col -1: Error: Expected Literal, got end of string. in /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/QueryException.php:52 Stack trace: #0 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(448): Doctrine\ORM\Query\QueryException::syntaxError('line 0, col -1:...', Object(Doctrine\ORM\Query\QueryException)) #1 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(2649): Doctrine\ORM\Query\Parser->syntaxError('Literal') #2 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(2837): Doctrine\ORM\Query\Parser->Literal() #3 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(2771): Doctrine\ORM\Query\Parser->ArithmeticPrimary() #4 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(2739): Doctrine\ORM\Query\Parser->ArithmeticFactor() #5 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(2713): Doctrine\ORM\Query\Parser->ArithmeticTerm() #6 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(2700): Doctrine\ORM\Query\Parser->SimpleArithmeticExpression() #7 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(3043): Doctrine\ORM\Query\Parser->ArithmeticExpression() #8 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(2564): Doctrine\ORM\Query\Parser->ComparisonExpression() #9 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(2447): Doctrine\ORM\Query\Parser->SimpleConditionalExpression() #10 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(2423): Doctrine\ORM\Query\Parser->ConditionalPrimary() #11 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(2391): Doctrine\ORM\Query\Parser->ConditionalFactor() #12 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(2366): Doctrine\ORM\Query\Parser->ConditionalTerm() #13 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(1333): Doctrine\ORM\Query\Parser->ConditionalExpression() #14 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(876): Doctrine\ORM\Query\Parser->WhereClause() #15 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(843): Doctrine\ORM\Query\Parser->SelectStatement() #16 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(268): Doctrine\ORM\Query\Parser->QueryLanguage() #17 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query/Parser.php(351): Doctrine\ORM\Query\Parser->getAST() #18 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query.php(281): Doctrine\ORM\Query\Parser->parse() #19 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/Query.php(293): Doctrine\ORM\Query->_parse() #20 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/AbstractQuery.php(969): Doctrine\ORM\Query->_doExecute() #21 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/AbstractQuery.php(924): Doctrine\ORM\AbstractQuery->executeIgnoreQueryCache(NULL, NULL) #22 /websites/distrigoboost/crm-application/vendor/doctrine/orm/lib/Doctrine/ORM/AbstractQuery.php(804): Doctrine\ORM\AbstractQuery->execute(NULL, NULL) #23 /websites/distrigoboost/crm-application/src/IZMO/MediaLibraryBundle/Entity/Repository/MediaFileRepository.php(128): Doctrine\ORM\AbstractQuery->getSingleResult() #24 /websites/distrigoboost/crm-application/src/IZMO/MediaLibraryBundle/Provider/MediaLibrary.php(237): IZMO\MediaLibraryBundle\Entity\Repository\MediaFileRepository->getMediaFile(NULL) #25 /websites/distrigoboost/crm-application/vendor/oro/platform/src/Oro/Bundle/UserBundle/Controller/SecurityController.php(99): IZMO\MediaLibraryBundle\Provider\MediaLibrary->getUrlOnLoginPage(NULL) #26 /websites/distrigoboost/crm-application/vendor/oro/platform/src/Oro/Bundle/UserBundle/Controller/SecurityController.php(32): Oro\Bundle\UserBundle\Controller\SecurityController->getBadgesInfo() #27 [internal function]: Oro\Bundle\UserBundle\Controller\SecurityController->loginAction() #28 /websites/distrigoboost/crm-application/app/bootstrap.php.cache(3242): call_user_func_array(Array, Array) #29 /websites/distrigoboost/crm-application/app/bootstrap.php.cache(3201): Symfony\Component\HttpKernel\HttpKernel->handleRaw(Object(Symfony\Component\HttpFoundation\Request), 1) #30 /websites/distrigoboost/crm-application/app/bootstrap.php.cache(3355): Symfony\Component\HttpKernel\HttpKernel->handle(Object(Symfony\Component\HttpFoundation\Request), 1, true) #31 /websites/distrigoboost/crm-application/app/bootstrap.php.cache(2540): Symfony\Component\HttpKernel\DependencyInjection\ContainerAwareHttpKernel->handle(Object(Symfony\Component\HttpFoundation\Request), 1, true) #32 /websites/distrigoboost/crm-application/web/app.php(22): Symfony\Component\HttpKernel\Kernel->handle(Object(Symfony\Component\HttpFoundation\Request)) #33 {main} [] {"user_info":"","url_info":" HOST: distrigoboostqa.izdemosite.com REQUEST URI: /user/login"} ``` this are the frequent log entries apart from this there are other different log entries also but those are not frequent from all those log entries i want to view ( timestamp, loglevel, host, request uri, and message field ) message field with whole log entry, only those fields! ```auto PUT _ingest/pipeline/discuss-test { "processors": [ { "dissect": { "field": "message", "pattern": "[%{timestamp}][%{loglevel}][HOST: %{host.name} REQUEST URI: %{request_uri}] %{message_detail}" } } ] } ``` for this pipeline for first log entry im not able see to loglevel it dispalying empty ![image](https://us1.discourse-cdn.com/elastic/original/3X/5/3/5308a3fe5ab4d6774c150fd67a6bd12244086559.png) for second log entry it giving an error like ```auto { "error": { "root_cause": [ { "type": "find_match", "reason": """Unable to find match for dissect pattern: [%{timestamp}][%{loglevel}][HOST: %{host} REQUEST URI: %{request_uri}] %{message_detail} ``` i want a pipeline which matches for log kind of log entries in that file and from that it have to extract these fields =\> ( timestamp, loglevel, host, request uri, and message field ) ---
### Author: ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb) #### Post date: [July 5, 2024, 8:16pm UTC](https://discuss.elastic.co/t/filebeat-is-not-sending-any-logs-to-logstash-please-help-urgent/362161/14 "2024-07-05T20:16:49Z")
Hi @evangelin No super simple solution. In general You are going to need to create a `grok` pattern match for each log pattern. `grok` takes an array of patterns `dissect` only takes one pattern. Create the most common pattern first, then add the next most common pattern and keep going. ---
### Author: ![Sehran\_Rasool](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sehran_rasool/32/135927_2.png) [@Sehran\_Rasool](https://discuss.elastic.co/u/Sehran_Rasool) #### Post date: [July 10, 2024, 1:53am UTC](https://discuss.elastic.co/t/filebeat-is-not-sending-any-logs-to-logstash-please-help-urgent/362161/15 "2024-07-10T01:53:29Z")
How do you check these logs, I'm encountries the same same issue!