# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=72 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 73 --- ## [How to read the log file continuously and make it as key value in elastic UI](https://discuss.elastic.co/t/how-to-read-the-log-file-continuously-and-make-it-as-key-value-in-elastic-ui/335521)
**Author:** [@subash\_k](https://discuss.elastic.co/u/subash_k)\ **Replies:** 0\ **Last updated:** [June 8, 2023, 9:10am UTC](https://discuss.elastic.co/t/how-to-read-the-log-file-continuously-and-make-it-as-key-value-in-elastic-ui/335521 "2023-06-08T09:10:34Z")
Hi, How can i read the log file continuously and make it as readable in elastic UI, Below example is sample log. I tried with grok but it's not helping that much before few fields rolling in between lines (few log line … --- ## [How to format the grok pattern parsed field value in a new line based on timestamp?](https://discuss.elastic.co/t/how-to-format-the-grok-pattern-parsed-field-value-in-a-new-line-based-on-timestamp/335460)
**Author:** [@abhisheksa](https://discuss.elastic.co/u/abhisheksa)\ **Replies:** 0\ **Last updated:** [June 7, 2023, 6:15pm UTC](https://discuss.elastic.co/t/how-to-format-the-grok-pattern-parsed-field-value-in-a-new-line-based-on-timestamp/335460 "2023-06-07T18:15:42Z")
I have this log message which is filtered using grok pattern. Entire message gets displayed in a single line in the filtered output. { "message": \[ "Calling com.portal.ws.service.GvpV2Service@2ad03f20 method cr… --- ## [How to construct the customized fields from the fluentd output](https://discuss.elastic.co/t/how-to-construct-the-customized-fields-from-the-fluentd-output/335474)
**Author:** [@Chel\_Db](https://discuss.elastic.co/u/Chel_Db)\ **Replies:** 0\ **Last updated:** [June 7, 2023, 7:22pm UTC](https://discuss.elastic.co/t/how-to-construct-the-customized-fields-from-the-fluentd-output/335474 "2023-06-07T19:22:34Z")
I'm capturing the logs from fluentd output onto Logstash using a basic config. input { http { port =\> 8080 } } output { elasticsearch { hosts =\> \["\<%= @ipaddress%\>:9200"\] index =\> "fluentd-%{+YYYY… --- ## [How are logstash grok definitions updated?](https://discuss.elastic.co/t/how-are-logstash-grok-definitions-updated/335452)
**Author:** [@lreger](https://discuss.elastic.co/u/lreger)\ **Replies:** 1\ **Last updated:** [June 7, 2023, 5:35pm UTC](https://discuss.elastic.co/t/how-are-logstash-grok-definitions-updated/335452 "2023-06-07T17:35:41Z")
How do I find out what my current version of logstash core patterns are running on my logstash cluster? I am running 7.17.1, but I suspect I am not running grok core patterns 4.34 ecsv1. I would like to have access to s… --- ## [Fetching all external IP address from firewall logs using logstash](https://discuss.elastic.co/t/fetching-all-external-ip-address-from-firewall-logs-using-logstash/334934)
**Author:** [@libinmath](https://discuss.elastic.co/u/libinmath)\ **Replies:** 5\ **Last updated:** [June 7, 2023, 4:27pm UTC](https://discuss.elastic.co/t/fetching-all-external-ip-address-from-firewall-logs-using-logstash/334934 "2023-06-07T16:27:27Z")
I am working with fortinet firewall logs, trying to get all external IP address from the fields srcip and dstip into a text file. I am new to writing filters for the logstash. The sample documents are as follow but I am… --- ## [How to queue ECS formatted logs through RabbitMQ](https://discuss.elastic.co/t/how-to-queue-ecs-formatted-logs-through-rabbitmq/335105)
**Author:** [@bvoros](https://discuss.elastic.co/u/bvoros)\ **Replies:** 3\ **Last updated:** [June 7, 2023, 1:48pm UTC](https://discuss.elastic.co/t/how-to-queue-ecs-formatted-logs-through-rabbitmq/335105 "2023-06-07T13:48:16Z")
Hello all, Our logging infrastructure is the following: log shippers -\> logstash -\> rabbitmq -\> logstash -\> elasticsearch I am trying to start using ECS, have the template set up. However, when the first logstash plac… --- ## [Schema Registry integration with Logstash kafka input plugin](https://discuss.elastic.co/t/schema-registry-integration-with-logstash-kafka-input-plugin/335431)
**Author:** [@Hemanth\_Gowda](https://discuss.elastic.co/u/Hemanth_Gowda)\ **Replies:** 0\ **Last updated:** [June 7, 2023, 10:41am UTC](https://discuss.elastic.co/t/schema-registry-integration-with-logstash-kafka-input-plugin/335431 "2023-06-07T10:41:34Z")
Hi All, We are trying to setup Kafka Schema registry integration with Logstash. However we have below questions to understand before we start with. Can someone please help with this. We have multiple dynamic schemas … --- ## [Recommended RDMS ingestion approach can lead to lost updates](https://discuss.elastic.co/t/recommended-rdms-ingestion-approach-can-lead-to-lost-updates/332664)
**Author:** [@Alex\_McAusland](https://discuss.elastic.co/u/Alex_McAusland)\ **Replies:** 4\ **Last updated:** [June 7, 2023, 10:09am UTC](https://discuss.elastic.co/t/recommended-rdms-ingestion-approach-can-lead-to-lost-updates/332664 "2023-06-07T10:09:59Z")
The official RDMS ingestion docs recommend an approach based on tracking row modification time in the sql\_last\_value of the jdbc plugin. However this does not seem to account for database transactions; a row's modificat… --- ## [Logstash mysql](https://discuss.elastic.co/t/logstash-mysql/335400)
**Author:** [@adimi\_worou](https://discuss.elastic.co/u/adimi_worou)\ **Replies:** 4\ **Last updated:** [June 7, 2023, 9:56am UTC](https://discuss.elastic.co/t/logstash-mysql/335400 "2023-06-07T09:56:29Z")
Hi, i’ve the same problem. Logstash can’t load data from mysql db to elasticsearch. I use docker. Thanks for your help --- ## [Auditbeat \>=8, logstash, and elasticsearch data stream](https://discuss.elastic.co/t/auditbeat-8-logstash-and-elasticsearch-data-stream/335357)
**Author:** [@Mike\_Williams](https://discuss.elastic.co/u/Mike_Williams)\ **Replies:** 2\ **Last updated:** [June 7, 2023, 9:37am UTC](https://discuss.elastic.co/t/auditbeat-8-logstash-and-elasticsearch-data-stream/335357 "2023-06-07T09:37:50Z")
Hey, I'm preparing to upgrade a set of auditbeat agents from 7.17 to 8.something. Clients are not allowed to talk directly to elasticsearch, all messages go through logstash. More than happy with the requirement to us… --- ## [Can't (yet) decode flowset id 256 from source id 0, because no template to decode it with has been received. This message will usually go away after 1 minute on logstash 7.17 and elasticsearch 7.17](https://discuss.elastic.co/t/cant-yet-decode-flowset-id-256-from-source-id-0-because-no-template-to-decode-it-with-has-been-received-this-message-will-usually-go-away-after-1-minute-on-logstash-7-17-and-elasticsearch-7-17/335421)
**Author:** [@Hanginium65](https://discuss.elastic.co/u/Hanginium65)\ **Replies:** 0\ **Last updated:** [June 7, 2023, 9:32am UTC](https://discuss.elastic.co/t/cant-yet-decode-flowset-id-256-from-source-id-0-because-no-template-to-decode-it-with-has-been-received-this-message-will-usually-go-away-after-1-minute-on-logstash-7-17-and-elasticsearch-7-17/335421 "2023-06-07T09:32:33Z")
Hi, my config file for logstash looks like this: input { snmp { hosts =\> \[{host =\> "udp:192.168.56.3/161" version =\> "3"}\] get =\> \["1.3.6.1.2.1.25.3.3.1.2.1", "1.3.6.1.2.1.25.2.3.1.5.65536", "1.3.6.1.2.1.25.2… --- ## [JDBC INPUT plugin not syncing all eligible records from postgres db to elasticsearch](https://discuss.elastic.co/t/jdbc-input-plugin-not-syncing-all-eligible-records-from-postgres-db-to-elasticsearch/335409)
**Author:** [@Gio\_Vanni](https://discuss.elastic.co/u/Gio_Vanni)\ **Replies:** 0\ **Last updated:** [June 7, 2023, 8:43am UTC](https://discuss.elastic.co/t/jdbc-input-plugin-not-syncing-all-eligible-records-from-postgres-db-to-elasticsearch/335409 "2023-06-07T08:43:00Z")
Hi I have an issue whereby logstash doesn't update all records that are returned by the jdbc-input query to Elasticsearch.As a result we always have to restart logstash to force through the updates. input plugin config: … --- ## [Can't get text on a START\_OBJECT at 1:34](https://discuss.elastic.co/t/cant-get-text-on-a-start-object-at-1-34/335399)
**Author:** [@hackercat](https://discuss.elastic.co/u/hackercat)\ **Replies:** 0\ **Last updated:** [June 7, 2023, 7:40am UTC](https://discuss.elastic.co/t/cant-get-text-on-a-start-object-at-1-34/335399 "2023-06-07T07:40:52Z")
Hi everyone, I recently upgraded ELK from 7 to 8 and it was working fine for v7, but since v8, it continuously gave me the below error. Jun 07 16:48:00 gitlab-logger logstash\[115245\]: \[2023-06-07T16:48:00,346\]\[WARN \]\[l… --- ## [Certificate error when installing logstash plugin](https://discuss.elastic.co/t/certificate-error-when-installing-logstash-plugin/335391)
**Author:** [@fsaa](https://discuss.elastic.co/u/fsaa)\ **Replies:** 0\ **Last updated:** [June 7, 2023, 5:23am UTC](https://discuss.elastic.co/t/certificate-error-when-installing-logstash-plugin/335391 "2023-06-07T05:23:11Z")
I'm using a VPN because I'm using a company laptop, when I run the command RUN bin/logstash-plugin install logstash-input-sftp.zip in the DockerFile I get this error: =\> \[6/7\] RUN zip -r logstash-input-sftp.zip logstash… --- ## [Logstash not pulling data fast enough from Kafka](https://discuss.elastic.co/t/logstash-not-pulling-data-fast-enough-from-kafka/334377)
**Author:** [@Francisco\_Yanez](https://discuss.elastic.co/u/Francisco_Yanez)\ **Replies:** 3\ **Last updated:** [June 6, 2023, 10:43pm UTC](https://discuss.elastic.co/t/logstash-not-pulling-data-fast-enough-from-kafka/334377 "2023-06-06T22:43:48Z")
I have a huge problem. My kafka is on a different DC and we are using logstash to pull data. Our Elastic stack is running in kubernetes but our data is getting pulled very slow. How can I optimize logstash to pull data f… --- ## [Palo Alto Networks - Logstash \> Elastic \> Kibana](https://discuss.elastic.co/t/palo-alto-networks-logstash-elastic-kibana/335380)
**Author:** [@thunt](https://discuss.elastic.co/u/thunt)\ **Replies:** 7\ **Last updated:** [June 6, 2023, 9:16pm UTC](https://discuss.elastic.co/t/palo-alto-networks-logstash-elastic-kibana/335380 "2023-06-06T21:16:01Z")
Hello Everyone - Hoping I have a simple solution. Testing out Elastic Stack with Palo Alto syslogs, and running into issues with GeoIP's and combining the lon/lat to use Maps in Kibana. Not sure what else needs to be d… --- ## [Grok multi-line mode](https://discuss.elastic.co/t/grok-multi-line-mode/335101)
**Author:** [@M\_D](https://discuss.elastic.co/u/M_D)\ **Replies:** 7\ **Last updated:** [June 6, 2023, 7:44pm UTC](https://discuss.elastic.co/t/grok-multi-line-mode/335101 "2023-06-06T19:44:27Z")
I am using (?ms) in my grok filter, but got an error (see RegexpError: undefined). What should be the right way to lookup multiple lines using grok? in regular regex i amd doing (?sm)(?\\[0-9\]{4}-\[0-9\]{2}-\[0-9… --- ## [Log4j2 vulnerability mitigation - JndiLookup Removal](https://discuss.elastic.co/t/log4j2-vulnerability-mitigation-jndilookup-removal/335356)
**Author:** [@JosephAnis](https://discuss.elastic.co/u/JosephAnis)\ **Replies:** 1\ **Last updated:** [June 6, 2023, 6:43pm UTC](https://discuss.elastic.co/t/log4j2-vulnerability-mitigation-jndilookup-removal/335356 "2023-06-06T18:43:49Z")
Hi All, We are working on mitigating the Log4j2 vulnerability by removing the JndiLookup class as described here: We are using version 7.9.2 for all ELK components and currently we can't upgrade to newer version. My … --- ## [Logstash unable to collect logs from filebeat due to protocol mismatch](https://discuss.elastic.co/t/logstash-unable-to-collect-logs-from-filebeat-due-to-protocol-mismatch/335269)
**Author:** [@Chel\_Db](https://discuss.elastic.co/u/Chel_Db)\ **Replies:** 16\ **Last updated:** [June 6, 2023, 4:31pm UTC](https://discuss.elastic.co/t/logstash-unable-to-collect-logs-from-filebeat-due-to-protocol-mismatch/335269 "2023-06-06T16:31:33Z")
I've installed filebeat in our k8s following official elastic document (kubernetes/filebeat-kubernetes.yaml ) to collect logs of our microservices and push it to the Logstash which is installed in a different VM as a co… --- ## [CVE-2022-30123 - Rack Vulnerability](https://discuss.elastic.co/t/cve-2022-30123-rack-vulnerability/335274)
**Author:** [@priya\_dhana](https://discuss.elastic.co/u/priya_dhana)\ **Replies:** 1\ **Last updated:** [June 6, 2023, 5:46am UTC](https://discuss.elastic.co/t/cve-2022-30123-rack-vulnerability/335274 "2023-06-06T05:46:47Z")
Security Scan has flagged Critical CVE-2022-30123 Rack::RELEASE in the logstash 8.7.1 tar file. How can we remove rack or upgrade to a newer version? Thanks, Priya V --- ## [JDBC input error when using schedule without last run](https://discuss.elastic.co/t/jdbc-input-error-when-using-schedule-without-last-run/335243)
**Author:** [@ElasticLiver](https://discuss.elastic.co/u/ElasticLiver)\ **Replies:** 3\ **Last updated:** [June 5, 2023, 10:19pm UTC](https://discuss.elastic.co/t/jdbc-input-error-when-using-schedule-without-last-run/335243 "2023-06-05T22:19:11Z")
Hi, I need to query a database every 1 minute and get all the results of the query, so I use schedule but no last\_run\_metadata\_path. logstash give an error, but still que the data indexed in ES. logstash look for this… --- ## [Log4j2 vulnerability mitigation](https://discuss.elastic.co/t/log4j2-vulnerability-mitigation/335213)
**Author:** [@mostafaelsayed](https://discuss.elastic.co/u/mostafaelsayed)\ **Replies:** 6\ **Last updated:** [June 5, 2023, 3:33pm UTC](https://discuss.elastic.co/t/log4j2-vulnerability-mitigation/335213 "2023-06-05T15:33:18Z")
Hello all, I was checking the actions needed from our side in the ELK cluster to mitigate the Log4j2 vulnerability found in Dec 2021. we are using 7.9.2 for all ELK components. After investigating and checking the below… --- ## [Considering using L4 or kafka](https://discuss.elastic.co/t/considering-using-l4-or-kafka/335238)
**Author:** [@a01066278824](https://discuss.elastic.co/u/a01066278824)\ **Replies:** 1\ **Last updated:** [June 5, 2023, 3:15pm UTC](https://discuss.elastic.co/t/considering-using-l4-or-kafka/335238 "2023-06-05T15:15:30Z")
im considering two ways. first, using L4 between Beats and logstash. second, using Kafka between beats and logstahs. which way is more effective one? and im wondering if is it possible Beats - Kafka - L4 - Logstash. … --- ## [Enriching data with ProxyIP database](https://discuss.elastic.co/t/enriching-data-with-proxyip-database/335169)
**Author:** [@Hitz2403](https://discuss.elastic.co/u/Hitz2403)\ **Replies:** 5\ **Last updated:** [June 5, 2023, 8:50am UTC](https://discuss.elastic.co/t/enriching-data-with-proxyip-database/335169 "2023-06-05T08:50:40Z")
Hi everyone, I need help enriching data with IP Proxy database like geoip plugin, has anyone done this before? Docs or something can help? --- ## [Grok on logstash not working](https://discuss.elastic.co/t/grok-on-logstash-not-working/334172)
**Author:** [@nitisha](https://discuss.elastic.co/u/nitisha)\ **Replies:** 0\ **Last updated:** [May 24, 2023, 6:17am UTC](https://discuss.elastic.co/t/grok-on-logstash-not-working/334172 "2023-05-24T06:17:13Z")
Hi, I am monitoring CPU metric threshold of containers in our infra using elasticsearch using Connector as "Server Log" which defaults to kibana.log. I have created the following logstash configuration file to intercep… --- ## [Configure es with logstash](https://discuss.elastic.co/t/configure-es-with-logstash/334604)
**Author:** [@sujata\_g](https://discuss.elastic.co/u/sujata_g)\ **Replies:** 6\ **Last updated:** [June 4, 2023, 10:25am UTC](https://discuss.elastic.co/t/configure-es-with-logstash/334604 "2023-06-04T10:25:52Z")
input { s3 { access\_key\_id =\> "" secret\_access\_key =\> "" bucket =\> "dumpsampleperigon" region =\> "us-west-1" } } output { elasticsearch { hosts =\> \["http://elasticsearch:9200"\] index =\> "logs-%{+YYYY.MM.dd}" … --- ## [Logstash giving error which is not clear](https://discuss.elastic.co/t/logstash-giving-error-which-is-not-clear/335126)
**Author:** [@Patr123](https://discuss.elastic.co/u/Patr123)\ **Replies:** 7\ **Last updated:** [June 3, 2023, 7:06pm UTC](https://discuss.elastic.co/t/logstash-giving-error-which-is-not-clear/335126 "2023-06-03T19:06:59Z")
I am getting the following error in logstash-plain.log: \[2023-06-03T01:33:34,256\]\[INFO \]\[logstash.runner \] Log4j configuration path used is: /etc/logstash/log4j2.properties \[2023-06-03T01:33:34,272\]\[INFO \]\[logs… --- ## [Grouping And Ordering Log is Posible?](https://discuss.elastic.co/t/grouping-and-ordering-log-is-posible/335017)
**Author:** [@aidensV](https://discuss.elastic.co/u/aidensV)\ **Replies:** 2\ **Last updated:** [June 3, 2023, 1:11pm UTC](https://discuss.elastic.co/t/grouping-and-ordering-log-is-posible/335017 "2023-06-03T13:11:30Z")
I have Log with example : (Case 1) CHAN1 : 23:57:05:89 |Message Start CHAN1 : 23:57:05:89 |Lorem CHAN1 : 23:57:05:89 |Ipsum CHAN1 : 23:57:05:89 |Dolor CHAN99i : 23:57:05:89 |Message Start CHAN99i : 23:57:05:89 |Lo… --- ## [ No config files found in path {:path=\>"/etc/logstash/conf.d/\*.conf"}](https://discuss.elastic.co/t/no-config-files-found-in-path-path-etc-logstash-conf-d-conf/335106)
**Author:** [@karma\_services](https://discuss.elastic.co/u/karma_services)\ **Replies:** 1\ **Last updated:** [June 3, 2023, 5:28am UTC](https://discuss.elastic.co/t/no-config-files-found-in-path-path-etc-logstash-conf-d-conf/335106 "2023-06-03T05:28:55Z")
I have installed ELK stack via debian package on Ubuntu Server. I want to send pfsense logs to logstash. File Settings: 1- /etc/logstash/conf.d/syslog.conf input { tcp { port =\> 514 type =\> "pfsense" } udp { … --- ## [RegexpError: undefined](https://discuss.elastic.co/t/regexperror-undefined/334714)
**Author:** [@M\_D](https://discuss.elastic.co/u/M_D)\ **Replies:** 4\ **Last updated:** [June 2, 2023, 2:37pm UTC](https://discuss.elastic.co/t/regexperror-undefined/334714 "2023-06-02T14:37:30Z")
I am getting the following error using logstash:8.6.2 docker image: \[2023-05-30T18:42:18,144\]\[ERROR\]\[logstash.javapipeline \]\[main\] Pipeline error {:pipeline\_id=\>"main", :exception=\>#\