# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=332 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 333 --- ## [Time difference in last\_run\_metadata\_path file when Logstash and source database are in different timezones](https://discuss.elastic.co/t/time-difference-in-last-run-metadata-path-file-when-logstash-and-source-database-are-in-different-timezones/240190)
**Author:** [@saif3r](https://discuss.elastic.co/u/saif3r)\ **Replies:** 3\ **Last updated:** [July 9, 2020, 7:35am UTC](https://discuss.elastic.co/t/time-difference-in-last-run-metadata-path-file-when-logstash-and-source-database-are-in-different-timezones/240190 "2020-07-09T07:35:29Z")
Hello, Our ELK Stack is placed in UTC timezone (OS timezone is set to UTC). The database from which we pull the data from using JDBC is in CET (UTC+2) timezone. The config file for the pipeline looks like this: jdbc … --- ## [Question in using elasticsearch-certutil to generate cert](https://discuss.elastic.co/t/question-in-using-elasticsearch-certutil-to-generate-cert/240433)
**Author:** [@azteker](https://discuss.elastic.co/u/azteker)\ **Replies:** 0\ **Last updated:** [July 8, 2020, 9:46pm UTC](https://discuss.elastic.co/t/question-in-using-elasticsearch-certutil-to-generate-cert/240433 "2020-07-08T21:46:04Z")
I have generated elastic-stack-ca.p12 Then I use this command to generate crt and key files for logstash. ./bin/elasticsearch-certutil cert --ca elastic-stack-ca.p12 --name logstash --dns \[?\] --ip \[elk server's ip\] --p… --- ## [Parsing Error Logs with line breaks](https://discuss.elastic.co/t/parsing-error-logs-with-line-breaks/240419)
**Author:** [@milkene](https://discuss.elastic.co/u/milkene)\ **Replies:** 1\ **Last updated:** [July 8, 2020, 7:49pm UTC](https://discuss.elastic.co/t/parsing-error-logs-with-line-breaks/240419 "2020-07-08T19:49:27Z")
Hi, I am trying to parse logs with Logstash using Grok filters. Most of the logs are a single line but some have line breaks and spaces between them. Is there a way to get Logstash to recognize them as one log rather tha… --- ## [Flatten json and create dynamic field](https://discuss.elastic.co/t/flatten-json-and-create-dynamic-field/240360)
**Author:** [@elasticforme](https://discuss.elastic.co/u/elasticforme)\ **Replies:** 9\ **Last updated:** [July 8, 2020, 5:03pm UTC](https://discuss.elastic.co/t/flatten-json-and-create-dynamic-field/240360 "2020-07-08T17:03:23Z")
I know ruby will be involve I have try few code that I found on forum. but didn't work out here is my input message =\> '{"jobid":348332180, "runtime":{"cputime": 24, "totalelapsed": 105},"dataflow":\[{"gate": "v204.06",… --- ## [Logstash windows memory offset delete](https://discuss.elastic.co/t/logstash-windows-memory-offset-delete/240347)
**Author:** [@amaleswar](https://discuss.elastic.co/u/amaleswar)\ **Replies:** 4\ **Last updated:** [July 8, 2020, 4:17pm UTC](https://discuss.elastic.co/t/logstash-windows-memory-offset-delete/240347 "2020-07-08T16:17:28Z")
Hello Team, I want to delete anything after dll, example showing below. I\`m trying to use mutate gsub but failing Parsed data: "CallTrace" =\> \[ \[ 0\] "c:\\windows\\system32\\nll.dll+92c34", \[ 1\] "c:\\windows\\system32\\ker… --- ## [Grouping of similar errors](https://discuss.elastic.co/t/grouping-of-similar-errors/240399)
**Author:** [@Shloka](https://discuss.elastic.co/u/Shloka)\ **Replies:** 0\ **Last updated:** [July 8, 2020, 4:13pm UTC](https://discuss.elastic.co/t/grouping-of-similar-errors/240399 "2020-07-08T16:13:49Z")
Hi I'm looking for help with logstash configuration We have filebeat on the target servers which reads the log lines and sends to logstash installed on another central server. The logstash further filters the lines bas… --- ## [What will be the best way to process OAM logs](https://discuss.elastic.co/t/what-will-be-the-best-way-to-process-oam-logs/240371)
**Author:** [@Ashish\_kapoor](https://discuss.elastic.co/u/Ashish_kapoor)\ **Replies:** 0\ **Last updated:** [July 8, 2020, 2:46pm UTC](https://discuss.elastic.co/t/what-will-be-the-best-way-to-process-oam-logs/240371 "2020-07-08T14:46:28Z")
Hi, I am trying to Process below logs. These logs look likes CSV logs. But I am not able to use CSV filter as in "NewSetting field", data is not properly formated for some logs("AUTHZ-POLICY:Policy name="Protected Reso… --- ## [Logstash filtering iterating over list](https://discuss.elastic.co/t/logstash-filtering-iterating-over-list/240364)
**Author:** [@ash2](https://discuss.elastic.co/u/ash2)\ **Replies:** 1\ **Last updated:** [July 8, 2020, 2:46pm UTC](https://discuss.elastic.co/t/logstash-filtering-iterating-over-list/240364 "2020-07-08T14:46:04Z")
I am trying to parse the following format of logs, starting value is a hostname, and the ending is timestamp always. blb12bkl4, data1 value1, data2 value2, data3 value3, 1594216629094 and the final format for output sh… --- ## [Logstash windows scheduler does not stop](https://discuss.elastic.co/t/logstash-windows-scheduler-does-not-stop/240344)
**Author:** [@Kiran\_Bhadani](https://discuss.elastic.co/u/Kiran_Bhadani)\ **Replies:** 0\ **Last updated:** [July 8, 2020, 12:28pm UTC](https://discuss.elastic.co/t/logstash-windows-scheduler-does-not-stop/240344 "2020-07-08T12:28:54Z")
I have setup windows task scheduler to run log stash, when I stop the task from task scheduler, logstash still keeps running in background --- ## [Logstash unable to connect to Redshift](https://discuss.elastic.co/t/logstash-unable-to-connect-to-redshift/240338)
**Author:** [@abhisheksimion](https://discuss.elastic.co/u/abhisheksimion)\ **Replies:** 0\ **Last updated:** [July 8, 2020, 11:56am UTC](https://discuss.elastic.co/t/logstash-unable-to-connect-to-redshift/240338 "2020-07-08T11:56:20Z")
I am trying to load data from Amazon Redshift, but it always exists Logstash process by throwing below exception. Stacktrace \[2020-07-08T11:46:30,170\]\[INFO \]\[logstash.agent \] Pipelines running {:count=\>1, :… --- ## [How to use dot codec in Logstash?](https://discuss.elastic.co/t/how-to-use-dot-codec-in-logstash/240291)
**Author:** [@abhisheksimion](https://discuss.elastic.co/u/abhisheksimion)\ **Replies:** 2\ **Last updated:** [July 8, 2020, 9:31am UTC](https://discuss.elastic.co/t/how-to-use-dot-codec-in-logstash/240291 "2020-07-08T09:31:51Z")
I am trying to use the dot codec in Logstash, but it gives below error \[ERROR\]\[logstash.plugins.registry\] Tried to load a plugin's code, but failed. {:exception=\>#\… --- ## [Logstash-output-neo4j: Installation successful But NOT work](https://discuss.elastic.co/t/logstash-output-neo4j-installation-successful-but-not-work/240280)
**Author:** [@Aaronzk](https://discuss.elastic.co/u/Aaronzk)\ **Replies:** 0\ **Last updated:** [July 8, 2020, 7:15am UTC](https://discuss.elastic.co/t/logstash-output-neo4j-installation-successful-but-not-work/240280 "2020-07-08T07:15:23Z")
Hello Everyone, We are solving a use case where we'd benefit from sending some connected data via logstash into neo4j, but we don't yet know how. We have tried to install: but it would not even appear in the list … --- ## [Can't get geoip working](https://discuss.elastic.co/t/cant-get-geoip-working/238574)
**Author:** [@danona45](https://discuss.elastic.co/u/danona45)\ **Replies:** 9\ **Last updated:** [July 8, 2020, 2:28am UTC](https://discuss.elastic.co/t/cant-get-geoip-working/238574 "2020-07-08T02:28:00Z")
I know there are gazillion other posts but I have tried pretty much anything at this point. Background: I am using AWS Opendistro and send events using Logstash both versions 7.4.0 Indexing template: { "pano… --- ## [Logstash with kafka plugin not get messages in a new topic](https://discuss.elastic.co/t/logstash-with-kafka-plugin-not-get-messages-in-a-new-topic/238655)
**Author:** [@Altamir\_Dias](https://discuss.elastic.co/u/Altamir_Dias)\ **Replies:** 1\ **Last updated:** [July 7, 2020, 11:23pm UTC](https://discuss.elastic.co/t/logstash-with-kafka-plugin-not-get-messages-in-a-new-topic/238655 "2020-07-07T23:23:49Z")
Hello guys, I need a help in specific case. My environment have a DLL application who create messages and topics in Kafka, than logstash get messages of the all topics. But problem is new topic only collect when a next… --- ## [Full pipeline Logstash with dead letter queue](https://discuss.elastic.co/t/full-pipeline-logstash-with-dead-letter-queue/239855)
**Author:** [@Altamir\_Dias](https://discuss.elastic.co/u/Altamir_Dias)\ **Replies:** 2\ **Last updated:** [July 7, 2020, 11:14pm UTC](https://discuss.elastic.co/t/full-pipeline-logstash-with-dead-letter-queue/239855 "2020-07-07T23:14:39Z")
Hello Guys, I have a pipeline in my ELK that get data from Kafka and send Elasticsearch. But I don't know how I can set the dead letter queue. My goal is every errors be send a topic called dlq+date. I tried several so… --- ## [Logstash7.6 - data loss when input file logrotation happens](https://discuss.elastic.co/t/logstash7-6-data-loss-when-input-file-logrotation-happens/239719)
**Author:** [@manu1](https://discuss.elastic.co/u/manu1)\ **Replies:** 3\ **Last updated:** [July 7, 2020, 10:09pm UTC](https://discuss.elastic.co/t/logstash7-6-data-loss-when-input-file-logrotation-happens/239719 "2020-07-07T22:09:28Z")
Hello, We have observed data loss when input file's logrotation happens every day. input { file { path =\> "/weblogs/biweb.log" sincedb\_path =\> "/opt/logshipper/ls7\_bi/sincedbs/biweb.db" … --- ## [Grock Parse Date and time](https://discuss.elastic.co/t/grock-parse-date-and-time/240216)
**Author:** [@ElasticLiver](https://discuss.elastic.co/u/ElasticLiver)\ **Replies:** 1\ **Last updated:** [July 7, 2020, 8:05pm UTC](https://discuss.elastic.co/t/grock-parse-date-and-time/240216 "2020-07-07T20:05:58Z")
Hi, Im currently parsing the date an time on this log: \<187\>Jul 07 2020 11:29:55: with this Grok \<%{POSINT:syslog\_pri}\>%{MONTH:mes} %{MONTHDAY:dia} %{YEAR:año} %{TIME:hora}: Is there a better way to do this? because… --- ## [Beats to Logstash Connectivity](https://discuss.elastic.co/t/beats-to-logstash-connectivity/240198)
**Author:** [@robinhood\_lko](https://discuss.elastic.co/u/robinhood_lko)\ **Replies:** 3\ **Last updated:** [July 7, 2020, 7:50pm UTC](https://discuss.elastic.co/t/beats-to-logstash-connectivity/240198 "2020-07-07T19:50:07Z")
Hi Team I am facing one weird issue when receiving connection on Logstash from filebeats. Logstash Version : 7.7.0 Beats version: 7.7.0 Connection - Beats -\> Traefik LB -\> Logstash Service -\> Logstash Configuration… --- ## [Error to Push CQV files from Logstash to Elastic](https://discuss.elastic.co/t/error-to-push-cqv-files-from-logstash-to-elastic/239887)
**Author:** [@zizou90](https://discuss.elastic.co/u/zizou90)\ **Replies:** 1\ **Last updated:** [July 7, 2020, 10:36am UTC](https://discuss.elastic.co/t/error-to-push-cqv-files-from-logstash-to-elastic/239887 "2020-07-07T10:36:39Z")
Helllo, I have 5 CSV files (separator : , ) with the same structures but the last one haven't the same order of columns : example : the columns of the Files number 1 2 3 4 like this : Price |Date | Quantity | The f… --- ## [The correct use of logstash](https://discuss.elastic.co/t/the-correct-use-of-logstash/239823)
**Author:** [@Katia](https://discuss.elastic.co/u/Katia)\ **Replies:** 5\ **Last updated:** [July 7, 2020, 12:59pm UTC](https://discuss.elastic.co/t/the-correct-use-of-logstash/239823 "2020-07-07T12:59:03Z")
I have four types of log files. Every day I get the 4 types from several machines and, I have for each machine : t1-20200703, t2-20200703, t3-20200703 and t4-20200703. After a few days I will have hundreds of files and… --- ## [How to add the value count of 2 search results in Logstash](https://discuss.elastic.co/t/how-to-add-the-value-count-of-2-search-results-in-logstash/240153)
**Author:** [@Matish\_Bhuyan](https://discuss.elastic.co/u/Matish_Bhuyan)\ **Replies:** 1\ **Last updated:** [July 7, 2020, 11:51am UTC](https://discuss.elastic.co/t/how-to-add-the-value-count-of-2-search-results-in-logstash/240153 "2020-07-07T11:51:35Z")
Hi All, I have written a scripted filed to add the count of these 2 values Identity Accepted and Identity Rejected. These values are searched under result keyword this is my scripted field if (doc\['result'\].value ==… --- ## [JMS Input Plugin for Logstash - Tibco JMS Provider](https://discuss.elastic.co/t/jms-input-plugin-for-logstash-tibco-jms-provider/240151)
**Author:** [@rohitdgp](https://discuss.elastic.co/u/rohitdgp)\ **Replies:** 0\ **Last updated:** [July 7, 2020, 11:42am UTC](https://discuss.elastic.co/t/jms-input-plugin-for-logstash-tibco-jms-provider/240151 "2020-07-07T11:42:47Z")
I was reading through JMS as input plugin for Logstash. Post reading the documentation i was trying to setup a Logstash EMS listener for Tibco JMS provider with JNDI configuration. Although the official elastic-matrix … --- ## [How to create map with 0 length](https://discuss.elastic.co/t/how-to-create-map-with-0-length/240093)
**Author:** [@Deepak\_Dandotiya](https://discuss.elastic.co/u/Deepak_Dandotiya)\ **Replies:** 2\ **Last updated:** [July 7, 2020, 11:42am UTC](https://discuss.elastic.co/t/how-to-create-map-with-0-length/240093 "2020-07-07T11:42:29Z")
Hi All, I need one help. how to create map with 0 length if data is null for that map fields. I am giving one example where student\_key is PK and for a student creating SUBJECT as map so if all column data is null then… --- ## [Pipeline is being aborted due to logstash\_jdbc\_last\_run File Not Found](https://discuss.elastic.co/t/pipeline-is-being-aborted-due-to-logstash-jdbc-last-run-file-not-found/240029)
**Author:** [@anurag397](https://discuss.elastic.co/u/anurag397)\ **Replies:** 2\ **Last updated:** [July 7, 2020, 11:27am UTC](https://discuss.elastic.co/t/pipeline-is-being-aborted-due-to-logstash-jdbc-last-run-file-not-found/240029 "2020-07-07T11:27:52Z")
I have two pipelines which sync data from different tables in a postgresql database. - pipeline.id: logstash\_test\_user\_table path.config: "/usr/share/logstash/pipeline/logstash.conf" pipeline.workers: 1 - pipeline.i… --- ## [Rsyslog + Beats together](https://discuss.elastic.co/t/rsyslog-beats-together/239994)
**Author:** [@akam](https://discuss.elastic.co/u/akam)\ **Replies:** 3\ **Last updated:** [July 7, 2020, 10:55am UTC](https://discuss.elastic.co/t/rsyslog-beats-together/239994 "2020-07-07T10:55:56Z")
Hello, I am trying to collect logs from rsyslog and filebeats. I have rsyslog configuration files: 01-json-template.conf template(name="json-template" type="list") { constant(value="{") constant(va… --- ## [Logstash studdown seconds after starting on its own](https://discuss.elastic.co/t/logstash-studdown-seconds-after-starting-on-its-own/240138)
**Author:** [@Jayesh\_Rathi](https://discuss.elastic.co/u/Jayesh_Rathi)\ **Replies:** 7\ **Last updated:** [July 7, 2020, 10:42am UTC](https://discuss.elastic.co/t/logstash-studdown-seconds-after-starting-on-its-own/240138 "2020-07-07T10:42:50Z")
hello, i was trying to start my logstash but it started and shutted down in a few seconds. Does anyone have any idea to resolve this issue? \[2020-07-07T15:17:11,564\]\[INFO \]\[logstash.runner \] Logstash shut down… --- ## [Logstash Events Stats API](https://discuss.elastic.co/t/logstash-events-stats-api/239212)
**Author:** [@Rahul\_Kumar4](https://discuss.elastic.co/u/Rahul_Kumar4)\ **Replies:** 4\ **Last updated:** [July 7, 2020, 9:57am UTC](https://discuss.elastic.co/t/logstash-events-stats-api/239212 "2020-07-07T09:57:07Z")
What do the fields duration\_in\_millis and queue\_push\_duration\_in\_millis in Logstash's Events Stats API response in the link below mean? https://www.elastic.co/guide/en/logstash/current/node-stats-api.html#event-stats --- ## [How gracefully clear DLQ? Is it possible?](https://discuss.elastic.co/t/how-gracefully-clear-dlq-is-it-possible/240137)
**Author:** [@dorinand](https://discuss.elastic.co/u/dorinand)\ **Replies:** 0\ **Last updated:** [July 7, 2020, 9:54am UTC](https://discuss.elastic.co/t/how-gracefully-clear-dlq-is-it-possible/240137 "2020-07-07T09:54:00Z")
I configured DLQ on my logstash, the problem is, DLQ is getting full very often. I know, there is issue about DLQ HERE . I can delete data from DLQ after it is processed, but then I have to restart logstash - pretty usel… --- ## [Logstash, Could not index event to Elasticsearch](https://discuss.elastic.co/t/logstash-could-not-index-event-to-elasticsearch/240123)
**Author:** [@defalt](https://discuss.elastic.co/u/defalt)\ **Replies:** 3\ **Last updated:** [July 7, 2020, 9:23am UTC](https://discuss.elastic.co/t/logstash-could-not-index-event-to-elasticsearch/240123 "2020-07-07T09:23:05Z")
Hi, I am trying to index some data with logstash. My config: input { redis{ host =\> "localhost" data\_type =\> "list" key =\> "timeset1" threads =\> 4 } } filter { date { match =\> \[ "timestamp" , "yyyy-MM-… --- ## [\[indices:data/write/bulk\[s\]\] is unauthorized for user \[loader-user\]"})](https://discuss.elastic.co/t/indices-data-write-bulk-s-is-unauthorized-for-user-loader-user/239169)
**Author:** [@eemtzc](https://discuss.elastic.co/u/eemtzc)\ **Replies:** 2\ **Last updated:** [July 7, 2020, 5:55am UTC](https://discuss.elastic.co/t/indices-data-write-bulk-s-is-unauthorized-for-user-loader-user/239169 "2020-07-07T05:55:46Z")
Hello, I am starting a new pipeline on one of my logstash servers, but when I start the service I get this error. Jun 29 10:50:00 mryt2ls02 logstash: \[2020-06-29T10:50:00,449\]\[INFO \]\[logstash.outputs.elasticsearch\]\[Tre… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=331) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=333)