# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=183 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 184 --- ## [Logstash Logging setting to output rubydebug info for running service](https://discuss.elastic.co/t/logstash-logging-setting-to-output-rubydebug-info-for-running-service/288435)
**Author:** [@teebu](https://discuss.elastic.co/u/teebu)\ **Replies:** 11\ **Last updated:** [November 5, 2021, 2:23am UTC](https://discuss.elastic.co/t/logstash-logging-setting-to-output-rubydebug-info-for-running-service/288435 "2021-11-05T02:23:28Z")
I'm using the default settings. When I run LS as a service, the logging in the plain.log file doesn't have any of the rubydebug output. When I run it with the -f flag, I see all the output normally. What setting file d… --- ## [Azure event hub plugin modify offset](https://discuss.elastic.co/t/azure-event-hub-plugin-modify-offset/288452)
**Author:** [@ElasticLiver](https://discuss.elastic.co/u/ElasticLiver)\ **Replies:** 3\ **Last updated:** [November 4, 2021, 10:27pm UTC](https://discuss.elastic.co/t/azure-event-hub-plugin-modify-offset/288452 "2021-11-04T22:27:59Z")
Hi where is the offset of the Azure event hub plugin is located? and if si located in the blob storage can you give me some reference where to find it? I dont have access to Azure and have to give directions to a cowor… --- ## [Logstash aggregate task](https://discuss.elastic.co/t/logstash-aggregate-task/288348)
**Author:** [@rosboc](https://discuss.elastic.co/u/rosboc)\ **Replies:** 3\ **Last updated:** [November 4, 2021, 4:06pm UTC](https://discuss.elastic.co/t/logstash-aggregate-task/288348 "2021-11-04T16:06:09Z")
Hi Team, we are using aggregate function to collect all commands for a specific ssh session but recently our vendor changed the way to track session records and right now only start and stop records have the same task\_i… --- ## [Lagging in ELK stack](https://discuss.elastic.co/t/lagging-in-elk-stack/288414)
**Author:** [@Narayan\_Banik](https://discuss.elastic.co/u/Narayan_Banik)\ **Replies:** 2\ **Last updated:** [November 4, 2021, 2:53pm UTC](https://discuss.elastic.co/t/lagging-in-elk-stack/288414 "2021-11-04T14:53:50Z")
Hi Experts, I'm using ELK stack in docker environment: logstash\_syslog:7.4.0 Elasticsearch:7.1 kibana:7.1 Sometimes observe lagging of logs in kibana dashboard. After 2/3 hours, it becomes adjusted. Lagging usually… --- ## [Logstash - parse/ filter lines having a specific string](https://discuss.elastic.co/t/logstash-parse-filter-lines-having-a-specific-string/288103)
**Author:** [@zaeemmasood](https://discuss.elastic.co/u/zaeemmasood)\ **Replies:** 4\ **Last updated:** [November 4, 2021, 1:35pm UTC](https://discuss.elastic.co/t/logstash-parse-filter-lines-having-a-specific-string/288103 "2021-11-04T13:35:26Z")
Hi All, We are running ELK 7.6.2 stack in our environment. The log generated by our application throws out messages with different Log Levels. Please see below example: 2021-10-31 19:00:01,062|DEBUG|DispatcherServlet… --- ## [Elasticsearch filter failing to start with logstash because of cluster not being available yet](https://discuss.elastic.co/t/elasticsearch-filter-failing-to-start-with-logstash-because-of-cluster-not-being-available-yet/287843)
**Author:** [@Mrizzi](https://discuss.elastic.co/u/Mrizzi)\ **Replies:** 1\ **Last updated:** [November 4, 2021, 1:18pm UTC](https://discuss.elastic.co/t/elasticsearch-filter-failing-to-start-with-logstash-because-of-cluster-not-being-available-yet/287843 "2021-11-04T13:18:59Z")
Hi everyone! Hope you're doing okay, i'm making this post because I couldn't find anything online about the following issue and I wanted to make sure i'm not overlooking something simple. I have the stack installed in … --- ## [Importing data to elasticsearch from multiple dynamic databases](https://discuss.elastic.co/t/importing-data-to-elasticsearch-from-multiple-dynamic-databases/288317)
**Author:** [@enricog84](https://discuss.elastic.co/u/enricog84)\ **Replies:** 2\ **Last updated:** [November 4, 2021, 8:31am UTC](https://discuss.elastic.co/t/importing-data-to-elasticsearch-from-multiple-dynamic-databases/288317 "2021-11-04T08:31:37Z")
Hi, I am currently evaluating (and have no further experience so far) the ELK stack. I am wondering if it would be possible to import data from multiple dynamic created MariaDB databases into Elasticsearch. Short back… --- ## [Metricbeat Logstash module payload size](https://discuss.elastic.co/t/metricbeat-logstash-module-payload-size/287247)
**Author:** [@admlko](https://discuss.elastic.co/u/admlko)\ **Replies:** 1\ **Last updated:** [November 4, 2021, 7:49am UTC](https://discuss.elastic.co/t/metricbeat-logstash-module-payload-size/287247 "2021-11-04T07:49:57Z")
Hi, I opened a ticket about this issue in the github, but it got closed and the issue was redirected here: https://github.com/elastic/beats/issues/28569 It is my understanding that the Logstash UDP output plugin doesn'… --- ## [How to access logstash keystore in jvm.options?](https://discuss.elastic.co/t/how-to-access-logstash-keystore-in-jvm-options/287906)
**Author:** [@bhargaviaaa](https://discuss.elastic.co/u/bhargaviaaa)\ **Replies:** 4\ **Last updated:** [November 4, 2021, 5:31am UTC](https://discuss.elastic.co/t/how-to-access-logstash-keystore-in-jvm-options/287906 "2021-11-04T05:31:38Z")
Please provide example to access logstash keystore in jvm.options. --- ## [(Issue) Logstash JDBC Input Plugin seems to be changing datetime column value's at output](https://discuss.elastic.co/t/issue-logstash-jdbc-input-plugin-seems-to-be-changing-datetime-column-values-at-output/288353)
**Author:** [@GPartenza](https://discuss.elastic.co/u/GPartenza)\ **Replies:** 4\ **Last updated:** [November 4, 2021, 3:20am UTC](https://discuss.elastic.co/t/issue-logstash-jdbc-input-plugin-seems-to-be-changing-datetime-column-values-at-output/288353 "2021-11-04T03:20:48Z")
Hello, I have an SQLite database where new data is written every second. Currently, I am using Logstash and its JDBC plugin to read the data in real-time and ship it to Elasticsearch for visualization. Everything is wor… --- ## [How to identify the errors reported by logstash?](https://discuss.elastic.co/t/how-to-identify-the-errors-reported-by-logstash/288283)
**Author:** [@wajika](https://discuss.elastic.co/u/wajika)\ **Replies:** 5\ **Last updated:** [November 4, 2021, 3:05am UTC](https://discuss.elastic.co/t/how-to-identify-the-errors-reported-by-logstash/288283 "2021-11-04T03:05:33Z")
I can't understand where is the error? Who can share how to quickly identify the specific location of the error? input { elasticsearch { hosts =\> \["192.168.10.141:9200"\] index =\> "apm-\*-error-\*" query =\> '{ "query… --- ## [Gz file handling by interaction as XML files](https://discuss.elastic.co/t/gz-file-handling-by-interaction-as-xml-files/288002)
**Author:** [@INS](https://discuss.elastic.co/u/INS)\ **Replies:** 6\ **Last updated:** [November 4, 2021, 1:09am UTC](https://discuss.elastic.co/t/gz-file-handling-by-interaction-as-xml-files/288002 "2021-11-04T01:09:42Z")
Hi How Can I configure input for reading per only one file. It is very important that logstash reads one file at a time as soon as it finishes parsing it and reads another one. This is related to the xml structure. I h… --- ## [Add a field from one grok match to another](https://discuss.elastic.co/t/add-a-field-from-one-grok-match-to-another/288246)
**Author:** [@mwas](https://discuss.elastic.co/u/mwas)\ **Replies:** 2\ **Last updated:** [November 3, 2021, 11:38pm UTC](https://discuss.elastic.co/t/add-a-field-from-one-grok-match-to-another/288246 "2021-11-03T23:38:14Z")
I've got log that starts with a jobid on our build cluster live following: "Job \<7073381\> is submitted to queue " I'd like to extract the job id from that line and add it to all other matched lines that look like this: … --- ## [Logstash-jdbc-input plugin issue when time reverts from Daylight Savings to Standard Time](https://discuss.elastic.co/t/logstash-jdbc-input-plugin-issue-when-time-reverts-from-daylight-savings-to-standard-time/288365)
**Author:** [@AlexandrosPan](https://discuss.elastic.co/u/AlexandrosPan)\ **Replies:** 1\ **Last updated:** [November 3, 2021, 9:25pm UTC](https://discuss.elastic.co/t/logstash-jdbc-input-plugin-issue-when-time-reverts-from-daylight-savings-to-standard-time/288365 "2021-11-03T21:25:58Z")
Hello, I use logstash with jdbc input plugin to create an Elasticsearch index and keep it synced with a Database table. My database's timezone is Asia/Nicosia and so I've set jdbc\_default\_timezone accordingly. The date… --- ## [Date to string using localtime](https://discuss.elastic.co/t/date-to-string-using-localtime/288360)
**Author:** [@ElasticLiver](https://discuss.elastic.co/u/ElasticLiver)\ **Replies:** 2\ **Last updated:** [November 3, 2021, 9:16pm UTC](https://discuss.elastic.co/t/date-to-string-using-localtime/288360 "2021-11-03T21:16:46Z")
Hi, Im trying to convert a date to string with timezone different from UTC tz = "-03:00" t = event.get("lastHeartbeat") event.set("last\_heartbeat\_text", Time.at(t.to\_f).localtime(tz)) and no matter what time is in las… --- ## [Need to make logs in order](https://discuss.elastic.co/t/need-to-make-logs-in-order/287845)
**Author:** [@cool999](https://discuss.elastic.co/u/cool999)\ **Replies:** 15\ **Last updated:** [November 3, 2021, 8:05pm UTC](https://discuss.elastic.co/t/need-to-make-logs-in-order/287845 "2021-11-03T20:05:23Z")
Hi, I have below app logs, like below but they are not in oder in kibana i.e the last log line below is not the first in kibana discover. 2021-10-27 16:51:14.864 | INFO | | | | | \[http-nio-8080-exec-32\] | next-con… --- ## [Logstash fails to start: "no such file to load -- concurrent"](https://discuss.elastic.co/t/logstash-fails-to-start-no-such-file-to-load-concurrent/288268)
**Author:** [@dwasss](https://discuss.elastic.co/u/dwasss)\ **Replies:** 9\ **Last updated:** [November 3, 2021, 7:47pm UTC](https://discuss.elastic.co/t/logstash-fails-to-start-no-such-file-to-load-concurrent/288268 "2021-11-03T19:47:06Z")
I recently upgraded my Elastic stack from 7.12 to 7.15.1 on Ubuntu 18.04. Logstash is running via systemd. Now Logstash fails to start with the following error: Nov 02 19:58:28 server logstash\[30340\]: Using bundled JDK:… --- ## [Replace paths in every field](https://discuss.elastic.co/t/replace-paths-in-every-field/287106)
**Author:** [@AnneHermann](https://discuss.elastic.co/u/AnneHermann)\ **Replies:** 5\ **Last updated:** [November 3, 2021, 6:15pm UTC](https://discuss.elastic.co/t/replace-paths-in-every-field/287106 "2021-11-03T18:15:59Z")
Hello, I've been trying to edit my logstash.config so that all file paths in all fields are changed to "anonym". input { http { port =\> 5044 codec =\> json } } filter { split { field… --- ## [Logstash + Syslog + Security](https://discuss.elastic.co/t/logstash-syslog-security/288337)
**Author:** [@MKirby](https://discuss.elastic.co/u/MKirby)\ **Replies:** 2\ **Last updated:** [November 3, 2021, 4:29pm UTC](https://discuss.elastic.co/t/logstash-syslog-security/288337 "2021-11-03T16:29:28Z")
Hello ELK gurus. I have been working on getting my first production ELK SIEM working and for the most part is it doing what is required. The one area that is really not functioning the way it needs is the Logstash and … --- ## [Logstash (Running as Windows Service) - cannot obtain logs from Windows Network Drive](https://discuss.elastic.co/t/logstash-running-as-windows-service-cannot-obtain-logs-from-windows-network-drive/288290)
**Author:** [@coldycc](https://discuss.elastic.co/u/coldycc)\ **Replies:** 3\ **Last updated:** [November 3, 2021, 4:12pm UTC](https://discuss.elastic.co/t/logstash-running-as-windows-service-cannot-obtain-logs-from-windows-network-drive/288290 "2021-11-03T16:12:08Z")
Hi Everyone, I'm having issue to get Logstash to send logs from Windows network drive while running it as Windows Service (using NSSM), it works perfectly fine using the CLI. Here's the logstash.conf: input { file {… --- ## [Call synchronous a pipeline (ex. jdbc/http input plugin) logstash from Java code and get Reponse](https://discuss.elastic.co/t/call-synchronous-a-pipeline-ex-jdbc-http-input-plugin-logstash-from-java-code-and-get-reponse/288344)
**Author:** [@ANWARIM](https://discuss.elastic.co/u/ANWARIM)\ **Replies:** 0\ **Last updated:** [November 3, 2021, 3:31pm UTC](https://discuss.elastic.co/t/call-synchronous-a-pipeline-ex-jdbc-http-input-plugin-logstash-from-java-code-and-get-reponse/288344 "2021-11-03T15:31:43Z")
Hi, I am completely new in Logstash & ES world. Is it possible to call a pipleline synchronous from java code and wait till get the reponse, in this case we want to block the recored till update is completed and Elastic… --- ## [Redis pipeline is recognized but not receiving logs](https://discuss.elastic.co/t/redis-pipeline-is-recognized-but-not-receiving-logs/288327)
**Author:** [@nyquillus](https://discuss.elastic.co/u/nyquillus)\ **Replies:** 0\ **Last updated:** [November 3, 2021, 12:45pm UTC](https://discuss.elastic.co/t/redis-pipeline-is-recognized-but-not-receiving-logs/288327 "2021-11-03T12:45:31Z")
Hi, I'm trying to add redis as a buffer to my stack in docker-compose. My logstash.conf input section is like below: input { redis { host =\> "redistest" port =\> "6379" ssl =\> false data\_type =\> "list"… --- ## [Create timestamp from nested JSON elements](https://discuss.elastic.co/t/create-timestamp-from-nested-json-elements/287992)
**Author:** [@alt-glitch](https://discuss.elastic.co/u/alt-glitch)\ **Replies:** 1\ **Last updated:** [November 3, 2021, 11:44am UTC](https://discuss.elastic.co/t/create-timestamp-from-nested-json-elements/287992 "2021-11-03T11:44:02Z")
Hi all, I am indexing nested JSON files using Logstash and want to extract the timestamp from them. Here are a couple of examples. Example 1 { "ip": "127.0.0.1", "data": { "postgres": { "status": "succes… --- ## [How to create elasticsearch template in logstash pipeline](https://discuss.elastic.co/t/how-to-create-elasticsearch-template-in-logstash-pipeline/287898)
**Author:** [@gokulnath112](https://discuss.elastic.co/u/gokulnath112)\ **Replies:** 3\ **Last updated:** [November 3, 2021, 9:11am UTC](https://discuss.elastic.co/t/how-to-create-elasticsearch-template-in-logstash-pipeline/287898 "2021-11-03T09:11:58Z")
Hi Team, Greetings..! Is there any way to create Elasticsearch index template in logstash pipeline? I want push csv data to Elasticsearch via logstash, but I want to store data in custom data type. Please help me to cr… --- ## [Grok filter in logstash for more than one different lines](https://discuss.elastic.co/t/grok-filter-in-logstash-for-more-than-one-different-lines/288108)
**Author:** [@root\_linux](https://discuss.elastic.co/u/root_linux)\ **Replies:** 5\ **Last updated:** [November 3, 2021, 6:37am UTC](https://discuss.elastic.co/t/grok-filter-in-logstash-for-more-than-one-different-lines/288108 "2021-11-03T06:37:56Z")
I have logs in which there are different types of lines are present. And I want to write grok filter for each line. How I can I achieve it? My logs are as given below: Oct 5 14:05:53 centos-8gb-nbg1-1 openvpn: Tue Oct … --- ## [How to get logstash logs in stack monitoring (k8s)](https://discuss.elastic.co/t/how-to-get-logstash-logs-in-stack-monitoring-k8s/288159)
**Author:** [@antondubek](https://discuss.elastic.co/u/antondubek)\ **Replies:** 0\ **Last updated:** [November 1, 2021, 4:03pm UTC](https://discuss.elastic.co/t/how-to-get-logstash-logs-in-stack-monitoring-k8s/288159 "2021-11-01T16:03:21Z")
We run our elastic stack utilising ECK however have run into an issue when it comes to Logstash logs. Our Logstash's are deployed using the official helm chart - Logstash Helm We expected, as outlined with the Elastics… --- ## [Replace my \\r\\n with a new line using mutate gsub](https://discuss.elastic.co/t/replace-my-r-n-with-a-new-line-using-mutate-gsub/288240)
**Author:** [@rkhapre](https://discuss.elastic.co/u/rkhapre)\ **Replies:** 6\ **Last updated:** [November 2, 2021, 9:35pm UTC](https://discuss.elastic.co/t/replace-my-r-n-with-a-new-line-using-mutate-gsub/288240 "2021-11-02T21:35:49Z")
Hi All I have a csv data in this format { "message" : "value1,value2,value3\\r\\nvalue4,value5,value6" } My expected result is { "message" : "value1,value2,value3 value4,value5,value6" } I have tried mutate g… --- ## [Renaming fields dinamically](https://discuss.elastic.co/t/renaming-fields-dinamically/288237)
**Author:** [@ElasticLiver](https://discuss.elastic.co/u/ElasticLiver)\ **Replies:** 4\ **Last updated:** [November 2, 2021, 5:57pm UTC](https://discuss.elastic.co/t/renaming-fields-dinamically/288237 "2021-11-02T17:57:12Z")
Hi, I got multiple fields with composed names likes the ones below records.properties.multiVmGroupId records.properties.rpoInSeconds I want to get rid of the records.properties part, and left only the last word. is th… --- ## [Using Logstash HTTP\_poller for NTLM authentication](https://discuss.elastic.co/t/using-logstash-http-poller-for-ntlm-authentication/288249)
**Author:** [@jondayko](https://discuss.elastic.co/u/jondayko)\ **Replies:** 0\ **Last updated:** [November 2, 2021, 4:32pm UTC](https://discuss.elastic.co/t/using-logstash-http-poller-for-ntlm-authentication/288249 "2021-11-02T16:32:55Z")
So, currently I am trying to use the http\_poller input plugin to send OData (JSON) to Kibana using Logstash. However, I keep running into problems trying to authenticate using NTLM. I was able to figure out that you can … --- ## [Nested aggregation](https://discuss.elastic.co/t/nested-aggregation/288061)
**Author:** [@mofaris](https://discuss.elastic.co/u/mofaris)\ **Replies:** 3\ **Last updated:** [November 2, 2021, 4:13pm UTC](https://discuss.elastic.co/t/nested-aggregation/288061 "2021-11-02T16:13:43Z")
Hello guys, I am trying to aggregate some data from my db by Logstash and send them to Elasticsearch. My data looks like this: The table consists of 4 columns. each question\_id may have one or more answer\_id and eac… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=182) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=184)